Agricultural and construction machines with artificial intelligence

Abstract

A machine with a chassis, one or more ground engaging elements, a microphone, a speaker, a machine system actuator and a controller for receiving verbal requests or instructions from an operator throu

Description

[0001] The following detailed description of embodiments of the invention references the accompanying drawings. The embodiments are intended to describe aspects of the invention in sufficient detail to enable those skilled in the art to practice the invention. Other embodiments can be utilized and changes can be made without departing from the spirit and scope of the invention as defined by the claims. The following description is, therefore, not to be taken in a limiting sense. Further, it will be appreciated that the claims are not necessarily limited to the particular embodiments set out in this description.

[0002] In this description, references to “one embodiment”, “an embodiment”, or “embodiments” mean that the feature or features being referred to are included in at least one embodiment of the technology. Separate references to “one embodiment”, “an embodiment”, or “embodiments” in this description do not necessarily refer to the same embodiment and are also not mutually exclusive unless so stated and / or except as will be readily apparent to those skilled in the art from the description. For example, a feature, structure, act, etc. described in one embodiment may also be included in other embodiments, but is not necessarily included. Thus, the present technology can include a variety of combinations and / or integrations of the embodiments described herein.

[0003] When elements or components are referred to herein as being “connected” or “coupled,” the elements or components may be directly connected or coupled together or one or more intervening elements or components may also be present. In contrast, when elements or components are referred to as being “directly connected” or “directly coupled,” there are no intervening elements or components present.

[0004] The present invention relates to a talking machine that can answer questions, provide information and respond to requests from an operator in real time. This allows the operator to receive information about the machine, the task, the environment and other aspects of their business; to have their questions answered; and to request the machine to perform certain functions all while they are working and without interrupting their work, leading to greater efficiency and customer satisfaction. While the operator may interact with the machine using various methods including text and audible submissions, in some embodiments it is desirable for the operator to communicate audibly so that it is a more natural interaction without requiring the operator to take his or her hands off the machine’s controls. This technology may be implemented on various machines in the agriculture and construction industries including, without limitation, tractors, bulldozers, combine harvesters, sprayers, earth movers, loaders and so forth.

[0005] Turning now to the drawing figures, and initially Fig. 1, a machine constructed in accordance with embodiments of the invention is illustrated. The illustrated machine is a tractor 10, and the tractor 10 is attached to and pulling an implement 12. The tractor 10 and implement 12 may be used in the agriculture or construction industries. The implement 12 is pivotally attached to a hitch 14 of the tractor 10. A working portion 16 of the implement 12 is separated from the tractor 10 by a distance 18, wherein the working portion 16 of the implement is a portion of the implement 12 that performs work, such as the portion of a tillage implement that engages the ground.

[0006] The tractor 10 and implement 16 include an electronic system 20 illustrated schematically in Fig. 2. The system 20 broadly includes a controller 22, a positioning system 24, a user interface 26, a plurality of sensors 28, one or more actuators 30, one or more storage components 32, one or more input / out ports 34, and a communications gateway 36.

[0007] The positioning system 24 uses one or more technologies to determine a location of the tractor 10, a location of the implement 12, or both. The positioning system 24 may include one or more global navigation satellite system (GNSS) receivers, such as GNSS receivers configured to receive signals from one or more positioning systems such as the United States’ global positioning system (GPS), the European GALILEO system, the Chinese Beidou system and / or the Russian GLONASS system, and to determine a location of the tractor 10 using the received signals. Alternatively or additionally, the positioning system 24 may use light detection and ranging (LiDAR) technology, radio detecting and ranging (RADAR) technology, one or more cameras, one or more inertial measurement units (IMUs), or a combination thereof to determine position or assist in determining position. One or more components of the positioning system 24 may be located on the tractor 10 and one or more components of the positioning system 24 may be located on the implement 16. By way of example, a first GNSS receiver and a first IMU may be located on the tractor 10 and a second GNSS receiver and a second IMU may be located on the implement 16 to assist with guidance. The user interface 26 includes components for receiving information, instructions or other input from a user and may include buttons, switches, dials, and microphones, as well as components for presenting information or data to users, such as displays, light-emitting diodes, audio speakers and so forth. The user interface 26 may include one or more touchscreen displays capable of presenting visual representations of information or data and receiving instructions or input from the user via a single display surface.

[0008] The sensors 28 may be associated with any of various components or functions of the tractor 10 and / or the implement 12 including, for example, various elements of the engine, transmission(s), and hydraulic and electrical systems. One or more of the sensors 28 is or includes a microphone or other sound sensor for detecting speech from the operator. Furthermore, one or more of the sensors 28 may be configured and positioned to detect information about the environment in which the tractor 10 and implement 12 are operating including temperature, ambient light, objects external to the tractor 10 and a ground surface. The actuators 30 are configured and placed to drive certain functions of the tractor 10 and / or the implement 12 and may take virtually any form but are generally configured to receive control signals or instructions from the controller 22 (or other component of the system 20) and to generate a mechanical response to the control signals or instructions. One or more of the actuators 30 is or includes a speaker to communicate audibly with the operator. By way of example, the sensors 28 and actuators 30 may be used in automated steering of the tractor 10 wherein the sensors 28 detect a current position or state of steered wheels and the actuators 30 drive steering action of the steered wheels. According to another example, the sensors 28 include a microphone for detecting speech from the operator and the actuators 30 include electric actuators or hydraulic actuators for responding to a request submitted by the operator and received via the microphone. The operator may may an audible request that an implement be raised, for example, wherein the system 20 responds by raising the implement by controlling a hydraulic actuator. In this example the actuators 30 include a speaker to provide information to the operator that he or she verbally requests.

[0009] The controller 22 is a computing device or system and includes one or more integrated circuits programmed or configured to implement the functions described herein and associated with the tractor 10 and / or the implement 12. By way of example, the controller 22 may be a digital controller and may include one or more general purpose microprocessors or microcontrollers, programmable logic devices, application specific integrated circuits or other computing or logic devices. The controller 22 may include multiple computing components, such as electronic control units, placed in various different locations on the tractor 10 and / or the implement 12, and may include one or more computing devices connected to the system 20 through the I / O port 34 and / or the gateway 36. The controller 22 may also include one or more discrete and / or analog circuit components operating in conjunction with the one or more integrated circuits or computing components. Furthermore, the controller 22 may include or have access to one or more memory elements (not illustrated) operable to store executable instructions, data, or both. The storage component 32 stores data and preferably includes a non-volatile storage medium such as solid state, optic or magnetic technology. The storage component 32 may store, for example, map information relating to a field in which the tractor 10 is operating or wayline information for use in automated guidance, as explained below in greater detail.

[0010] The communications gateway 36 includes one or more wireless transceivers configured to communicate with external systems, machines or devices using wireless communications technology. The communications gateway 36 may include one or more wireless transceivers configured to communicate according to one or more wireless communications protocols or standards, such as one or more protocols based on the IEEE 802.11 family of standards (“Wi-Fi”), the Bluetooth wireless communications standard, a 433 MHz wireless communications protocol or a protocol for communicating over a cellular telephone network. Alternatively or additionally, the communications gateway 36 may include one or more wireless transceivers configured to communicate according to one or more proprietary or non-standardized wireless communication technologies or protocols, such as proprietary wireless communications protocols using 2.4 GHz or 5 GHz radio signals. Thus, the communications gateway 36 enables wireless communications with other machines such as other harvesters or tractors, with external devices such as laptop or tablet computers or smartphones, and with external communications networks such as a cellular telephone network or Wi-Fi network.

[0011] It will be appreciated that, for simplicity, certain elements and components of the system 20 have been omitted from the present discussion and from the diagram illustrated in Fig. 2. A power source or power connector is also associated with the system 20, for example, but is conventional in nature and, therefore, is not discussed herein.

[0012] The controller 22 is configured to use the other components of the system 20 to interact with an operator in a natural and convenient manner. The following is an exemplary verbal interact between an operator named Jack and a machine implementing an embodiment of the invention: Machine: Good morning, Jack. Operator: Good morning, Machine. Machine: I'm setting the tractor up for you. Statistics from last week, your average fuel consumption was twenty-nine liters per hour, which is excellent for the tasks driven. You could focus even more on choosing a higher gear for spraying. Operator: All right, thanks. Machine: Please notice that next oil service is due within sixty hours. Contact dealer service for this. Please let them also know about fault code A311. It is about the transmission oil temperature sensor malfunction. Operator: All right, thanks. Please let me know how to set up the smart turn for this field. Machine: For activating smart turn, please choose auto steering menu, then go to settings and choose the width of implement. Then define the field boundary and you are good to go. Machine: We have forty hectares to be cultivated and drilled today. As we have four hours before the rain will begin, we should reserve one more unit for the work. Operator: OK, call David now. Hi David, can you please join us for drilling the main field with the other Q tractor? David: Hi Jack, Sure, I'm on my way soon. Operator: Machine, please set up the tire pressures. Machine: Adjusting tire pressure. Operator: Please set up the auto guide for this field. Machine: Roger that. Here we go. Just let the auto guide take it from here. Operator: Are we still good with fuel with the old Q series as well? Do we need to refuel during the day? Machine: Nope, with three tractors we will be more than fine and get it done in less than four hours. Also, all the other tractors are ready for speed. Machine: It is time for oil service for two tractors next week. As Tuesday seems to be a rainy day. Should I let the service book a visit for that? Operator: Please do that. Machine: You and the other driver will need some refueling yourselves. Operator: Coffee and some fruits will be fine. I'll have my dinner with my family and absolutely no snack is needed for the fully automatic tractor. Machine: Absolutely, and everything is fine with it as usual. By the way, the fuel prices just dropped. Should we order our annual fuel refill now? Operator: Definitely. I will talk to our Rep, please call him. Hi Fuelie, how is it with the diesel and electricity prices today? Do you want to make a good deal today? Operator: Machine, what is your estimation for finishing this field? Machine: We will be done in less than two hours. Operator: Thanks call my honey. Hi dear, I will be back home for dinner in less than two hours. How about having sauna tonight? Wife: Wonderful. We will be waiting for you dear. Hugs. Machine: Thanks machine for today. Send the reports to the FMIS. See you tomorrow.

[0013] An embodiment of the invention comprises a single board computer capable of running a Large Language Model locally and parsing text from speech. The hardware and software specs are below and the operational flow is illustrated in Fig. 3.

[0014] Hardware Specifications: The program uses a NVIDIA Edge GPU -specifically the Jetsen Orin.

[0015] Software Specifications: Speech to text: Whisper by OpenAI LLM: Customized search and summary package based on GPT4 by OpenAI. The customization includes limiting the LLM to only generate answers from the tractor’s operator manual. Text to speech: PiperTTS

[0016] Advantages of this system include having an audio-based assistant which can answer any questions about the machine, about the task it is performing or about other matters and can perform actions in response to operator requests or instructions. The operator would not need to stop what he or she is doing and consult the operator’s manual. This will increase accessibility, easier resolution of tractor issues and higher tech adoption of different tractor features. All of this has the potential of greatly increasing customer satisfaction. Problem <:;£• hetti ArN? witLiN the trier's sob. the ysw hev* tyo spoors ■ Co?; o ror < search o of codes fif?d the Levant Proposed Solution Use edge computings large language models, and text-to-speech algorithms to allow farmers to talk to their tractors when they receive error codes. Farmers can easily learn what is wrong with their tractor and if possible, repair the issue an their awn. By utilizing a Raspberry Pi 5 with a Google Carol Al TPU for hardware acceleration, responses can be rapidly generated completely locally on the tractor. Business Impact Reduces downtime for faulty machinery on the farm, allowing farmers to diagnose and fix problems with their equipment rapidly. Empowers farmers to repair their equipment, increasing customer satisfaction since farmers greatly appreciate the ability to diagnose and fix their equipment rapidly. Saws time and money for technicians who would otherwise have to be assisting farmers in diagnosing and fixing minor problems that the farmer could solve on their own. Adding an LLM that runs iacaHy an the Raspberry Ph accelerated by the Goalie Coral TPUf allowing farmers to converse with their tractors to discuss the current problem and possible solutions. By utilising search algorithms, relevant portions of documents related to the current fault codes could be inputted into this LL.M to ensure the accuracy of results during these conversations,

[0017] Large language models (LLMs) have a large application scope which could be useful for high knowledge applications. In agriculture, one such use case is as a supportive copilot, that has access to the broader scope of digital information related to a specific farmer, etc historic machine information via the farm management software. I.E.

[0018] A context aware copilot may include three parts. There is the agronomic information part, which comes from the farm management software (yield maps, seed charts, application schedules etcetera). There is the machine context, which relates to machine specific telematics etcetera, and lastly there is the user context which relates to the farming methods used by the farmer, their maintenance and service planning, dealer interactions etcetera.

[0019] All three of these contexts need to come together to meaningfully integrate AI / LLMs into agriculture. One challenge associated with implementing AI / LLMs in the agriculture context is providing sufficient computing power. Embodiments of this invention relate to the creation of an architecture, which includes cloud, local hardware and smartphones (and other mobile devices), to solve the compute and context problem specific to the agriculture use case.

[0020] Why is the smartphone relevant? Well, many smartphone companies, especially Samsung, Google and Apple, are very heavily investing in increasing the local hardware capabilities of their phones to provide a better AI experience to users. Traditionally in agriculture, new tech on the machine equates to a new terminal (or interaction with the existing terminal). As such, the conventional approach would involve developing our own AI terminal and / or hardware that the user can retrofit to the cab. This approach would involve duplicating hardware already existing on smartphones and similar devices. Rather than doing that, this invention involves using the smartphone with the machine.

[0021] With reference to Fig. 4, suppose a farmer had an AI copilot that they could interact with using text and / or natural language. The farmer would interact with the copilot via their smartphone or some kind of mobile device capable of travelling with the farmer (including tablets or a self-contained HW unit). That mobile device would be responsible for holding the user context, as that context should travel with the farmer.

[0022] When the farmer enters the cab of a tractor (or any other machine), that tablet ideally would need access to the machine as well. This is where the hardware comes in. The machine could be outfitted with various AI controllers or telematics units. These controllers could be simple CANBUS dongles to look at machine telematics and diagnostics. It could also be a hardware controller specific to an implement (such as a planter, tillage equipment or sprayer), or to a specific complex subsystem (such as engine or hydraulics).

[0023] The tablet or mobile device can then connect to these controllers, which themselves could contain AI models trained and developed specific to their use. The tablet through which the farmer interacts with would now have access to both user context and machine specific context. A data connection on the tablet would provide access to the cloud where the agronomic information resides. Now you have an AI agent in the hands of the farmer, that has access to all the information, without the need to centralize the information into a high compute device or location. An exemplary system is illustrated in Fig. 5, wherein a phone, table or other external, portable electronic device 50 is in communication with the system 20 via the gateway 36 or the I / O ports 34 and provides the AI / LLM functionality for the machine as described herein.

[0024] We are building out some very advanced tech in the form of retrofit kits. This architecture here would allow us to break apart the functionality, and better serve the farmers. Instead of needing to buy a big expensive terminal that is specific to a platform or machine, you could buy AI-Telematics dongles that are machine specific, but which interface with a common hardware device such as a smartphone or tablet. We as a company can then release new units in the form of retrofit hardware that allows older machines to interface with our advanced tech. We could also easily integrate these units as dealer options for new machines.

[0025] An example use scenario: Farmer: Hey tractor, tell me about my upcoming maintenance. AI: Your machine has 21 hours before the next scheduled service (machine context). That service item entails xyz processes. From your tillage schedule (agronomic context) I see we have some downtime on Thursday, would you like me to setup an appointment at your dealer for a service (user context)?

[0026] Although the invention has been described with reference to the preferred embodiment illustrated in the attached drawing figures, it is noted that equivalents may be employed and substitutions made herein without departing from the scope of the invention as recited in the claims.

[0027] The claims at the end of this patent application are not intended to be construed under 35 U.S.C. § 112(f) unless traditional means-plus-function language is expressly recited, such as “means for” or “step for” language being explicitly recited in the claim(s).

[0028] Having thus described the preferred embodiment of the invention, what is claimed as new and desired to be protected by Letters Patent includes the following:

Claims

1. A machine comprising:a chassis;one or more ground engaging elements for supporting the chassis on a ground surface and moving the machine along the ground surface;a microphone;a speaker;a machine system actuator; anda controller for-receiving verbal requests or instructions from an operator through the speaker,providing a response to the verbal requests or instructions through the speaker, andproviding a response to the verbal requests or instructions using the machine system actuator.

2. A machine comprising:a chassis;one or more ground engaging elements for supporting the chassis on a ground surface and moving the machine along the ground surface;a communications and control system for controlling operation of the machine and communications with other machines, devices and systems external to the machine; andan external, handheld electronic device in communication with the communications and control system, the handheld electronic device providing artificial intelligence for operation of the machine through the communications and control system.