System and method for detecting crop lodging

Abstract

A system for detecting crop lodging areas (710) in a field. A camera (220, 302) mounted to an agricultural machine (300) (e.g., a combine harvester) captures images (702, 802) of a crop and the field and a computing system (200) determines attributes (708) of the crop and the field in the images (702, 802). The aforesaid two steps (steps 502, 504) can occur continuously as the machine (300) moves through a field. The determined attributes (708) of the crop and the field are shown on a display device (216) in a cabin (301) of the machine (300) so that an operator can visualize the crop and the field and areas affected by crop lodging in front of the machine. In some embodiments, the determined attributes (708) of the crop and the field are used as input for a controller (220, 230) of the machine (300) that can control operation of the machine according to the determined attributes (708) (step 512).

Description

SYSTEM AND METHOD FOR DETECTING CROP LODGINGTECHNICAL FIELD

[0001] The present disclosure relates to a system for detecting crop lodging in a crop field.BACKGROUND

[0002] Lodging is the shifting of plant roots or stems from more typical vertical positions. Lodging makes it difficult to harvest crops, can cause damage to crops, and often reduces yield. It may occur when the root system or stem base does not have adequate support for the plant. Lodging can be a result of environmental conditions such as rainfall, wind, soil condition, and others. Lodging can also include the bending over of stems near ground level of grain crops. Lodging in cereal crops is often a result of the combined effects of inadequate standing power of the crop and environmental conditions. For example, lodging often affects cereals such as wheat and rice.

[0003] Reducing lodging is a significant goal in agricultural research and precision farming and agriculture, and precision farming and agriculture may improve upon known ways for reducing lodging such as producing shorter varieties of crops.

[0004] Precision agriculture or precision farming is a farming management model based on measuring and responding to inter and intra-field variability in crops and farming conditions. The goal of precision agriculture research is to define a decision support system (DSS) for farming management to enhance returns and increase preservation of resources. Specifically, the precision in responses to variability in farming can be improved when known and predetermined farming information is processed and organized to enhance the information and then used to assist in the control and management of farming. Although precision farming can enhance returns and increase preservation of resources, it can complicate farming information systems.

[0005] Currently, farming management information systems (FMISs) are pervasive in farming and a significant factor in the furthering of precision agriculture. Such information systems can track measuring and responding to inter and intra-field variability in crops and farming conditions as well as enhance DDS for farming management. FMISs allow for new opportunities to improvefarming and precision agriculture. However, even though FMISs are improving precision farming, present FMISs have limitations and can be dramatically improved upon considering relatively recent advancements in computer engineering and computer science. One problem with previous systems is the collection and organization of information from farming, including the collection and organization of information on crop lodging. This can be a problem since farming conditions and lodging can vary greatly in the operations from one field to another.

[0006] These are just some of the many issues that can be improved upon in farming, and specifically, in precision agriculture as well as crop lodging tracking.SUMMARY

[0007] Described herein are technologies for detecting crop lodging in a crop field, such as for detecting lodging while harvesting a crop to improve upon some technical problems in the precision farming and agriculture. The techniques disclosed herein provide specific technical solutions to at least overcome the technical problems mentioned in the background section or other parts of the application as well as other technical problems not described herein but recognized by those skilled in the art.

[0008] As mentioned in the background section, reducing lodging may begin with shorter crops; however, precision agriculture or farming may independently improve lodging in crop fields or at least provide improvement to known tactics. Reducing lodging or at least tracking it can possibly increase yields as well as lessen lodging in future crops. With the techniques described herein, precision agriculture or farming can be improved to include the detection of lodging by automated systems as well as using the detected lodging to provide visual operator assistance of agricultural machinery and improve control systems as well as FMIS mapping.

[0009] In some embodiments, a system or method to detect lodging areas in an agricultural field is implemented through an agricultural machine (e.g., a combine harvester) equipped with a machine location tracking system (e.g., one of many known real-time kinematic positioning (RTK) or global positioning satellite systems, or a combination thereof) as well as equipped with a camera configured to capture images of an area in front of the machine (e.g., an RGB camera, a grayscale camera, CMYK camera, etc.). Also, to implement the system or method a computing system is provided locally with the machine or remotely to process data captured by the camera and possibly fromadditional sensors to detect lodging and use the detected lodging to provide additional functionality such as a visual interface for an operator or input for a control system of the machine.

[0010] In some embodiments, the camera can be replaced with another type of sensor that can capture data points (such as enough data points to provide an image) of the area in front of the machine. For example, in some embodiments, LIDAR attached to the machine captures data points or an image of an area in front of the machine and the computing system processes such data points or the image to perform the detection of the lodging. In some embodiments, the data points captured by the LIDAR are converted into an image for processing to detect lodging.

[0011] In some embodiments, an agricultural vehicle includes an RGB camera mounted in the front of the vehicle with a wide-angle horizontal field of view. The possible mounting points can include the top of a cabin or on the hood of the cabin. The camera can capture the images of the crop in front of the vehicle or machine during operation. The images can be processed by a computing system, such as through a deep learning process for pixel-wise semantic segmentation. In some instances, SEGNET or another type of a deep convolutional encoder-decoder architecture for image segmentation is used in or as the deep-learning process such as U-NET (which is a network for biomedical image segmentation), fully convolutional network (FCN), DEEPLABV2 or DEEPLABV3, and DECONVNET. For an RGB input image, the deep-learning process can provide an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background', for example. Based on the known field of view of the camera, its mounting position and rotation with respect to the vehicle, the detected areas can be transformed from camera frame to vehicle frame. Since the vehicle can be equipped with a location tracking system or a machine position tracking system (such as RTK-GPS), the data from the vehicle frame can then be transformed into input for a vehicle control system or other types of functionality such as map generation. In some other embodiments, the RGB camera can be replaced with a grayscale camera, a CMYK camera, or any other type of camera having a different color mode than RGB, grayscale or CMYK. And, the captured images of such other cameras can be processed by the computing system using similar techniques as those used for the RGB images to detect or determine lodging using similar techniques as those used for the RGB images. In some other embodiments, the RGB camera can be replaced with LIDAR or radar or another way of capturing data points of the field in front of the machine. And, the data points captured by such a sensor can be converted into the image for processing todetect or determine lodging using similar techniques as those used for the RGB images or other types of camera captured images.

[0012] With respect to some embodiments, disclosed herein are computerized methods for detecting crop lodging and using such information, as well as a non-transitory computer-readable storage medium for carrying out technical operations of the computerized methods. The non- transitory computer-readable storage medium has tangibly stored thereon, or tangibly encoded thereon, computer readable instructions that when executed by one or more devices (e.g., one or more personal computers or servers) cause at least one processor to perform a method for improved systems and methods for detecting crop lodging and using such information.

[0013] For example, some embodiments are or include a method for detecting crop lodging areas (e.g., see crop lodging areas 710 shown in FIG. 7) in a field while an agricultural machine (e.g., see any one of the agricultural machines shown in FIG. 1 as well as combine harvester 300 depicted in FIG. 7) is moving through the field. Such as method includes capturing, by a camera (e.g., see electronics 220 depicted in FIG. 2 and camera 302 shown in FIG. 3) mounted to the agricultural machine, images (e.g., see images 702 and 802 shown in FIGS. 7 and 8 respectively) of the field and a crop in the field (e.g., see step 502 of method 500 depicted in FIG. 5). Such as method also includes determining, by a computing system (e.g., see computing system 200 shown in FIG. 2) communicatively coupled to the camera, attributes (e.g., see attributes 708 shown in FIGS. 7 and 8) of the crop and the field in the images (e.g., see step 504 of method 500 shown in FIG. 5). The determined attributes include crop lodging areas (e.g., see crop lodging areas 710). In some examples, the capturing of the images and the determining of the attributes occur continuously as the agricultural machine moves through a field. In some examples of the method, the captured images include RGB images. Also, in some embodiments, it is to be understood that the determinations described herein can include assigning geolocation to each pixel or groups of pixels in an image.

[0014] Also, in some embodiments, where there are multiple cameras of the harvester or an agricultural machine (in general), the capturing of the images described herein can include multiple cameras capturing images from different positions and thus capturing different perspectives of the field and the crop such that different perspectives of crop lodging areas are captured and more information is provided for the determining of attributes of the crop and the field. Further, theagricultural machine can have multiple cameras mounted around itself that can provide a greater field of view and coverage of the area around the machine (even a 360-degree view). For example, in some cases, multiple cameras are mounted around the machine, and the determinations from the images of the attributes are based on combining images taken from different perspectives into a single representation of the crop and field around the machine.

[0015] In some examples, the method further includes communicating, by the computing system, the determined attributes over a network (e.g., see bus 218 depicted in FIG. 2 or network 104 shown in FIG. 1) to a user interface device (e.g., see user interface 216 shown in FIG. 2), e.g., see step 506 of method 500. The method can also include displaying at least the crop lodging areas by a display of the user interface device, e.g., see step 508 of method 500. In some of such examples, the capturing of the images, the determining and communicating of the attributes, and the displaying of the crop lodging areas occur continuously as the agricultural machine moves through a field. Also, in some of such examples, the displaying of the crop lodging areas includes displaying an RGB video of the crop lodging areas (e.g., see image 702 shown in FIG. 7), and the crop lodging areas are identified in the video or a separate video (e.g., see prediction view 706 shown in FIG. 7). More specifically, in some cases, the displaying the RGB video of the crop lodging areas includes displaying the RGB video along with a second video of a corresponding area of the crop and the field that is displayed without the identification of the crop lodging areas (e.g., see videos 902 and 906 shown in FIG. 9). Also, in some instances, the RGB video overlaps the second video and both of the videos are displayed to at least some extent (e.g., see video 906).

[0016] In some examples of the method, the camera is mounted to a front and top portion of the agricultural machine. In some examples, the camera includes a wide field of view (e.g., see field of view 401 shown in FIG. 4).

[0017] In some examples of the method, the determination of the attributes of the crop and the field in the images is based at least on digital signal processing (e.g., see step 602 of step 504 show in FIG. 6). In some examples of the method, the determination of the attributes of the crop and the field in the images is based at least on a computer vision analysis (e.g., see FIG. 6). More specifically, in some cases, the determination of the attributes of the crop and the field in the images is further based on digital signal processing (e.g., see step 602). Further, in some instances, the digital signal processing occurs prior to the computer vision analysis as a pre-processing step togenerate enhanced input for the computer vision analysis. Also, even more specifically, in such cases, the computer vision analysis includes inputting the enhanced input into an artificial neural network (ANN) (e.g., see scheme 607 and step 604 of step 504 shown in FIG. 6), and the determination of the attributes of the crop and the field in the images is based at least on output of the ANN (e.g., see step 606 of step 504 depicted in FIG. 6).

[0018] In general, some examples include the computer vision analysis including inputting an input into an ANN (e.g., see scheme 607 shown in FIG. 6), and the determination of the attributes of the crop and the field in the images is based at least on output of the ANN (e.g., see step 608 of step 504 shown in FIG. 6). More specifically, the computer vision analysis can include inputting aspects of the images or derivatives of aspects of the images into an ANN (e.g., see scheme 607), and the determination of the attributes of the crop and the field in the images is based at least on output of the ANN. Such these more specific examples, the ANN can include or can be a part of a deep learning process that determines attributes of the crop and the field or can be a basis for the determination of the attributes of the crop and the field. In such cases, the deep learning process can include a convolutional neural network (CNN) for semantic pixel-wise segmentation (e.g., see scheme 704 shown in in FIGS. 7 and 8).

[0019] With respect to some embodiments, a system is provided that includes at least one computing device configured to provide improved ways for detecting crop lodging and using such information to improve farming. And, with respect to some embodiments, a method, such as one of the aforesaid methods, is provided to be performed by at least one computing device. In some example embodiments, computer program code can be executed by at least one processor of one or more computing devices to implement functionality in accordance with at least some embodiments described herein; and the computer program code being at least a part of or stored in a non-transitory computer-readable medium.

[0020] For example, some embodiments are or include a system for detecting crop lodging areas (e.g., see crop lodging areas 710 depicted in FIG. 7) in a field while an agricultural machine (e.g., see any one of the agricultural machines shown in FIG. 1 as well as combine harvester 300 shown in FIG. 3) is moving through the field. Such a system includes a camera (e.g., see electronics 220 shown in FIG. 2 as well as camera 302 shown in FIG. 3) mounted to the agricultural machine and configured to capture images (e.g., see images 702 and 802 shown in FIGS. 7 and 8 respectively) ofthe field and a crop in the field. The system also includes a computing system (e.g., see computing system 200 depicted in FIG. 2) communicatively coupled to the camera and configured to determine attributes (e.g., see attributes 708 shown in FIGS. 7 and 8) of the crop and the field in the images (e.g., see instructions 222 shown in FIG. 2). The determined attributes include crop lodging areas, and in some cases, the capturing of the images and the determining of the attributes occurs continuously as the agricultural machine moves through a field.

[0021] Also, for example, some embodiments are or include a system for detecting crop lodging areas ((e.g., see crop lodging areas 710) in a field while an agricultural machine (e.g., see any one of the agricultural machines shown in FIG. 1 as well as combine harvester 300) is moving through the field. Similarly, such as system includes camera mounted to the agricultural machine and configured to capture images of the field and a crop in the field. Also, the system includes a computing system communicatively coupled to the camera, as well as a display of a user interface (e.g., see user interface 216 shown in FIG. 2). The computing system is configured to: determine attributes of the crop and the field in the images (e.g., see instructions 222), wherein the determined attributes include crop lodging areas (e.g., see lodging areas 710); and communicate the determined attributes over a network (e.g., see bus 218 or network 104) to the user interface device. The display of the user interface is configured to display at least the crop lodging areas. And, in some cases, the capturing of the images, the determining and communicating of the attributes, and the displaying of the crop lodging areas occurs continuously as the agricultural machine moves through a field.

[0022] Also, in some other embodiments of the system, the RGB camera can be replaced with a grayscale camera, a CMYK camera, or any other type of camera having a different color mode than RGB, grayscale or CMYK. And, the captured images of such other cameras can be processed by the computing system using similar techniques as those used for the RGB images to detect or determine lodging. In some other embodiments, the RGB camera can be replaced with LIDAR or radar or another way of capturing data points of the field in front of the machine. And, the data points captured by such a sensor can be converted into the image for processing to detect or determine lodging using similar techniques as those used for the RGB images or other types of camera captured images.

[0023] The systems and methods described herein overcome some technical problems in farming, in general, as well as some technical problems in detecting lodging and using suchinformation to improve farming, specifically. Also, the techniques disclosed herein provide specific technical solutions to at least overcome the technical problems mentioned in the background section or other parts of the application as well as other technical problems not described herein but recognized by those skilled in the art.

[0024] These and other important aspects of the invention are described more fully in the detailed description below. The invention is not limited to the particular methods and systems described herein. Other embodiments can be used and changes to the described embodiments can be made without departing from the scope of the claims that follow the detailed description.

[0025] Within the scope of this application it should be understood that the various aspects, embodiments, examples and alternatives set out herein, and individual features thereof may be taken independently or in any possible and compatible combination. Where features are described with reference to a single aspect or embodiment, it should be understood that such features are applicable to all aspects and embodiments unless otherwise stated or where such features are incompatible.BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The present disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the disclosure. Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0027] FIG. 1 illustrates an example network of combine harvesters that communicate with a computing system through a communication network, in accordance with some embodiments of the present disclosure;

[0028] FIG. 2 illustrates is a block diagram of example aspects of a computing system (such as one of the computing systems shown in FIG. 1), in accordance with some embodiments of the present disclosure;

[0029] FIG. 3 illustrates a schematic side view of a combine harvester (such as one of the combine harvesters shown in FIG. 1) with some portions of the harvester being broken away to reveal internal details of construction, in accordance with some embodiments of the present disclosure;

[0030] FIG. 4 illustrates a perspective view of the combine harvester shown in FIG. 3 along with a field of view of a camera of the harvester, in accordance with some embodiments of the present disclosure;

[0031] FIGS. 5 and 6 illustrate methods in accordance with some embodiments of the present disclosure;

[0032] FIG. 7 illustrates an example RGB image of a field derived from image data captured by a camera of a harvester as well as a data processing scheme that receives input derived from the RGB image and processes the input to output data for a prediction view of the image showing determined attributes of the field (wherein the attributes shown include lodging areas), in accordance with some embodiments of the present disclosure;

[0033] FIG. 8 illustrates another example RGB image of a field derived from image data captured by a camera of a harvester as well as the data processing scheme shown in FIG. 7 processes input derived from the RGB image to output data for a prediction view of the image showing determined attributes of the field (wherein the attributes shown do not include lodging areas), in accordance with some embodiments of the present disclosure; and

[0034] FIG. 9 illustrates a display of a user interface device displaying an RGB video and an attribute-predictions video running next to each other (wherein the attribute-predictions video includes lodging areas), in accordance with some embodiments of the present disclosure.

[0035] FIG. 10 illustrates graphical representations of crop lodging areas including graphical indicators of a lodging pattern.

[0036] FIG. 11 illustrates a display of a user interface device displaying a graphical user interface (GUI) of a crop field and an agricultural machine moving through the field, the GUI being generated based on determined attributes of the field (including determined crop lodging) and the GUI also showing areas of the field affected and unaffected by weeds and lodging as the areas of the field approach the header of the machine, in accordance with some embodiments of the present disclosure;

[0037] FIG. 12 illustrates a display of the user interface device displaying a GUI of a crop field and an agricultural machine moving through the field, the GUI being generated based on determined attributes of the field (including determined crop lodging) and the GUI including separate windowsshowing areas of the field affected and unaffected by lodging as the areas of the field approach the machine, in accordance with some embodiments of the present disclosure; and

[0038] FIGS. 13 to 15 illustrate respective visualizations of a crop field and an agricultural machine moving through the field, in which each one of the visualizations can be displayed in any one of the GUIs mentioned herein, and the visualizations being generated based on determined attributes of the field (including determined crop lodging), in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0039] Details of example embodiments of the invention are described in the following detailed description with reference to the drawings. Although the detailed description provides reference to example embodiments, it is to be understood that the invention disclosed herein is not limited to such example embodiments. But to the contrary, the invention disclosed herein includes numerous alternatives, modifications and equivalents as will become apparent from consideration of the following detailed description and other parts of this disclosure.

[0040] Described herein are technologies for detecting crop lodging (e.g., see crop loading areas 710 shown in FIG. 7) in a crop field, such as for detecting lodging while harvesting a crop to improve upon some technical problems in the field. The techniques disclosed herein provide specific technical solutions to at least overcome the technical problems mentioned in the background section or other parts of the application as well as other technical problems not described herein but recognized by those skilled in the art. As mentioned in the background section, reducing lodging may begin with shorter crops; however, precision agriculture or farming may independently improve lodging in crop fields or at least provide improvement to known tactics. Lodging can include the displacement of the plant roots or stems from vertical positions, making such plants difficult to harvest. Reducing lodging or at least tracking it can possibly increase yields as well as lessen lodging in future crops. With the techniques described herein, precision agriculture or farming can be improved.

[0041] FIG. 1 illustrates a network 100 of farming machines that communicate with remote computing systems through a communications network 104. Specifically, FIG. 1 illustrates the network 100 including a remote computing system 102, the communications network 104, farmingmachines (e.g., see farming machines 106, 108, and 110). The remote computing system (e.g., see remote computing system 102) is remote in that it is physically and geographically separated from the farming machines of the network 100. The farming machines are shown communicating with the remote computing system 102 of the network 100 through a communications network 104. As shown in FIG. 1, the farming machines of the network 100 can each include its own computing system including electronics such as connected sensors, cameras, busses, and computers (e.g., see computing systems 116, 118, and 120, and see electronics 126, 128, and 130). A computing system of a farming machine can include a processor, memory, a communication interface and one or more sensors that can make the farming machines individual computing devices. In the case of the communications network 104 including the Internet, the farming machines of the network 100 are considered Internet of Things (loT) devices. Also, in some embodiments, the remote computing system 102 is a part of a cloud computing system.

[0042] As shown in FIG. 1 , each one of the farming machines includes a respective computer and respective electronics (e.g., see computing systems 116, 118, and 120, and see electronics 126, 128, and 130). In some embodiments, the electronics of a farming machine include electronic hardware and software of the farming machine such as sensors and other types of electrical and / or mechanical feedback devices that are communicatively coupled with the computer of the farming machine (e.g., see camera 302 of combine harvester 300 shown in FIG. 3). And, in some embodiments, the computer of the farming machine is configured to connect or integrate with the farming machine electronics (e.g., see electronics 126, 128, and 130) and communicate with a remote computing system (such as the remote computing system 102) via the communications network 104.

[0043] In some embodiments, the farming machine (e.g., see farming machine 106, 108, or 110) includes a vehicle. In some embodiments, the farming machine is a combine harvester (e.g., see combine harvester 300 show in FIGS. 3 and 4). In some embodiments, the farming machine is a tractor. In some embodiments, the farming machine is a planter. In some embodiments, the farming machine is a sprayer. In some embodiments, the farming machine is a baler. In some embodiments, the farming machine is or includes a harvester, a planter, a sprayer, a baler, any other type of farming implement, or any combination thereof. In some of such embodiments, the farming machine can be or include a vehicle in that is self-propelling. Also, in some embodiments, the group of similarfarming machines is a group of vehicles (e.g., see farming machines 106, 108, and 110). In some embodiments, the group of vehicles is a group of combine harvesters. And, in some embodiments, the group of vehicles is a group of combine harvesters, planters, sprayers, balers, another type of implement, or any combination thereof.

[0044] The communications network 104 includes one or more local area networks (LAN(s)) and / or one or more wide area networks (WAN(s)). In some embodiments, the communications network 104 includes the Internet and / or any other type of interconnected communications network. The communications network 104 can also include a single computer network or a telecommunications network. More specifically, in some embodiments, the communications network 104 includes a local area network (LAN) such as a private computer network that connects computers in small physical areas, a wide area network (WAN) to connect computers located in different geographical locations, and / or a middle area network (MAN) to connect computers in a geographic area larger than that covered by a large LAN but smaller than the area covered by a WAN.

[0045] At least each shown component of the network 100 (including remote computing system 102, communications network 104, and farming machines 106, 108, and 110) can be or include a computing system which includes memory that includes media. The media includes or is volatile memory components, non-volatile memory components, or a combination of thereof. In general, in some embodiments, each of the computing systems includes a host system that uses memory. For example, the host system writes data to the memory and read data from the memory. The host system is a computing device that includes a memory and a data processing device. The host system includes or is coupled to the memory so that the host system reads data from or writes data to the memory. The host system is coupled to the memory via a physical host interface. The physical host interface provides an interface for passing control, address, data, and other signals between the memory and the host system.

[0046] FIG. 2 illustrates is a block diagram of example aspects of a computing system 200. In some embodiments, the computing system 200 is a part of a farming machine or any one of the agricultural machines described herein (e.g., see harvester 300 depicted in FIG. 3). In some embodiments, the computing system 200 is local to the farming machine (e.g., see computing systems 116, 118, and 120 shown in FIG. 1). In some alternative embodiments, the computingsystem can include a remote computing system (e.g., see remote computing system 102), wherein the remote computing system can be the sole computing system of the farming machine in some examples or in combination with a local computing system of the farming machine in some other examples.

[0047] FIG. 2 illustrates parts of the computing system 200 within which a set of instructions, for causing a machine (such as a computer processor or processing device 202) to perform any one or more of the methodologies discussed herein performed by a computing system, are executed (e.g., see the method steps of method 500 shown in FIG. 5). In some embodiments, the computing system 200 operates with additional computing systems (e.g., such as additional remote computing systems) to provide increased computing capacity in which multiple computing systems operate together to perform any one or more of the methodologies discussed herein that are performed by a computing system. In some embodiments, the computing system 200 corresponds to a host system that includes, is coupled to, or utilizes memory or is used to perform the operations performed by any one of the computing systems described herein. In some embodiments, the machine is connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. In some embodiments, the machine operates in the capacity of a server in client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server in a cloud computing infrastructure or environment. In some embodiments, the machine is a personal computer (PC), a tablet PC, a cellular telephone, a web appliance, a server, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term “machine” shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein performed by computing systems.

[0048] The computing system 200 includes a processing device 202, a main memory 204 (e.g., read-only memory (ROM), flash memory, dynamic random-access memory (DRAM), etc.), a static memory 206 (e.g., flash memory, static random-access memory (SRAM), etc.), and a data storage system 210, which communicate with each other via a bus 218. The processing device 202 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, or the like. More particularly, the processing device can include a microprocessoror a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Or, the processing device 202 is one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 202 is configured to execute instructions 214 for performing the operations discussed herein performed by a computing system. In some embodiments, the computing system 200 includes a network interface device 208 to communicate over the communications network 104 shown in FIG. 1.

[0049] The data storage system 210 includes a machine-readable storage medium 212 (also known as a computer-readable medium) on which is stored one or more sets of instructions 214 or software embodying any one or more of the methodologies or functions described herein performed by a computing system. The instructions 214 also reside, completely or at least partially, within the main memory 204 or within the processing device 202 during execution thereof by the computing system 200, the main memory 204 and the processing device 202 also constituting machine-readable storage media.

[0050] In some embodiments, the instructions 214 include specific instructions to implement functionality described herein related to the methods described herein and that can correspond to any one of the computing devices, data processors, user interface devices, and VO devices described herein related to a computing system. For example, the instructions 214 include crop lodging detection instructions 222 (which include instructions configured to determine attributes of a crop field and the crops with in the field), data linking and recording instructions 224 (which includes instructions configured to record and associate determined attributes of field and its crops with other parameters such as geographic location of the agricultural machine where the input data for determining the attributes was captured), data enhancement instructions 226 (which includes instructions for enhanced attribute determination such as instructions for a computing scheme, e.g., ANN, CNN, etc. specifically adapted for determining and identifying lodging captured in images), graphical lodging identification instructions 228 (which can provide graphics for showing an operator instances of lodging and other attributes in a video or an image), and controller instructions 230 (which are configured to control parts and operations of an agricultural according to input and feedback such as including the determined attributes of the field and crop). In some embodiments, the data enhancement instructions include different types of data analysis libraries as well differenttypes of data processing libraries— including various mathematical and statistical modeling and operations libraries and machine learning, artificial intelligence, and deep learning libraries as well as specific libraries for ANN and CNN data processing and for training ANNs, CNNs and other types of computing schemes or systems.

[0051] While the machine-readable storage medium 212 is shown in an example embodiment to be a single medium, the term “machine-readable storage medium” should be taken to include a single medium or multiple media that store the one or more sets of instructions. The term “machine- readable storage medium” shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure performed a computing system. The term “machine-readable storage medium” shall accordingly be taken to include solid-state memories, optical media, or magnetic media.

[0052] Also, as shown, the computing system 200 includes user interface 216 that includes a display, in some embodiments, and, for example, implements functionality corresponding to any one of the user interface devices disclosed herein. A user interface, such as user interface 216, or a user interface device described herein includes any space or equipment where interactions between humans and machines occur. A user interface described herein allows operation and control of the machine from a human user, while the machine simultaneously provides feedback information to the user. Examples of a user interface (UI), or user interface device include the interactive aspects of computer operating systems (such as graphical user interfaces), machinery operator controls, and process controls. A UI described herein includes one or more layers, including a human-machine interface (HMI) that interfaces machines with physical input hardware and output hardware.

[0053] Also, as shown, the computing system 200 includes farming machine electronics 220 that includes one or more sensors, cameras, other types of electrical or mechanical feedback devices, user interfaces (e.g., any one of the UI described herein), and any other type of computer hardware and software configured to interface and communicatively couple to operational components of a farming machine (e.g., see electronics 126, 128, and 130 shown in FIG. 1). The electronics 220 includes farming-machine controller hardware that can operate in combination with controller instructions 230, in some embodiments. Also, in some embodiments, the farming machine electronics 220 as well as the electronics 126, 128, and 130 include any one of the cameras describedherein for capturing images of a field and crop within the field (e.g., see camera 302 show in FIG. 3).

[0054] In some systems of the technologies disclosed herein, any steps of embodiments of the methods described herein are implementable by executing instructions corresponding to the steps, which are stored in memory (e.g., see instructions 214, 222, 224, 226, 228 and 230 shown in FIG. 2).

[0055] FIG. 3 illustrates a schematic side view of a combine harvester 300 (which can be one of the farming machines shown in FIG. 1) with some portions of the harvester being broken away to reveal internal details of construction. As shown in FIG. 3, the combine harvester 300 includes at least one camera. The at least one camera includes a camera 302 mounted to the harvester 300 at the front of the harvester. The camera 302 is configured to capture images of crop and its surrounding while the crop is being harvested or just before the crop is harvested (e.g., see image 702 shown in FIG. 7). In some embodiments, the at least one camera includes a second camera (not depicted). In embodiments where there are multiple cameras of the harvester or an agricultural machine (in general), the capturing of the images described herein can include multiple cameras capturing images from different positions and thus capturing different perspectives of the field and the crop such that different perspectives of crop lodging areas are captured and more information is provided for the determining of attributes of the crop and the field. For example, in some cases, multiple cameras are mounted around the machine, and the determinations from the images of the attributes are based on combining images taken from different perspectives into a single representation of the crop and field around the machine. Also, in some cases, the capturing of the images can be performed by a stereo pair of cameras mounted to the machine that can capture depth information.

[0056] Also, it should be understood that in some embodiments, the camera 302 can be an RGB camera or a grayscale camera, a CMYK camera, or any other type of camera having a different color mode than RGB, grayscale or CMYK. And, the captured images of such a camera can be processed by the computing system using similar techniques as those used for the RGB images to detect or determine lodging. It should also be understood that in some embodiments, the camera 302 can be replaced with LIDAR or radar or another way of capturing data points of the field in front of the farming machine. And, the data points captured by such a sensor can be converted into the imagefor processing to detect or determine lodging using similar techniques as those used for the RGB images or other types of camera captured images.

[0057] The combine harvester 300 has processing system 312 that extends generally parallel with the path of travel of the harvester. It is to be understood that such a harvester is being used to illustrate principals herein and the subject matter described herein is not limited to harvesters with processing systems designed for axial flow, nor to axial flow harvesters having only a single processing system. The combine harvester 300 also includes a harvesting header 314 at the front of the machine that delivers collected crop materials to a feeder house 315. Such materials are moved upwardly and rearwardly within feeder house 315 by a conveyer 316 until reaching a beater 318 that rotates about a transverse axis. Beater 318 feeds the material upwardly and rearwardly to a rotary processing device, in the illustrated instance to a rotor 322 having an infeed auger 320 on the front end thereof. Infeed auger 320, in turn, advances the materials axially into the processing system 312 for threshing and separating. The processing system 312 is housed by processing system housing 313. In other types of systems, conveyer 316 may deliver the crop directly to a threshing cylinder. The crop materials entering processing system 312 can move axially and helically therethrough during threshing and separating. During such travel, the crop materials are threshed and separated by rotor 322 operating in chamber 323 which concentrically receives the rotor 322. The lower part of the chamber 323 contains concave assembly 324 and a separator grate assembly 326. Rotation of the rotor 322 impels the crop material rearwardly in a generally helical direction about the rotor 322. A plurality of rasp bars and separator bars (not shown) mounted on the cylindrical surface of the rotor 322 cooperate with the concave assembly 324 and separator grate assembly 326 to thresh and separate the crop material, with the grain escaping laterally through concave assembly 324 and separator grate assembly 326 into cleaning mechanism 328. Bulkier stalk and leaf materials are retained by the concave assembly 324 and the separator grate assembly 326 and are impelled out the rear of processing system 312 and ultimately out of the rear of the combine harvester 300. Such parts and operations of the harvester 300 as well as parts and operations of a harvester can be controlled by a control system of such a harvester (e.g., see controller instructions 230 and electronics 220 shown in FIG. 2) according to at least the determined attributes of the field and the crops of the field described herein (e.g., see the determined attributes 708 and 908 shown in FIGS.7 to 9 as well as the determination of the attributes at step 504 of the method 500 depicted in FIG. 5).

[0058] FIG. 4 illustrates a perspective view of the combine harvester 300 shown in FIG. 3 along with a field of view 401 of a camera of the harvester, such as camera 302 shown in FIG. 3. As shown in FIG 4, a sensor platform 400 is attached to the roof 303 of the cabin 301 of the harvester 300. The sensor platform 400 includes the camera 302 (shown in FIG. 3) and can include additional sensors such as a second optical sensor (such as a second camera in a stereo pair of cameras), an ultrasonic transducer, a radio frequency (RF) sensor, lidar, radar, or any type of sensor that senses structure in a crop field without contacting the crop field, or some combination thereof. One or more of the additional sensors can capture data that can be used by a computing system (such as the computing system of the harvester) to verify the images or the data captured by the camera 302. Also shown, the sensor platform 400 is mounted to a front and top portion of the harvester 300 and the platform is mounted such that a field of view 401 is not limited by the header 314 attached to the front of the harvester. In some embodiments, the camera is directly mounted to the cabin 301, and, in other examples, the sensor platform 400 is directly mounted to the cabin. And, in some instances, the camera is directly mounted to the roof 303 of the cabin, and in other examples, the sensor platform is directly mounted to the hood. In some embodiments, the camera includes a wide field of view (e.g., see field of view 401). And, in some instances, the camera includes a wide-angle horizontal field of view (e.g., see field of view 401 which is an example of a wide-angle horizontal field of view). In some embodiments the wide angle 402 of the field of view is at least sixty degrees. In some instances the wide angle 402 can be as much as 360 degrees. In such cases, additional cameras are used and the captured data from the cameras is used as input to generate a combined image of the 360-degrees wide angle.

[0059] FIGS. 5 and 6 illustrate methods in accordance with some embodiments of the present disclosure. Method 500 is performed by any one of the cameras and any one of the computing systems described herein (e.g., see computing system 200 and electronics 220 as well as camera 302). The method shown in FIG. 6 is a sub-method of the method step 504, which is a method step included in method 500.

[0060] As shown in FIG. 5, method 500 begins with step 502, which includes capturing images of a field and a crop in the field (e.g., see images 702 and 802 shown in FIGS. 7 and 8 respectively)by a camera (e.g., see camera 302 shown in FIG. 3) mounted to an agricultural machine (e.g., see harvester 300 or any one of the agricultural machines depicted in FIG. 1) moving through the field. In some cases, the capturing of the images can be performed by a stereo pair of cameras mounted to the machine. At step 504, the method 500 continues with determining, by a computing system (e.g., see computing system 200 shown in FIG. 2) communicatively coupled to the camera, attributes (e.g., see attributes 708 shown in FIG. 7) of the crop and the field in the images. The determined attributes include crop lodging areas (e.g., see crop lodging areas 710 depicted in FIG. 7). And, such a determination of attributes of the crop and field and other determinations of aspects of a crop and field described herein can be performed by executing instructions 222. Also, in some embodiments, it is to be understood that the determinations described herein can include assigning geolocation to each pixel or groups of pixels in an image.

[0061] In some embodiments of the method 500, the computing system is part of the agricultural machine to limit delay in the determining of the attributes. Delay is limited because the determination is performed locally at the agricultural machine. In some embodiments of the method 500, the images include RGB images. In some embodiments of the method 500, the agricultural machine is a combine harvester or another type of harvester. In some embodiments, the images are of images of the crop and the field in front of the machine. In some embodiments of the method 500, the capturing of the images and the determining of the attributes occur continuously as the agricultural machine moves through a field. And, in some embodiments of the method 500, the captured images include RGB images.

[0062] Also, it should be understood that in some embodiments of the method 500, the camera can capture grayscale images, CMYK images, or any other type of color mode capturable by a camera (depending on the embodiment). And, the captured images of such a camera can be processed by the computing system using similar techniques as those used for RGB images to detect or determine lodging. It should also be understood that in some embodiments, the camera of the method 500 can be replaced with LIDAR or radar or another way of capturing data points of the field in front of the farming machine. And, the data points captured by such a sensor can be converted into an image for processing to detect or determine lodging using similar techniques as those used for the RGB images or other types of camera captured images.

[0063] In some embodiments, the camera is mounted to a front and top portion of the agricultural machine (e.g., see FIGS. 3 and 4). In some embodiments, the camera is mounted such that a field of view (e.g., see field of view 401 shown in FIG. 4) is not limited by an implement (e.g., see feeder 315 shown in FIG. 3) attached to the front of the agricultural machine. In some embodiments, the camera is mounted to the cabin (e.g., see cabin 301) of the machine. And, in some examples, the camera is mounted to the hood (e.g., see hood 303) of the cabin. In some embodiments, the camera includes a wide field of view (e.g., see field of view 401). And, in some instances, the camera includes a wide-angle horizontal field of view (e.g., see field of view 401).

[0064] Also shown, the method 500 includes, at step 506, communicating, by the computing system, the determined attributes over a network (e.g., see bus 218 shown in FIG. 2 or network 104 shown in FIGS. 1 and 2) to a user interface device (e.g., see user interface 216 depicted in FIG. 2). The method 500 also includes, at step 508, displaying at least the crop lodging areas by a display of the user interface device (e.g., see prediction view 706 shown in FIG. 7). In some embodiments of the method 500, the displayed lodging areas include a top-view of a lodging pattern (e.g., see prediction view 706 shown in FIG. 7). In some embodiments of the method 500, the capturing of the images, the determining and communicating of the attributes, and the displaying of the crop lodging areas occur continuously as the agricultural machine moves through a field. In some embodiments of the method 500, the displaying of the crop lodging areas includes displaying an RGB video of the crop lodging areas, and the crop lodging areas are identified in the video (e.g., see FIG. 9). In some embodiments of the method 500, the identification of the crop lodging areas in the video include graphical identifications such as a solid color in the video representing lodging (e.g., see FIG. 9). With method 500 and other methods described herein, other attributes (e.g., see attributes 708 show in FIG. 7) are identified beyond crop lodging areas, including the sky, weeds areas, unharvested areas, harvested areas, miscellaneous areas, and unidentified or unlabeled areas (e.g., see prediction views 706 and 806 shown in FIGS. 7 and 8). And, the other identified attributes are represented by corresponding graphical identifications such as a respective solid color in the video for each of the other identified attributes (e.g., see prediction views 706 and 806). Also, with method 500 and other methods described herein, the display can be for the machine operator and it can show the RGB image with highlighted lodging areas of the field and issue a warning signal so that the operation can be alert in such situations. E.g., wherein, for an RGB input image, thedetermination can provide an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background' (this can be broken down into more specific classes such as 'weeds', 'harvested area', 'swath', etc.) (e.g., see prediction views 706 and 806 as well as FIG. 9 showing an image of an RGB video 902 and attribute-predictions video 906 running adjacent to each other, in which the videos show that same area of the crop field simultaneously while the machine moves through the field. Also, shown in the attributes of the field 908 in a legend as well as on the attribute-predictions video 906. And, as shown in FIG. 9, the attribute-predictions video 906 is overlapping a second instance of the RGB video. Such an example features, can also be a part of the display of the attributes of method 500 or other methods described herein. To put in another way, in some embodiments of the method 500, the displaying the RGB video of the crop lodging areas includes displaying the RGB video along with a second video of a corresponding area of the crop and the field that is displayed without the identification of the crop lodging areas (e.g., see videos 902 and 906 shown in FIG. 9 showing a video identifying attributes and the other video only including the RGB images of the video). In some embodiments of the method 500, the RGB video overlaps the second video and both of the videos are displayed to at least some extent (e.g., see video 906).

[0065] In some embodiments, the implementation of rendering and displaying of images and videos of a crop field and lodging can be performed at least in part by executing instructions 228.

[0066] Also shown, the method 500 includes, at step 510, communicating, by the computing system, the determined attributes over a network (e.g., see bus 218 shown in FIG. 2 or network 104 shown in FIGS. 1 and 2) to a controller (e.g., see electronics 220 and controller instructions 230 depicted in FIG. 2) of the agricultural machine. The method 500 also includes, at step 512, controlling, by the controller, one or more operations of the agricultural machine according to the determined attributes. In some embodiments of method 500, the capturing of the images, the determining and communicating of the attributes, and the controlling of the operation of the agricultural machine occur continuously as the agricultural machine moves through a field.

[0067] In some embodiments of the method 500, the determination of the attributes of the field and the crop in the field in the images (e.g., see images 702 and 802) is based at least on digital signal processing (e.g., see step 602 shown in FIG. 6). Also, in some embodiments, the determination of the attributes is based at least on a computer vision analysis (e.g., see steps 604 to 608 shown in FIG. 6 as well as data enhancement instructions 226 shown in FIG. 2). Further, insome of such examples, the determination of characteristics of the crop in the images is further based on digital signal processing in addition to the computer vision analysis (e.g., see all the steps of FIG. 6 as well as instructions 226). As shown in FIG. 6, in some examples, the digital signal processing (such as at step 602) occurs prior to the computer vision analysis as a pre-processing step to generate enhanced input for the computer vision analysis (such as the analysis at step 606 by computing scheme 607). In some embodiments, as shown in FIG. 6, the method step 504 includes, at step 606, processing, by the computing scheme 607, the determined attributes of step 504 or the enhance input of step 602. For example, at step 504, as shown in FIG. 6, further determining attributes of the field and the crop in the images is based on digital signal processing at step 602. In some embodiments, the method includes inputting the enhanced input from the signal processing into an artificial neural network (e.g., see step 604) and the computer vision analysis includes inputting the enhanced input into an ANN (e.g., see step 604 as well as instructions 226). And, the determination of the attributes of the crop and the field is based at least on the output of the ANN (e.g., see step 608). In some cases, as shown in FIG. 6, at step 608, the step 504 continues with using an output of the computing scheme 607 or a derivative thereof to further determine the attributes of the field and the crop in the images. In some examples, the ANN includes or is part of a deep learning process that detects the different aspects of the crop or is a basis for the detection of the different aspects of the crop (e.g., see instructions 226). Also, in some examples, the deep learning process includes a convolutional neural network (CNN) (e.g., see instructions 226). Also, in some examples, the deep learning process includes a network of convolutional neural networks (CNNs) (e.g., see instructions 226).

[0068] In some embodiments, the computer vision analysis includes inputting aspects of the images or derivatives of aspects of the images into an artificial neural network (ANN) (e.g., see scheme 607), and the determination of the attributes of the crop and the field in the images is based at least on output of the ANN. In some examples, the ANN includes or is part of a deep learning process that determines attributes of the crop and the field or is a basis for the determination of the attributes of the crop and the field. Also, in some examples, the deep learning process includes a convolutional neural network (CNN). And, the deep learning process can include a network of convolutional neural networks (CNNs) and the CNN is one of multiple CNNs of the network.

[0069] Also, the ANN can include or be a part of a deep learning process that determines attributes of the crop and the field or is a basis for the determination of the attributes of the crop andthe field, and the deep learning process includes a convolutional neural network (CNN) for semantic pixel-wise segmentation (e.g., see scheme 704 shown in FIG. 7). In some examples, the use of pixel-wise semantic segmentation includes the use of SEGNET or another type of a deep convolutional encoder-decoder architecture for image segmentation such as U-NET (which is a network for biomedical image segmentation), fully convolutional network (FCN), DEEPLABV2 or DEEPLABV3, and DECONVNET (e.g., see scheme 704). In some examples, for an RGB input image (e.g., see images 702 and 802 of FIGS. 7 and 8 respectively), the determination provides an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background' (this can be broken down into more specific classes such as 'weeds', 'harvested area', 'swath', etc.) (e.g., see prediction views 706 and 806 shown in FIGS. 7 and 8 respectively). Also, in some case, the ANN includes or is part of a deep learning process that determines attributes of the crop and the field or is a basis for the determination of the attributes of the crop and the field, and the deep learning process includes a convolutional neural network (CNN) for crop lodging classification.

[0070] In some embodiments, the implementation of the aforesaid computing schemes or computer vision analysis can be performed by executing instructions 226.

[0071] In some embodiments of the method 500, values of crop lodging are recorded by the computing system as the machine moves through the field (not depicted). In such embodiments, such values can be compared against satellite images of crop lodging to validate the values.

[0072] In some embodiments, a system or method to detect lodging areas in an agricultural field is implemented through an agricultural machine or vehicle (e.g., a combine harvester) equipped with a machine location tracking system (e.g., one of many known real-time kinematic RTK or GPS systems, or a combination thereof) as well as equipped with a camera configured to capture images of an area in front of the machine (e.g., an RGB camera, greyscale camera, CMYK camera, etc.). Also, to implement the system or method a computing system is provided locally with the machine or remotely to process data captured by the camera and possibly from additional sensors to detect lodging and used the detected lodging to provide additional functionality such as a visual interface for an operator or input for a control system of the machine.

[0073] In some embodiments of the method 500, the machine or agricultural vehicle includes an RGB camera mounted in the front of the vehicle with a wide-angle horizontal field of view (e.g.,see camera 302 shown in FIG. 3 as well as view 401 depicted in FIG. 4). The possible mounting points can include the top of a cabin or on the hood of the cabin (e.g., see cabin 301 and hood 303 shown in FIGS. 3 and 4). The camera can capture the images of the crop in front of the vehicle or machine during operation. The images can be processed by a computing system (e.g., see any one of the computing systems shown in FIG. 1 as well as computing system 200 shown in FIG. 2). The processing can occur through a deep learning process for pixel-wise semantic segmentation. In some instances, SEGNET or another type of a deep convolutional encoder-decoder architecture for image segmentation is used in or as the deep-learning process (e.g., see scheme 704 shown in FIGS. 7 and 8). For an RGB input image (e.g., see images 702 and 802 shown in FIGS. 7 and 8 respectively), the deep-learning process can provide an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background', for example. Based on the known field of view of the camera, its mounting position and rotation with respect to the vehicle, the detected areas can be transformed from camera frame to vehicle frame. Since the vehicle can be equipped with a location tracking system or a machine position tracking system (such as RTK-GPS), the data from the vehicle frame can then be transformed into input for a vehicle control system or other types of functionality such as map generation.

[0074] FIG. 7 illustrates an example RGB image 702 of a crop field derived from image data captured by a camera of a harvester (such as camera 302) as well as a data processing scheme 704 that receives input derived from the RGB image 702 and processes the input to output data for a prediction view 706 of the image showing determined attributes 708 of the field (wherein the attributes shown include lodging areas 710). Referring back to method 500, the images captured at step 502 can include RGB images such as image 702. And, the determined attributes from step 504 can include attributes such as the attributes 708 shown in FIG. 7. Also, as shown in FIG. 7, the attributes can include crop lodging areas 710, the sky, weeds, unharvested crop, harvested crop, other attributes of a field or its surroundings and unlabeled or undetermined items of the images.

[0075] FIG. 8 illustrates another example RGB image 802 of a field derived from image data captured by a camera of a harvester as well as the data processing scheme 704 shown in FIG. 7 that receives and processes input derived from the RGB image 802 to output data for a prediction view 806 of the image showing determined attributes 708 of the field (wherein the attributes shown do not include lodging areas).

[0076] FIG. 9 illustrates a display of a user interface device displaying an RGB video 902 and an attribute-predictions video 906 running next to each other (wherein the attribute-predictions video is show including lodging areas 910). Referring back to method 500 and with respect to FIG. 9, the displaying of the crop lodging areas 910 includes displaying an RGB video 902 of the crop lodging areas, and the crop lodging areas are identified in the video (e.g., see video 906). As mentioned in some embodiments of the method 500, the identification of the crop lodging areas in the video include graphical identifications such as a solid color in the video representing lodging (e.g., see video 906). Also, with method 500 and other methods described herein, other attributes (e.g., see attributes 908) are identified beyond crop lodging areas, including the sky, weeds areas, unharvested areas, harvested areas, etc. And, the other identified attributes are represented by corresponding graphical identifications such as a respective solid color in the video for each of the other identified attributes. Also, as depicted by the FIG. 9 and with respect to some examples of method 500 and other methods described herein, an image of an RGB video 902 and attributepredictions video 906 running adjacent to each other is shown in which the videos show that same area of the crop field simultaneously while the machine moves through the field. Also, shown are the attributes 908 of the field in a legend as well as on the attribute-predictions video 906. And, as shown in FIG. 9, the attribute-predictions video 906 is overlapping a second instance of the RGB video 902. To put in another way, in some embodiments, the displaying the RGB video of the crop lodging areas includes displaying the RGB video along with a second video of a corresponding area of the crop and the field that is displayed without the identification of the crop lodging areas (e.g., see videos 902 and 906 showing a video identifying attributes and the other video only including the RGB images of the video). Also, the RGB video overlaps or is beneath the second video and both of the videos are displayed to at least some extent (e.g., see video 906).

[0077] FIG. 11 illustrates a display 1202 of a user interface device 1200 (e.g., see user interface 216 shown in FIG. 2) displaying a GUI 1203 that displays a visualization of a harvester 1204 as it moves through a crop field. The GUI 1203 is configured to display a crop field surrounding the harvester in real time as the harvester moves through the field. The GUI 1203 is generated based on determined attributes of the field (including determined crop lodging, weeds, and normal standing crops) and the GUI is also configured to show areas of the field affected and unaffected by weeds and lodging as the areas of the field approach the header 1208 of the harvester 1204. Forexample, the area 1206a is shown as an area of the field affected by lodging and approaching the header 1208. Also, the area 1206b is depicted as an area of the field affected by weeds and lodging and approaching the header 1208. Whereas, the area 1206c is shown as an area of the field approaching the header 1208 and as having normal standing crops that are not affected by lodging or weeds. The areas that are labeled that are approaching the header 1208 (e.g., areas 1206a, 1206b, and 1206c) are categorized and labeled according to the determination of the attributes of the field (such as according to step 904 of method 900).

[0078] Also, as shown in FIG. 12, the GUI 1203 includes a top-down view of a graphical representation of a harvester with grid-cell graphical indications of determined attributes of a crop field (e.g., areas 1206a, 1206b, and 1206c) as well as a graphical representation of a header of the harvester (e.g., see header 1208) and a plurality of inputs of the header (e.g., see inputs 1210a, 1210b, and 1210c). And, the GUI 1203 also shows grid-cell graphical indications of the determined attributes appearing in-line with the graphical representations of the plurality of inputs of the header (e.g., areas 1206a, 1206b, and 1206c). Also, as shown in FIG. 12, the top-down view of the graphical representation of the harvester includes a top-down view of the entirety of the machine as well as a nearby portion of the crop field encompassing the machine (e.g., see the harvester 1204 and the surround area of the field 1212 surrounding the harvester). And, as shown in FIG. 12, the user interface device 1200 is configured to provide a visual warning when a computing system determines crop lodging or weeds or some other determined attribute of the field that is not a selected crop or a standing crop is in-line with at least one input of the graphical representations of the plurality of inputs of the header (e.g., see the warnings of lodging or weeds in areas 1206a and 1206b). Not depicted, the UI device 1200 is also configured to provide an audio warning when a computing system determines crop lodging or weeds or some other determined attribute of the field that is not a selected crop or a standing crop is in-line with at least one input of the graphical representations of the plurality of inputs of the header.

[0079] FIG. 12 illustrates the display 1202 of the user interface device 1200 displaying a GUI 1303 of a crop field and a harvester 1302 moving through the field, the GUI being generated based on determined attributes of the field (including determined crop lodging) and the GUI including separate windows 1310a, 1310b, and 1310c showing areas of the field affected and unaffected by lodging as the areas of the field approach the harvester. The GUI 1303 displays a visualization of aharvester 1302 as it moves through a crop field in window 1310a. The GUI 1303 is configured to display a crop field surrounding the harvester in real time as the harvester moves through the field as shown by windows 1310a, 1310b, and 1310c. The GUI 1303 is generated based on determined attributes of the field (including determined crop lodging, weeds, and normal standing crops) and the GUI is also configured to show areas of the field affected and unaffected by weeds and lodging as the areas of the field approach the header 1304 of the harvester 1302 (e.g., see areas 1306a and 1306b which are affected by lodging and areas 1307a and 1307b which are unaffected). For example, in window 1310a, the area 1306a is shown as an area of the field affected by lodging and approaching the header 1304. Also, for example, in window 1310b, the area 1306b is shown as an area of the field affected by lodging and approaching the header 1304, even though the header is now shown in window 1310b. Whereas, the areas 1307a and 1307b are shown as an area of the field approaching the header 1304 and as having normal standing crops that are not affected by lodging or weeds. The areas that are labeled that are approaching the header 1304 (e.g., see areas 1306a, 1306b, 1307a, and 1307b) are categorized according to the determination of the attributes of the field (such as according to step 904 of method 900). As it is apparent from FIG. 13, areas 1306a and 1306b represent a same part of the field, in which 1306a is a top-down 2D view of the part and in which 1306b is a top-down perspective 3D view of the part. And, as it is apparent from FIG. 13, areas 1307a and 1307b represent a same part of the field, in which 1307a is a top-down 2D view of the part and in which 1307b is a top-down perspective 3D view of the part.

[0080] Also, as shown in FIG. 12, the GUI 1303 includes 2D and 3D views of parts of the field approaching the harvester in a graphical representation with grid-cell graphical indications of determined attributes of a crop field (e.g., see areas 1306a, 1306b, 1307a and 1307b). The window 1310a provides the 2D view. And, the window 1310b provides the 3D view. Also, the GUI 1303 includes the window 1310c, which provides a video of at least some of the same parts of the crop field as shown by the windows 1310a or 1310b. Also, the window 1310a shows a graphical representation of a header of the harvester (e.g., see header 1304) and a plurality of inputs of the header (e.g., see input 1312a). And, the window 1310a of GUI 1303 also shows grid-cell graphical indications of the determined attributes appearing in-line with the graphical representations of the plurality of inputs of the header (e.g., see areas 1306a). Also, as shown in FIG. 12, the top-down view of the graphical representation of the harvester includes a top-down view of the entirety of themachine as well as a nearby portion of the crop field encompassing the machine (e.g., see the harvester 1302 and the surround area of the field 1308 surrounding the harvester). And, as shown in FIG. 12, the grid-cell graphical indications of the determined attributes are color-coded to represent different types of attributes of a crop field such as crop type, weeds, standing crop, and crop lodging. And, not depicted, but a part of the functionality of GUI 1303, the views in window 1310a and 1310b are zoomable and configured to be zoomed- in on by the computing system to focus on the plurality of inputs of the header and at least some of the grid-cell graphical indications of the determined attributes that appear in-line with graphical representations of the plurality of inputs within a selected distance of the header. And, as shown in FIG. 12, the user interface device 1200 is configured to provide a visual warning when a computing system determines crop lodging or weeds or some other determined attribute of the field that is not a selected crop or a standing crop is in-line with at least one input of the graphical representations of the plurality of inputs of the header (e.g., see the warnings of lodging in areas 1306a and 1306b).

[0081] FIGS. 13 to 15 illustrate respective visualizations of a crop field and an agricultural machine 1402 moving through the field, in which each one of the visualizations can be displayed in any one of the GUIs mentioned herein, and the visualizations being generated based on determined attributes of the field (including determined crop lodging), in accordance with some embodiments of the present disclosure. As shown in FIGS. 13 to 15, the areas of the crop field with detected lodging are colored with a first color and labeled accordingly (e.g., see areas 1406, 1506, and 1606). Also, as shown, the areas of the crop field unaffected by lodging are colored with a second color and labeled accordingly as “standing crop” (e.g., see areas 1407, 1507, and 1607). Further, as shown, the areas of the field that have been operated on (such as harvested by a harvester) are colored with a third color and labeled accordingly as “operated on” (e.g., see areas 1408 and 1508). With respect to FIG. 15, in addition to labeling lodging areas, the visualization shown also includes areas having weeds and a tree.

[0082] In some embodiments of the methods or the systems described herein, the generation of the visualizations by the computing system includes surface mapping of the crop field according to a raster-based mapping technique or a vector-based mapping technique. The area 1406, showing lodging in FIG. 13, is generated according to a raster-based mapping technique as is the area 1606 shown in FIG. 15. Also, FIG. 15 shows the raster-based mapping technique outputting an areahaving weeds and an area having a tree. On the other hand, the area 1506, showing the lodging in FIG. 15, is generated according to a vector-based mapping technique. In general, there are several mapping techniques that can be used to create the visualizations discussed herein, and it is to be understood that raster-based and vector- based mapping are just some of the many known techniques that can be used by the technologies described herein.

[0083] In some embodiments, the raster-based mapping technique includes creating a grid of cells, each of which is assigned a value based on the presence or absence of lodging or another field attribute in that cell. In some embodiments, the vector-based mapping technique includes generating a map using polygons, lines and points to represent the features of the scene, such as the outline of a field or the location of a specific plant. Depending on a specific embodiment, both raster-based and vector-based mapping techniques can be used in 2D or 3D coordinates, depending on the requirements of the visualization.

[0084] In some embodiments, raster-based mapping includes generating a grid of cells, each of which is assigned a value based on the presence or absence of a particular feature in that cell. In such examples and others, raster maps can be useful for representing continuous data, such as elevation or land use, and can be used to create detailed maps with a high level of spatial resolution. In some embodiments, vector-based mapping includes generating a map using lines and points to represent the features of the scene. Vector maps can be useful for representing discrete features, such as the outlines of fields or the locations of specific plants. Vector maps can be more efficient and easier to manipulate than raster maps, because they contain less data and can be scaled up or down without loss of quality.

[0085] In another embodiment of the invention the computing system is configured to determine a lodging pattern of at least one of the one or more lodging areas and to present on the display and as part of the graphical representation of the one or more crop lodging areas, a graphical indicator of the lodging pattern. The lodging pattern can be determined using an artificial neural network trained to identify different patterns. One type of lodging, called wind lodging, results from wind causing the crop to lean. Wind lodging is typically characterized by a uniformly leaning crop pattern. It is often possible to harvest portions of crop that have sustained wind lodging damage if the harvester approaches the crop from a particular direction and / or using particular machine settings, such as moving into the direction of lean of the stalks at a reduced speed. Another type of lodging,called height lodging, results from the crop growing larger than the stem or roots can support such that the stem or roots yield to the weight of the plant. This type of lodging is typically characterized by a non-uniform leaning pattern of the crop. In other words, each of the plants leans in a different direction. Height lodging can be more problematic than wind lodging because, as the plants do not lean in a uniform direction, it is more difficult to harvest and may not be harvested.

[0086] When the artificial neural network model is trained to identify lodging areas of the field it is also trained to identify the lodging pattern of each lodging area such that, during operation of the machine in the field, the model not only identifies portions of the field where lodging exists but also identifies the type of lodging. When the computing system presents the graphical representation of the lodging areas on the display it also presents a graphical indicator of the lodging pattern. Whether the graphical representation of the lodging areas is presented as a video or a generated visualization, the graphical indicator may include arrows or a series of lines indicating a direction of the lodging or, in the case of height lodging, may simply present an outline of the lodging area with no arrows or lines or, alternatively, may present a particular color of shading. Identifying the lodging pattern of each lodging area enables the machine operator to determine whether he should avoid the area during a harvest operation or harvest it in a particular manner. A heat map could be used to indicate the type of lodging pattern — different colors indicate different lodging patterns / angles.

[0087] Figure 10 illustrates exemplary graphical representations of lodging areas on a display along with graphical indicators of lodging pattern. The top portion 1700 illustrates a video presenting an indicator 1702 of a lodging area with graphical indicators 1704 of the lodging pattern. The graphical indicators 1704 include arrows pointing in the direction of crop lodging. In other words, the arrows point in the direction the crop is leaning. The bottom portion 1708 illustrates a generated visualization of a crop lodging area 1710 and includes graphical indicators 1712 of the lodging pattern. The graphical indicators 1704 include arrows pointing in the direction of crop lodging. The top portion 1700 may be presented alone, the bottom portion 1708 may be presented alone, or both the top portion 1700 and the bottom portion 1708 may be presented simultaneously.

[0088] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the dataprocessing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self-consistent sequence of operations leading to a predetermined result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be borne in mind, however, that these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure can refer to the action and processes of a computing system, or similar electronic computing device, which manipulates and transforms data represented as physical (electronic) quantities within the computing system's registers and memories into other data similarly represented as physical quantities within the computing system memories or registers or other such information storage systems.

[0089] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed for the intended purposes, or it can include a general- purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer readable storage medium, such as any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, readonly memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computing system bus.

[0090] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the methods. The structure for a variety of these systems will appear as set forth in the description herein. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the disclosure as described herein.

[0091] The present disclosure can be provided as a computer program product, or software, which can include a machine-readable medium having stored thereon instructions, which can beused to program a computing system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as a read only memory (“ROM”), random access memory (“RAM”), magnetic disk storage media, optical storage media, flash memory components, etc.

[0092] While the invention has been described in conjunction with the specific embodiments described herein, it is evident that many alternatives, combinations, modifications and variations are apparent to those skilled in the art. Accordingly, the example embodiments of the invention, as set forth herein are intended to be illustrative only, and not in a limiting sense. Various changes can be made without departing from the spirit and scope of the invention.

Claims

CLAIMSWhat is claimed is:

1. A method for detecting crop lodging areas (710) in a field while an agricultural machine (300) is moving through the field, comprising: capturing, by a camera (220, 302) mounted to the agricultural machine (300), images (702, 802) of the field and a crop in the field (step 502); determining, by a computing system (200) communicatively coupled to the camera (220, 302), attributes (708) of the crop and the field in the images (504, 702, 802), wherein the determined attributes (708) comprise crop lodging areas (710); and presenting, on a display, a graphical representation of one or more of the crop lodging areas.

2. The method as set forth in claim 1, the step of presenting the graphical representation of the one or more of the crop lodging areas including presenting the crop lodging areas in a video and identifying the crop lodging areas in the video (706).

3. The method of claim 2, further comprising presenting the video along with a second video of a corresponding area of the crop and the field that is displayed without the identification of the crop lodging areas (e.g., see videos 902 and 906).

4. The method as set forth in claim 1, further comprising - generating, by the computing system (200), visualizations according to the determined attributes (708), wherein the generated visualizations comprise visualizations of the crop lodging areas (710), the step of presenting the graphical representation of the one or more of the crop lodging areas including presenting the generated visualizations.

5. The method of claim 4, wherein the determination of the attributes by the computing system comprises using semantic segmentation on the capture image(s) to segment parts of each one of the image(s) into different regions, and wherein each region represents a different type of crop or other feature of the crop field such as weeds, standing crop, and crop lodging.

6. The method as set forth in any preceding claim, further comprising - determining, by the computing system (200), a lodging pattern of at least one of the one or more lodging areas, the step of presenting the graphical representation of the one or more of the crop lodging areas including presenting a graphical indicator of the lodging pattern.

7. The method as set forth in any preceding claim

1. , wherein the capturing of the images (702, 802) and the determining of the attributes (708) occur continuously as the agricultural machine (300) moves through a field.

8. The method as set forth in any preceding claim, wherein the camera (220, 302) is mounted to a front and top portion of the agricultural machine (300).

9. The method as set forth in any preceding claim, wherein the determination of the attributes (708) of the crop and the field in the images (702, 802) is based at least on digital signal processing (step 602).

10. The method of claim 9, wherein the determination of the attributes (708) of the crop and the field in the images (702, 802) is based at least on a computer vision analysis (e.g., see FIG. 6).

11. The method of claim 10, wherein the digital signal processing occurs prior to the computer vision analysis as a pre-processing step to generate enhanced input for the computer vision analysis.

12. The method of claim 11, wherein the computer vision analysis comprises inputting the enhanced input into an artificial neural network (ANN) (604, 607), and wherein the determination of the attributes (708) of the crop and the field in the images (702, 802) is based at least on output of the ANN (606).

13. The method of claim 10, wherein the computer vision analysis comprises inputting an input into an artificial neural network (ANN) (607), and wherein the determination of the attributes (708) of the crop and the field in the images (702, 802) is based at least on output of the ANN (608).

14. The method as set forth in claim 12 or claim 13, wherein the ANN (607) includes or is part of a deep learning process that determines attributes (708) of the crop and the field or is a basis for the determination of the attributes (708) of the crop and the field, wherein the deep learning process includes a convolutional neural network (CNN) for semantic pixel-wise segmentation (704).

15. A system for detecting crop lodging areas (710) in a field while an agricultural machine (300) is moving through the field, comprising: a camera (220, 302) mounted to the agricultural machine (300) and configured to capture images (702, 802) of the field and a crop in the field; a computing system (200) communicatively coupled to the camera (220, 302) and configured to determine attributes (708) of the crop and the field in the images (702, 802) (instructions 222), wherein the determined attributes (708) comprise crop lodging areas (710); and a display configured to display a graphical representation of one or more of the crop lodging areas.

16. The system as set forth in claim 15, the display configured to present the graphical representation of the one or more of the crop lodging areas by presenting the crop lodging areas in a video and identifying the crop lodging areas in the video (706).

17. The system as set forth in claim 15, the computing system (200) configured to generate visualizations according to the determined attributes (708), wherein the generated visualizations comprise visualizations of the crop lodging areas (710), the display configured to present the generate visualizations as part of the graphical representation of the one or more of the crop lodging areas.

18. The system as set forth in claim 15, wherein the computer vision analysis comprises inputting an input into an artificial neural network (ANN) (607), and wherein the determination of the attributes (708) of the crop and the field in the images (702, 802) is based at least on output of the ANN (608).

19. The system as set forth in claim 18, wherein the ANN (607) includes or is part of a deep learning process that determines attributes (708) of the crop and the field or is a basis for the determination of the attributes (708) of the crop and the field, wherein the deep learning process includes a convolutional neural network (CNN) for semantic pixel-wise segmentation (704).

20. The system as set forth in any of claims 15 through 19, further comprising - the computing system (200) configured to detect a lodging pattern of at least one of the one or more lodging areas in images captured by the camera (220, 302), the display configured to present a graphical indicator of the lodging pattern as part of the graphical representation of the one or more of the crop lodging.

Images