System for mapping crop lodging

Abstract

Described herein is a system for mapping crop lodging areas (1110) in a field. In some embodiments, a camera (220, 302) mounted to an agricultural machine (300) (e.g., a combine harvester) captures images (1102, 1202) of a crop and the field and a computing system (200) determines attributes (1108) of the crop and the field in the images (such as crop lodging areas 1110) as well as associates the attributes (1108) with geographic locations of the machine (300). The computing system (200) also generates a map (1004) of crop lodging areas (1110) based on data from the images (1102, 1202) and the geographic locations. In some embodiments, the images (1102, 1202) are geotagged and the geotagged images are used as input for the generation of the map (1004). And, in some embodiments, additional information is used in combination with the data from the from the images (1102, 1202) and the geographic locations as a basis for the generation of the map (1004).

Description

SYSTEM FOR MAPPING CROP LODGINGCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of the filing date of United States Provisional Patent Application Serial No. 63 / 583,932, filed September 20, 2023, the disclosure of which is hereby incorporated herein in its entirety by this reference.TECHNICAL FIELD

[0002] The present disclosure relates to a system for mapping crop lodging in a crop field.BACKGROUND

[0003] Lodging is the shifting of plant roots or stems from more typical vertical positions. Lodging makes it difficult to harvest crops, can cause damage to crops, and often reduces yield. It may occur when the root system or stem base does not have adequate support for the plant. Lodging can be a result of environmental conditions such as rainfall, wind, soil condition, and others. Lodging can also include the bending over of stems near ground level of grain crops. Lodging in cereal crops is often a result of the combined effects of inadequate standing power of the crop and environmental conditions. For example, lodging often affects cereals such as wheat and rice.

[0004] Reducing lodging is a significant goal in agricultural research and precision farming and agriculture, and precision farming and agriculture may improve upon known ways for reducing lodging such as producing shorter varieties of crops.

[0005] Precision agriculture or precision farming is a farming management model based on measuring and responding to inter and intra-field variability in crops and farming conditions. The goal of precision agriculture research is to define a decision support system (DSS) for farming management to enhance returns and increase preservation of resources. Specifically, the precision in responses to variability in farming can be improved when known and predetermined farming information is processed and organized to enhancethe information and then used to assist in the control and management of farming. Although precision farming can enhance returns and increase preservation of resources, it can complicate farming information systems.

[0006] Currently, farming management information systems (FMISs) are pervasive in farming and a significant factor in the furthering of precision agriculture. Such information systems can track measuring and responding to inter and intra-field variability in crops and farming conditions as well as enhance DDS for farming management. FMISs allow for new opportunities to improve farming and precision agriculture. However, even though FMISs are improving precision farming, present FMISs have limitations and can be dramatically improved upon considering relatively recent advancements in computer engineering and computer science. One problem with previous systems is the collection and organization of information from farming, including the collection and organization of information on crop lodging. This can be a problem since farming conditions and lodging can vary greatly in the operations from one field to another.

[0007] These are just some of the many issues that can be improved upon in farming, and specifically, in precision agriculture as well as crop lodging tracking.BRIEF SUMMARY

[0008] Described herein are technologies for mapping crop lodging in a crop field, such as for detecting lodging while harvesting a crop and then mapping the lodging to improve upon some technical problems in the precision farming and agriculture. The techniques disclosed herein provide specific technical solutions to at least overcome the technical problems mentioned in the background section or other parts of the application as well as other technical problems not described herein but recognized by those skilled in the art.

[0009] As mentioned in the background section, reducing lodging may begin with shorter crops; however, precision agriculture or farming may independently improve lodging in crop fields or at least provide improvement to known tactics. Reducing lodging or at least tracking it can possibly increase yields as well as lessen lodging in future crops. With the techniques described herein, precision agriculture or farming can be improved to include thedetection of lodging by automated systems as well as using the detected lodging to provide visual operator assistance of agricultural machinery and improve control systems as well as FMIS mapping. Particular to this disclosure, mapping of crop lodging areas in the field is important for the improvement of crop management, estimating yield loss and counteracting the possible causes for the following year.

[0010] In some embodiments, a system or method to detect and map crop lodging areas in an agricultural field is implemented through an agricultural machine (e.g., a combine harvester) equipped with a machine location tracking system (e.g., one of many known realtime kinematic positioning (RTK) or global positioning satellite systems, or a combination thereof) as well as equipped with a camera configured to capture images of an area in front of the machine (e.g., an RGB camera). Also, to implement the system or method a computing system is provided locally with the machine or remotely to process data captured by the camera and possibly from additional sensors to detect lodging and use the detected lodging to provide additional functionality such as a visual interface for an operator or input for a control system of the machine. Particular to this disclosure, the computing system also provides mapping of crop lodging areas in the field for the improvement of crop management, estimating yield loss and counteracting the possible causes for the following year. Such mapping can include the generation of a FMIS map of the crop field and areas affected by crop lodging. For example, mapping may include a FMIS map with sectors and the sectors are scored with respect to an amount of lodging per sector.

[0011] In some embodiments, an agricultural vehicle includes an RGB camera mounted in the front of the vehicle with a wide-angle horizontal field of view. The possible mounting points can include the top of a cabin or on the hood of the cabin. The camera can capture the images of the crop in front of the vehicle or machine during operation. The images can be processed by a computing system, such as through a deep learning process for pixel-wise semantic segmentation. In some instances, SEGNET or another type of a deep convolutional encoder-decoder architecture for image segmentation is used in or as the deep-learning process such as U-NET (which is a network for biomedical image segmentation), fullyconvolutional network (FCN), DEEPLABV2 or DEEPLABV3, and DECONVNET. For RGB an input image, the deep-learning process can provide an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background', for example. Based on the known field of view of the camera, its mounting position and rotation with respect to the vehicle, the detected areas can be transformed from camera frame to vehicle frame.

[0012] S ince the vehicle can be equipped with a location tracking system or a machine position tracking system (such as RTK-GPS), the data from the vehicle frame can then be transformed into input for a vehicle control system or other types of functionality such as map generation. In embodiments described herein, the input is used to generate a map providing an indication of lodging levels in the field. Also, since the same area of the field can be visible on several consecutive images, the same map areas can be updated as more information about them becomes available, such as by using Bayesian approaches. Furthermore, a simple visualization system for the machine operator can show the map of the field and display results of the deep-learning process or a derivative thereof on the map as the vehicle moves in the field. Such maps and the results of the deep-learning process can be stored for analysis purposes and preparation for the next season as well.

[0013] With respect to some embodiments, disclosed herein are computerized methods for mapping crop lodging, as well as a non-transitory computer-readable storage medium for carrying out technical operations of the computerized methods. The non- transitory computer-readable storage medium has tangibly stored thereon, or tangibly encoded thereon, computer readable instructions that when executed by one or more devices (e.g., one or more personal computers or servers) cause at least one processor to perform a method for improved systems and methods for mapping crop lodging.

[0014] For example, some embodiments are or include a method for mapping crop lodging areas (e.g., see crop lodging areas 1110 shown in FIG. 11) in a field. The method includes capturing, by a camera (e.g., see electronics 220 shown in FIG. 2 as well as camera 302 shown in FIG. 3) mounted to an agricultural machine (e.g., see any one of the agricultural machines shown in FIG. 1 as well as combine harvester 300 shown in FIG. 3) moving throughthe field, images (e.g., see images 1102 and 1202 shown in FIGS. 11 and 12 respectively) of the field and a crop in the field (e.g., see step 502 depicted in FIG. 5). The method also includes capturing, by a location tracking system (e.g., see location tracking system 215 depicted in FIG. 2), geographic locations of the agricultural machine while capturing the images (e.g., see step 504 depicted in FIG. 5). The method also includes determining, by a computing system (e.g., see computing system 200 depicted in FIG. 2) communicatively coupled to the camera, attributes (e.g., see attributes 1108 shown in FIGS. 11 and 12) of the crop and the field in the images (e.g., see step 506 depicted in FIG. 5). The determined attributes include crop lodging areas (e.g., see crop lodging areas 1110).

[0015] In some embodiments of the method, the capturing of the images and the geographic locations as well as the determining of the attributes occur continuously as the agricultural machine moves through a field.

[0016] The method also includes associating, by the computing system, the determined attributes to the geographic locations based on timing of the capturing of the images and the geographic locations (e.g., see step 508) as well as generating, by the computing system, a farming management information system (FMIS) map (e.g., see map 1004 shown in FIG. 10) based on the associations between the determined attributes and the geographic locations (e.g., see step 510). The FMIS map includes levels of crop lodging at different sectors in the field (e.g., see sectors 1006, 1008, 1016, and 1018 shown in FIG. 10).

[0017] Also, in some embodiments, where there are multiple cameras of the harvester or an agricultural machine (in general), the capturing of the images described herein can include multiple cameras capturing images from different positions and thus capturing different perspectives of the field and the crop such that different perspectives of crop lodging areas are captured and more information is provided for the determining of attributes of the crop and the field. For example, in some cases, multiple cameras are mounted around the machine, and the determinations from the images of the attributes are based on combining images taken from different perspectives into a single representation of the crop and field around the machine.

[0018] In some examples, the associations between the determined attributes and the geographic locations includes geotagging the images which includes adding geographical identification metadata to respective items including the images. Also, in such examples, the geographical identification metadata associates the images to the geographic locations based on timing of the capturing of the images and the geographic locations. 6. Further, in some instances, the associations between the determined attributes and the geographic locations further includes associating the determined attributes and the geotagged images in a database (e.g., see instructions 224 shown in FIG. 2). Also, it is to be understood that the associations described herein can include assigning geolocation to each pixel or groups of pixels in an image in some embodiments.

[0019] Also, in some examples, the FMIS map shows at least a plurality of graphical elements (e.g., see graphical elements 1020) corresponding to the geotagged images at a plurality of locations corresponding to geographic locations of the field where capturing of the images occurred. In such examples, when one of the graphical elements is selected (e.g., see graphical element 1021), a corresponding image (e.g., see image 1022) of the geotagged images is displayed with the map. In some of such embodiments, an identification of a crop lodging area (e.g., see crop lodging area 1023) is included in the corresponding image and the identification includes a graphical identification such as a solid color representing lodging. And, in some of such embodiments, the displaying the corresponding image includes displaying the corresponding image along with a respective RGB image (e.g., see image 1024) corresponding to the area of the crop and the field that is displayed with the identification of the crop lodging area.

[0020] F urthermore, in some examples, other attributes (e.g., see attributes 1108 shown in FIG. 11) are identified beyond crop lodging areas, including the sky, weeds areas, unharvested areas, harvested areas, miscellaneous areas, and unidentified or unlabeled areas, and wherein the other identified attributes are represented by corresponding graphical identifications such as a respective solid color in the image for each of the other identified attributes. Also, in some instances, the display can be for the machine operator and it canshow the RGB image with highlighted lodging areas of the field and issue a warning signal so that the operation can be extra alert in these situations. E.g., wherein, for an RGB input image, the determination provides an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background' (this can be broken down into more specific classes such as 'weeds', 'harvested area', 'swath', etc.).

[0021] In some embodiments of the method, the method includes rendering, by a mapping application (e.g., see instructions 228 shown in FIG. 2) of the computing system, the FMIS map to be displayed in a graphical user interface (e.g., see user interface 216 depicted in FIG. 2), wherein the display of the map shows the levels of crop lodging at different sectors in the field (e.g., see step 512 shown in FIG. 5 as well as the sectors depicted in FIG. 10). Also, in some cases, the rendering of the FMIS map includes rendering the FMIS map to be displayed along with a yield map of the field or another type of FMIS map of the field.

[0022] In some embodiments of the method, the method includes communicating, by the computing system, the generated FMIS map over a network (e.g., see bus 218 or network 104 shown in FIG. 2) to a controller (e.g., see electronics 220 and control instructions 230) of the agricultural machine (e.g., see step 514 depicted in FIG. 5). Also, such a method includes controlling, by the controller, one or more operations of the agricultural machine according to the generated FMIS map (e.g., see step 516).

[0023] In some embodiments of the method, the method includes generating, by the computing system, a route based on the FMIS map (e.g., see step 602 depicted in FIG. 6). And, such a method includes controlling, by a controller (e.g., see electronics 220 and control instructions 230) of the agricultural machine, one or more operations of the agricultural machine according to the generated route (e.g., see step 604). In some cases, the controlling of the operation(s) includes controlling direction and speed of the machine according to the generated route. Also, in some examples, the controlling of the operation(s) includes controlling an implement (e.g., see feeder 315 shown in FIG. 3) of the machine according to the generated FMIS map.

[0024] In some embodiments of the method, the method includes capturing, by a secondary sensor (e.g., see sensor 304) mounted to the agricultural machine, secondary information of the field and the crop (e.g., see step 702 depicted in FIG. 7). Such a method includes capturing, by the location tracking system, geographic locations of the agricultural machine while capturing the secondary information (e.g., see step 704 depicted in FIG. 7). It also includes determining, by the computing system, second attributes (e.g., see attributes 1108 shown in FIGS. 11 and 12) of the crop and the field in the secondary information (e.g., see step 706). The determined second attributes can include crop lodging areas too. The method also includes associating, by the computing system, the determined second attributes to the geographic locations based on timing of the capturing of the secondary information and the geographic locations (e.g., see step 708). And, the method includes generating, by the computing system, the FMIS map based on the associations between the determined attributes, the second determined attributes, and the geographic locations (e.g., see step 710). In some embodiments, the secondary sensor (e.g., sensor 304) is a second optical sensor (such as a second camera in a stereo pair of cameras that can capture depth information). In some embodiments, the secondary sensor is an ultrasonic transducer. In some examples, the secondary sensor is a radio frequency (RF) sensor. In some examples, the secondary sensor is part of a lidar system. In some cases, the secondary sensor, is part of a radar system. Or, in some cases, the secondary sensor (304) is a combination of one or more of the following type of sensors: a second optical sensor (such as a second camera), an ultrasonic transducer, a radio frequency (RF) sensor, lidar, radar, or any other type of sensor that senses structure in a crop field without contacting the crop field.

[0025] In some examples of the method, the camera or the secondary sensor is mounted to a front and top portion of the agricultural machine. In some examples, the camera or the secondary sensor includes a wide field of view (e.g., see field of view 401 shown in FIG. 4). In some cases, the capturing of the images can be performed by a stereo pair of cameras mounted to the machine.

[0026] In some examples of the method, the determination of the first or second attributes of the crop and the field in the images is based at least on digital signal processing (e.g., see step 902 of step 506 or step 706 shown in FIGS. 5 and 7 respectively). In some examples of the method, the determination of the first or second attributes of the crop and the field in the images is based at least on a computer vision analysis (e.g., see FIG. 9). More specifically, in some cases, the determination of the first or second attributes of the crop and the field in the images is further based on digital signal processing (e.g., see step 902). Further, in some instances, the digital signal processing occurs prior to the computer vision analysis as a pre-processing step to generate enhanced input for the computer vision analysis. Also, even more specifically, in such cases, the computer vision analysis includes inputting the enhanced input into an artificial neural network (ANN) (e.g., see scheme 907 and step 904), and the determination of the first or second attributes of the crop and the field in the images is based at least on output of the ANN (e.g., see step 906).

[0027] In general, some examples includes the computer vision analysis including inputting an input into an ANN (e.g., see scheme 907 shown in FIG. 9), and the determination of the first or second attributes of the crop and the field in the images is based at least on output of the ANN (e.g., see step 908). More specifically, the computer vision analysis can include inputting aspects of the images or derivatives of aspects of the images into an ANN (e.g., see scheme 907), and the determination of the first or second attributes of the crop and the field in the images is based at least on output of the ANN. Such these more specific examples, the ANN can include or can be a part of a deep learning process that determines attributes of the crop and the field or can be a basis for the determination of the first or second attributes of the crop and the field. In such cases, the deep learning process can include a convolutional neural network (CNN) for semantic pixel-wise segmentation (e.g., see scheme 1104 shown in in FIGS. 11 and 12).

[0028] With respect to some embodiments, a system is provided that includes at least one computing device configured to provide improved ways for mapping crop lodging. And, with respect to some embodiments, a method, such as one of the aforesaid methods, isprovided to be performed by at least one computing device. In some example embodiments, computer program code can be executed by at least one processor of one or more computing devices to implement functionality in accordance with at least some embodiments described herein; and the computer program code being at least a part of or stored in a non-transitory computer-readable medium.

[0029] For example, some embodiments are or include a system for mapping crop lodging areas (e.g., see crop lodging areas 1110 depicted in FIG. 11) in a crop field. Such a system includes a camera (e.g., see electronics 220 shown in FIG. 2 as well as camera 302 shown in FIG. 3) mounted to an agricultural machine (e.g., see any one of the agricultural machines shown in FIG. 1 as well as combine harvester 300 shown in FIG. 3) and configured to capture images (e.g., see images 1102 and 1202 shown in FIGS. 11 and 12 respectively) of the field and a crop in the field. The system also includes a location tracking system (e.g., see location tracking system 215 depicted in FIG. 2) configured to capture geographic locations of the agricultural machine while the camera captures the images and the machine is moving through the field. The system also includes a computing system (e.g., see computing system 200 depicted in FIG. 2) communicatively coupled to the camera and the location tracking system and configured to determine attributes (e.g., see attributes 1108 shown in FIGS. 11 and 12) of the crop and the field in the images (e.g., see instructions 222 shown in FIG. 2). The determined attributes include crop lodging areas, and in some cases, the capturing of the images and the determining of the attributes occurs continuously as the agricultural machine moves through a field. Also, the computing system is configured to associate the determined attributes (e.g., see attributes 1108) to the geographic locations based on timing of the capturing of the images and the geographic locations (e.g., see instructions 224 depicted in FIG. 2). And, the computing system is configured to generate a farming management information system (FMIS) map (e.g., see map 1004) based on the associations (e.g., see instructions 226). The FMIS map includes levels of crop lodging at different sectors in the field (e.g., see sectors 1006, 1008, 1016, 1018). Also, it is to be understood that the associations described herein can include assigning geolocation to each pixel or groups of pixels in an image in some embodiments.

[0030] Also, for example, some embodiments are or include a system for mapping crop lodging areas in a crop field, wherein, in a more general sense, a sensor and sensor data is used instead of specifically a camera and captured images. Such a system includes a sensor (e.g., see electronics 220 shown in FIG. 2) mounted to an agricultural machine (e.g., see any one of the agricultural machines shown in FIG. 1) and configured to capture data on aspects of the field and a crop in the field. The system also includes a location tracking system (e.g., see location tracking system 215) configured to capture geographic locations of the agricultural machine while the sensors captures such data and the machine is moving through the field. The system also includes a computing system (e.g., see computing system 200 depicted in FIG. 2) communicatively coupled to the sensor and the location tracking system and configured to determine attributes of the crop and the field in the captured data (e.g., see instructions 222 shown in FIG. 2). The determined attributes include crop lodging areas, and in some cases, the capturing of the data and the determining of the attributes occurs continuously as the agricultural machine moves through a field. Also, the computing system is configured to associate the determined attributes to the geographic locations based on timing of the capturing of the captured data and the geographic locations (e.g., see instructions 224 depicted in FIG. 2). And, the computing system is configured to generate a FMIS map based on the associations (e.g., see instructions 226). The FMIS map includes levels of crop lodging at different sectors in the field. Also, it is to be understood that the associations described herein can include assigning geolocation to each pixel or groups of pixels in an image in some embodiments.

[0031] The systems and methods described herein overcome some technical problems in farming, in general, as well as some technical problems in mapping lodging specifically. Also, the techniques disclosed herein provide specific technical solutions to at least overcome the technical problems mentioned in the background section or other parts of the application as well as other technical problems not described herein but recognized by those skilled in the art.

[0032] These and other important aspects of the invention are described more fully in the detailed description below. The invention is not limited to the particular methods and systems described herein. Other embodiments can be used and changes to the described embodiments can be made without departing from the scope of the claims that follow the detailed description.

[0033] Within the scope of this application it should be understood that the various aspects, embodiments, examples and alternatives set out herein, and individual features thereof may be taken independently or in any possible and compatible combination. Where features are described with reference to a single aspect or embodiment, it should be understood that such features are applicable to all aspects and embodiments unless otherwise stated or where such features are incompatible.BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0034] The present disclosure will be understood more fully from the detailed description given below and from the accompanying drawings of various embodiments of the disclosure. Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings, in which:

[0035] FIG. 1 illustrates an example network of combine harvesters that communicate with a computing system through a communication network, in accordance with some embodiments of the present disclosure;

[0036] FIG. 2 illustrates is a block diagram of example aspects of a computing system (such as one of the computing systems shown in FIG. 1), in accordance with some embodiments of the present disclosure;

[0037] FIG. 3 illustrates a schematic side view of a combine harvester (such as one of the combine harvesters shown in FIG. 1) with some portions of the harvester being broken away to reveal internal details of construction, in accordance with some embodiments of the present disclosure;

[0038] FIG. 4 illustrates a perspective view of the combine harvester shown in FIG. 3 along with a field of view of a camera of the harvester, in accordance with some embodiments of the present disclosure;

[0039] FIGS. 5 to 9 illustrate methods in accordance with some embodiments of the present disclosure;

[0040] FIG. 10 illustrates a display of a user interface device displaying a map of a field based on determined attributes of the field, including determined crop lodging, in accordance with some embodiments of the present disclosure;

[0041] FIG. 11 illustrates an example RGB image of a field derived from image data captured by a camera of a harvester as well as a data processing scheme that receives input derived from the RGB image and processes the input to output data for a prediction view of the image showing determined attributes of the field (wherein the attributes shown include lodging areas), in accordance with some embodiments of the present disclosure;

[0042] FIG. 12 illustrates another example RGB image of a field derived from image data captured by a camera of a harvester as well as the data processing scheme shown in FIG. 11 processes input derived from the RGB image to output data for a prediction view of the image showing determined attributes of the field (wherein the attributes shown do not include lodging areas), in accordance with some embodiments of the present disclosure; and

[0043] FIG. 13 illustrates a display of a user interface device displaying a map of a field based on determined attributes of the field, including determined crop lodging and graphical indicators of lodging patterns, in accordance with some embodiments of the present disclosure.DETAILED DESCRIPTION

[0044] Details of example embodiments of the invention are described in the following detailed description with reference to the drawings. Although the detailed description provides reference to example embodiments, it is to be understood that the invention disclosed herein is not limited to such example embodiments. But to the contrary, theinvention disclosed herein includes numerous alternatives, modifications and equivalents as will become apparent from consideration of the following detailed description and other parts of this disclosure.

[0045] Described herein are technologies for detecting crop lodging (e.g., see crop loading areas 1110 shown in FIG. 11) in a crop field as well as mapping the crop field according to at least the detected crop lodging, such as to improve upon some technical problems in the field. The techniques disclosed herein provide specific technical solutions to at least overcome the technical problems mentioned in the background section or other parts of the application as well as other technical problems not described herein but recognized by those skilled in the art. As mentioned in the background section, reducing lodging may begin with shorter crops; however, precision agriculture or farming may independently improve lodging in crop fields or at least provide improvement to known tactics. Lodging can include the displacement of the plant roots or stems from vertical positions, making such plants difficult to harvest. Reducing lodging or at least tracking it can possibly increase yields as well as lessen lodging in future crops. With the techniques described herein, precision agriculture or farming can be improved.

[0046] FIG. 1 illustrates a network 100 of farming machines that communicate with remote computing systems through a communications network 104. Specifically, FIG. 1 illustrates the network 100 including a remote computing system 102, the communications network 104, farming machines (e.g., see farming machines 106, 108, and 110). The remote computing system (e.g., see remote computing system 102) is remote in that it is physically and geographically separated from the farming machines of the network 100. The farming machines are shown communicating with the remote computing system 102 of the network 100 through a communications network 104. As shown in FIG. 1, the farming machines of the network 100 can each include its own computing system including electronics such as connected sensors, cameras, busses, and computers (e.g., see computing systems 116, 118, and 120, and see electronics 126, 128, and 130). A computing system of a farming machine can include a processor, memory, a communication interface and one or more sensors thatcan make the farming machines individual computing devices. In the case of the communications network 104 including the Internet, the farming machines of the network 100 are considered Internet of Things (loT) devices. Also, in some embodiments, the remote computing system 102 is a part of a cloud computing system.

[0047] As shown in FIG. 1, each one of the farming machines includes a respective computer and respective electronics (e.g., see computing systems 116, 118, and 120, and see electronics 126, 128, and 130). In some embodiments, the electronics of a farming machine include electronic hardware and software of the farming machine such as sensors and other types of electrical and / or mechanical feedback devices that are communicatively coupled with the computer of the farming machine (e.g., see camera 302 and sensor 304 of combine harvester 300 shown in FIG. 3). And, in some embodiments, the computer of the farming machine is configured to connect or integrate with the farming machine electronics (e.g., see electronics 126, 128, and 130) and communicate with a remote computing system (such as the remote computing system 102) via the communications network 104.

[0048] In some embodiments, the farming machine (e.g., see farming machine 106, 108, or 110) includes a vehicle. In some embodiments, the farming machine is a combine harvester (e.g., see combine harvester 300 show in FIGS. 3 and 4). In some embodiments, the farming machine is a tractor. In some embodiments, the farming machine is a planter. In some embodiments, the farming machine is a sprayer. In some embodiments, the farming machine is a baler. In some embodiments, the farming machine is or includes a harvester, a planter, a sprayer, a baler, any other type of farming implement, or any combination thereof. In some of such embodiments, the farming machine can be or include a vehicle in that is self-propelling. Also, in some embodiments, the group of similar farming machines is a group of vehicles (e.g., see farming machines 106, 108, and 110). In some embodiments, the group of vehicles is a group of combine harvesters. And, in some embodiments, the group of vehicles is a group of combine harvesters, planters, sprayers, balers, another type of implement, or any combination thereof.

[0049] The communications network 104 includes one or more local area networks (LAN(s)) and / or one or more wide area networks (WAN(s)). In some embodiments, the communications network 104 includes the Internet and / or any other type of interconnected communications network. The communications network 104 can also include a single computer network or a telecommunications network. More specifically, in some embodiments, the communications network 104 includes a local area network (LAN) such as a private computer network that connects computers in small physical areas, a wide area network (WAN) to connect computers located in different geographical locations, and / or a middle area network (MAN) to connect computers in a geographic area larger than that covered by a large LAN but smaller than the area covered by a WAN.

[0050] At least each shown component of the network 100 (including remote computing system 102, communications network 104, and farming machines 106, 108, and 110) can be or include a computing system which includes memory that includes media. The media includes or is volatile memory components, non-volatile memory components, or a combination of thereof. In general, in some embodiments, each of the computing systems includes a host system that uses memory. For example, the host system writes data to the memory and read data from the memory. The host system is a computing device that includes a memory and a data processing device. The host system includes or is coupled to the memory so that the host system reads data from or writes data to the memory. The host system is coupled to the memory via a physical host interface. The physical host interface provides an interface for passing control, address, data, and other signals between the memory and the host system.

[0051] FIG. 2 illustrates is a block diagram of example aspects of a computing system 200. In some embodiments, the computing system 200 is a part of a farming machine or any one of the agricultural machines described herein (e.g., see harvester 300 depicted in FIG. 3). In some embodiments, the computing system 200 is local to the farming machine (e.g., see computing systems 116, 118, and 120 shown in FIG. 1). In some alternative embodiments, the computing system can include a remote computing system (e.g., see remote computingsystem 102), wherein the remote computing system can be the sole computing system of the farming machine in some examples or in combination with a local computing system of the farming machine in some other examples.

[0052] FIG. 2 illustrates parts of the computing system 200 within which a set of instructions, for causing a machine (such as a computer processor or processing device 202) to perform any one or more of the methodologies discussed herein performed by a computing system, are executed (e.g., see the method steps of method 500 shown in FIG. 5). In some embodiments, the computing system 200 operates with additional computing systems (e.g., such as additional remote computing systems) to provide increased computing capacity in which multiple computing systems operate together to perform any one or more of the methodologies discussed herein that are performed by a computing system. In some embodiments, the computing system 200 corresponds to a host system that includes, is coupled to, or utilizes memory or is used to perform the operations performed by any one of the computing systems described herein. In some embodiments, the machine is connected (e.g., networked) to other machines in a LAN, an intranet, an extranet, or the Internet. In some embodiments, the machine operates in the capacity of a server in client-server network environment, as a peer machine in a peer-to-peer (or distributed) network environment, or as a server in a cloud computing infrastructure or environment. In some embodiments, the machine is a personal computer (PC), a tablet PC, a cellular telephone, a web appliance, a server, or any machine capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that machine. Further, while a single machine is illustrated, the term "machine" shall also be taken to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein performed by computing systems.

[0053] The computing system 200 includes a processing device 202, a main memory 204 (e.g., read-only memory (ROM), flash memory, dynamic random-access memory (DRAM), etc.), a static memory 206 (e.g., flash memory, static random-access memory (SRAM), etc.), and a data storage system 210, which communicate with each other via a bus 218. Theprocessing device 202 represents one or more general-purpose processing devices such as a microprocessor, a central processing unit, or the like. More particularly, the processing device can include a microprocessor or a processor implementing other instruction sets, or processors implementing a combination of instruction sets. Or, the processing device 202 is one or more special-purpose processing devices such as an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), network processor, or the like. The processing device 202 is configured to execute instructions 214 for performing the operations discussed herein performed by a computing system. In some embodiments, the computing system 200 includes a network interface device 208 to communicate over the communications network 104 shown in FIG. 1.

[0054] The data storage system 210 includes a machine-readable storage medium 212 (also known as a computer-readable medium) on which is stored one or more sets of instructions 214 or software embodying any one or more of the methodologies or functions described herein performed by a computing system. The instructions 214 also reside, completely or at least partially, within the main memory 204 or within the processing device 202 during execution thereof by the computing system 200, the main memory 204 and the processing device 202 also constituting machine-readable storage media.

[0055] In some embodiments, the instructions 214 include specific instructions to implement functionality described herein related to the methods described herein and that can correspond to any one of the computing devices, data processors, user interface devices, and I / O devices described herein related to a computing system. For example, the instructions 214 include crop lodging detection instructions 222 (which include instructions configured to determine attributes of a crop field and the crops with in the field), data linking and recording instructions 224 (which includes instructions configured to record and associate determined attributes of field and its crops with other parameters such as geographic location of the agricultural machine where the input data for determining the attributes was captured), data enhancement instructions 226 (e.g., which includes instructions for enhanced attribute determination such as instructions for a computing scheme, e.g., ANN, CNN, etc. specificallyadapted for determining and identifying lodging captured in images), map generation instructions 228 (which can generate an FMIS map, such as a map based on crop lodging levels), and controller instructions 230 (which are configured to control parts and operations of an agricultural according to input and feedback such as including the determined attributes of the field and crop or a map generated using the instructions 228). In some embodiments, the data enhancement instructions include different types of data analysis libraries as well different types of data processing li bra ries-i ncl udi ng various mathematical and statistical modeling and operations libraries and machine learning, artificial intelligence, and deep learning libraries as well as specific libraries for ANN and CNN data processing and for training ANNs, CNNs and other types of computing schemes or systems.

[0056] While the machine-readable storage medium 212 is shown in an example embodiment to be a single medium, the term "machine-readable storage medium" should be taken to include a single medium or multiple media that store the one or more sets of instructions. The term "machine-readable storage medium" shall also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies of the present disclosure performed a computing system. The term "machine-readable storage medium" shall accordingly be taken to include solid-state memories, optical media, or magnetic media.

[0057] Also, as shown, the computing system 200 includes user interface 216 that includes a display, in some embodiments, and, for example, implements functionality corresponding to any one of the user interface devices disclosed herein. A user interface, such as user interface 216, or a user interface device described herein includes any space or equipment where interactions between humans and machines occur. A user interface described herein allows operation and control of the machine from a human user, while the machine simultaneously provides feedback information to the user. Examples of a user interface (U I), or user interface device include the interactive aspects of computer operating systems (such as graphical user interfaces), machinery operator controls, and processcontrols. A Ul described herein includes one or more layers, including a human-machine interface (HMI) that interfaces machines with physical input hardware and output hardware.

[0058] Also, as shown, the computing system 200 includes farming machine electronics 220 that includes one or more sensors, cameras, other types of electrical or mechanical feedback devices, user interfaces (e.g., any one of the Ul described herein), and any other type of computer hardware and software configured to interface and communicatively couple to operational components of a farming machine (e.g., see electronics 126, 128, and 130 shown in FIG. 1). The electronics 220 includes farming-machine controller hardware that can operate in combination with controller instructions 230, in some embodiments. Also, in some embodiments, the farming machine electronics 220 as well as the electronics 126, 128, and 130 include any one of the cameras described herein for capturing images of a field and crop within the field (e.g., see camera 302 and sensor 304 show in FIG. 3).

[0059] In some systems of the technologies disclosed herein, any steps of embodiments of the methods described herein are implementable by executing instructions corresponding to the steps, which are stored in memory (e.g., see instructions 214, 222, 224, 226, 228 and 230 shown in FIG. 2).

[0060] FIG. 3 illustrates a schematic side view of a combine harvester 300 (which can be one of the farming machines shown in FIG. 1) with some portions of the harvester being broken away to reveal internal details of construction. As shown in FIG. 3, the combine harvester 300 includes at least one camera and at least one sensor. The at least one camera includes a camera 302 mounted to the harvester 300 at the front of the harvester. The camera 302 is configured to capture images of crop and its surrounding while the crop is being harvested or just before the crop is harvested (e.g., see image 1102 shown in FIG. 11). In some embodiments, the at least one camera includes a second camera (not depicted). Also shown in FIG. 3, is sensor 304, which can be one of the at least one sensor of harvester 300. The sensor 304 is or includes a second optical sensor (such as a second camera), an ultrasonic transducer, a radio frequency (RF) sensor, lidar, radar, or any type of sensor that senses structure in a crop field without contacting the crop field, or any combination thereof.

[0061] In embodiments where there are multiple cameras of the harvester 300 or an agricultural machine (in general), the capturing of the images described herein can include multiple cameras capturing images from different positions and thus capturing different perspectives of the field and the crop such that different perspectives of crop lodging areas are captured and more information is provided for the determining of attributes of the crop and the field. Also, the agricultural machine can have multiple cameras mounted around itself that can provide a greater field of view and coverage of the area around the machine (even a 360-degree view). For example, in some cases, multiple cameras are mounted around the machine, and the determinations from the images of the attributes are based on combining images taken from different perspectives into a single representation of the crop and field around the machine.

[0062] The combine harvester 300 has processing system 312 that extends generally parallel with the path of travel of the harvester. It is to be understood that such a harvester is being used to illustrate principals herein and the subject matter described herein is not limited to harvesters with processing systems designed for axial flow, nor to axial flow harvesters having only a single processing system. The combine harvester 300 also includes a harvesting header (not shown) at the front of the machine that delivers collected crop materials to the front end of a feeder house 314 of a feeder 315. Such materials are moved upwardly and rearwardly within feeder house 314 by a conveyer 316 until reaching a beater 318 that rotates about a transverse axis. Beater 318 feeds the material upwardly and rearwardly to a rotary processing device, in the illustrated instance to a rotor 322 having an infeed auger 320 on the front end thereof. Infeed auger 320, in turn, advances the materials axially into the processing system 312 for threshing and separating. The processing system 312 is housed by processing system housing 313. In other types of systems, conveyer 316 may deliver the crop directly to a threshing cylinder. The crop materials entering processing system 312 can move axially and helically therethrough during threshing and separating. During such travel, the crop materials are threshed and separated by rotor 322 operating in chamber 323 which concentrically receives the rotor 322. The lower part of the chamber 323 contains concave assembly 324 and a separator grate assembly 326. Rotation of the rotor 322impels the crop material rearwardly in a generally helical direction about the rotor 322. A plurality of rasp bars and separator bars (not shown) mounted on the cylindrical surface of the rotor 322 cooperate with the concave assembly 324 and separator grate assembly 326 to thresh and separate the crop material, with the grain escaping laterally through concave assembly 324 and separator grate assembly 326 into cleaning mechanism 328. Bulkier stalk and leaf materials are retained by the concave assembly 324 and the separator grate assembly 326 and are impelled out the rear of processing system 312 and ultimately out of the rear of the combine harvester 300. Such parts and operations of the harvester 300 as well as parts and operations of a harvester can be controlled by a control system of such a harvester (e.g., see controller instructions 230 and electronics 220 shown in FIG. 2) according to at least the determined attributes of the field and the crops of the field described herein (e.g., see the determined attributes 1108 and 1208 shown in FIGS. 11 to 12 as well as the determination of the attributes at step 506 of the method 500 depicted in FIG. 5). Also, in some embodiments, the input and basis of such control includes an FMIS map or a derivative of an FMIS map (such as a route for an agricultural machine derived from a FMIS map).

[0063] FIG. 4 illustrates a perspective view of the combine harvester 300 shown in FIG. 3 along with a field of view 401 of a camera of the harvester, such as camera 302 shown in FIG. 3. The field of view 401 can also be the field of view of the sensor 304 in some embodiments. As shown in FIG 4, a sensor platform 400 is attached to the hood 303 of the cabin 301 of the harvester 300. The sensor platform 400 includes the camera 302 (shown in FIG. 3) and can include additional sensors such as a second optical sensor (such as a second camera), an ultrasonic transducer, a radio frequency (RF) sensor, lidar, radar, or any type of sensor that senses structure in a crop field without contacting the crop field, or some combination thereof. For example, in some embodiments, the platform 400 includes the sensor 304 (shown in FIG. 3). One or more of the additional sensors can capture data that can be used by a computing system (such as the computing system of the harvester) to verify the images or the data captured by the camera 302. For example, sensor 304 can capture data that can be used by a computing system (such as the computing system of the harvester) to verify the images or the data captured by the camera 302. Also shown, the sensor platform400 is mounted to a front and top portion of the harvester 300 and the platform is mounted such that a field of view 401 is not limited by the feeder house 314 attached to the front of the harvester. In some embodiments, the camera 302 is directly mounted to the cabin 301, and, in other examples, the sensor platform 400 is directly mounted to the cabin. Further, in some cases, the sensor 304 is directly mounted to the cabin 301. And, in some instances, the camera 302 is directly mounted to the hood 303 of the cabin, and in other examples, the sensor platform is directly mounted to the hood. And, in some cases, the sensor 304 is directly mounted to the hood 303 of the cabin. In some embodiments, the camera 302 and / or the sensor 304 includes a wide field of view (e.g., see field of view 401). And, in some instances, the camera 302 and / or the sensor 304 includes a wide-angle horizontal field of view (e.g., see field of view 401 which is an example of a wide-angle horizontal field of view). In some embodiments the wide angle 402 of the field of view is at least sixty degrees. In some instances the wide angle 402 can be as much as 360 degrees. In such cases, additional cameras and / or sensors are used and the captured data from the cameras and / or sensors is used as input to generate a combined image of the 360-degrees wide angle.

[0064] FIGS. 5 to 9 illustrate methods in accordance with some embodiments of the present disclosure. Methods 500 to 800 of FIGS. 5 to 8 respectively are performed by any one of the cameras and any one of the computing systems described herein (e.g., see computing system 200 and electronics 220 as well as camera 302). Methods 500 to 800 are alternative embodiments; however, aspects of the methods can be combined to make a greater or more specific method. And, such a combination is to be considered part of this disclosure. The method shown in FIG. 9 is a sub-method of the method step 506 (see FIGS. 5 to 8) or method step 706 (see FIGS. 7 to 8).

[0065] As shown in FIG. 5, method 500 begins with step 502, which includes capturing images (e.g., see images 1102 and 1202 shown in FIGS. 11 and 12 respectively) of the field and a crop in the field by a camera (e.g., see electronics 220 as well as camera 302) mounted to an agricultural machine (e.g., see harvester 300 or any one of the agricultural machines depicted in FIG. 1) moving through a crop field. In some cases, the capturing of the imagescan be performed by a stereo pair of cameras mounted to the machine that can capture depth information. At step 504, the method 500 continues with capturing, by a location tracking system (e.g., see location tracking system 215 depicted in FIG. 2), geographic locations of the agricultural machine while capturing the images. At step 506, the method 500 continues with determining, by a computing system (e.g., see computing system 200 shown in FIG. 2) communicatively coupled to the camera, attributes (e.g., see attributes 1108 shown in FIGS. 11 and 12) of the crop and the field in the images. The determined attributes include crop lodging areas (e.g., see crop lodging areas 1110 depicted in FIG. 11). And, such a determination of attributes of the crop and field and other determinations of aspects of a crop and field described herein can be performed by executing instructions 222. In some embodiments, the capturing of the images and the geographic locations as well as the determining of the attributes occur continuously as the agricultural machine moves through a field.

[0066] At step 508, the method 500 continues with associating, by the computing system, the determined attributes to the geographic locations based on timing of the capturing of the images and the geographic locations. Such an association and other associations described herein can be performed by executing instructions 224. In some cases, the associations between the determined attributes and the geographic locations includes geotagging the images which includes adding geographical identification metadata to respective items including the images, and the geographical identification metadata associates the images to the geographic locations based on timing of the capturing of the images and the geographic locations. In some of such cases, the associations between the determined attributes and the geographic locations further includes associating the determined attributes and the geotagged images in a database (e.g., such as via database instructions of instructions 224). Also, it is to be understood that the associations described herein can include assigning geolocation to each pixel or groups of pixels in an image in some embodiments.

[0067] At step 510, the method 500 continues with generating, by the computing system, a farming management information system (FMIS) map (e.g., see map 1004 shown in FIG. 10) based on the associations between the determined attributes and the geographic locations. The FMIS map includes levels of crop lodging at different sectors in the field (e.g., see sectors 1006, 1008, 1016, 1018). Such a generation of the map and other map generations described herein can be performed by executing instructions 228 and such instructions can include a mapping application.

[0068] At step 512, the method 500 continues with rendering, by a mapping application (e.g., see instructions 228) of the computing system, the FMIS map to be displayed in a graphical user interface (e.g., see user interface 216). Such a rendering of the map and other map renderings described herein can be performed by executing instructions 228 and such instructions can include the mapping application. The display of the map shows the levels of crop lodging at different sectors in the field. In some embodiments, the FMIS map shows at least a plurality of graphical elements (e.g., see graphical elements 1020 depicted in FIG. 10) corresponding to the geotagged images at a plurality of locations corresponding to geographic locations of the field where capturing of the images occurred, and when one of the graphical elements is selected (e.g., see the selected graphical element 1021), a corresponding image (e.g., see the corresponding image 1022) of the geotagged images is displayed with the map. In some examples, an identification of a crop lodging area (e.g., see crop lodging area 1023) is included in the corresponding image and the identification includes a graphical identification such as a solid color representing lodging. And, the displaying the corresponding image includes displaying the corresponding image along with a respective RGB image (e.g., see RGB image 1024) corresponding to the area of the crop and the field that is displayed with the identification of the crop lodging area. In some examples, other attributes (e.g., see attributes 1108 shown in FIGS. 11 and 12) are identified beyond crop lodging areas (e.g., see crop lodging areas 1110), including the sky, weeds areas, unharvested areas, harvested areas, miscellaneous areas, and unidentified or unlabeled areas, and wherein the other identified attributes are represented by corresponding graphical identifications such as a respective solid color in the image for each of the other identified attributes. Also,in some examples, the display can be for an operator of the machine and it can show the RGB image with highlighted lodging areas of the field as well as issue a warning signal so that the operation can be extra alert in such situations. E.g., wherein, for an RGB input image, the determination provides an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background' (this can be broken down into more specific classes such as 'weeds', 'harvested area', 'swath', etc.).

[0069] Also, at step 514, the method 500 continues with communicating, by the computing system, the generated FMIS map over a network (e.g., see bus 218 or network 104 shown in FIG. 2) to a controller (e.g., see electronics 220 and instructions 230) of the agricultural machine. And, at step 516, the method 500 continues with controlling, by the controller, one or more operations of the agricultural machine according to the generated FMIS map. In some embodiments, the method can include generating a route from the map and controlling the machine according to the route.

[0070] As shown in FIG. 6, a method 600 includes the steps 502 to 512 of method 500 and additionally, at step 602, generating a route based on the FMIS map by the computing system. Method 600 also includes step 604 of controlling, by a controller (e.g., see electronics 220 and instructions 230) of the agricultural machine, one or more operations of the agricultural machine according to the generated route. In some cases, the controlling of the operation(s) includes controlling direction and speed of the machine according to the generated route. And, in some instances, the controlling of the operation(s) includes controlling an implement (e.g., see feeder 315) of the machine according to the generated route or the generated FMIS map. Route generations described herein can be performed by executing instructions 228 or 230 depending on the implementation.

[0071] In some embodiments, the method can include using a second camera or sensor to capture additional farming information to base determinations of attributes and creations of maps on additional data collected in the field. As shown in FIG. 7, a method 700 includes the steps 502 to 508 of method 500 and additionally steps 702 to 710 as well as steps 514 to 516 of method 500. Specifically, at step 702, the method 700 continues with capturing, by asecondary sensor (e.g., see sensor 304 shown in FIG. 3) mounted to the agricultural machine, secondary information of the field and the crop. Method 700 also includes step 704 of capturing, by the location tracking system (e.g., see location tracking system 215 depicted in FIG. 2), geographic locations of the agricultural machine while capturing the secondary information. At step 706, the method 700 continues with determining, by the computing system, second attributes (e.g., see attributes 1108) of the crop and the field in the secondary information. The determined second attributes include crop lodging areas as well (e.g., see crop lodging areas 1110). The secondary sensor can be or include a second optical sensor (such as a second camera), an ultrasonic transducer, a radio frequency (RF) sensor, lidar, radar, or any type of sensor that senses structure in a crop field without contacting the crop field, or any combination thereof. And, the second attributes can be derived from a second group of optical data, ultrasonic data, RF data, lidar data, radar data, or any other type of data captured by a sensor that senses structure in a crop field without contacting the crop field. For instance, the information used as a basis for the determination of the attributes can be 3D or stereoscopic image information derived from data captured by multiple cameras or sensors. Also, such secondary information can be used to confirm the initial determined attributes. At step 708, the method 700 continues with associating, by the computing system, the determined second attributes to the geographic locations based on timing of the capturing of the secondary information and the geographic locations. And, at step 710, the method 700 continues with generating, by the computing system, the FMIS map based on the associations between the determined attributes, the second determined attributes, and the geographic locations. Also, it is to be understood that the associations described herein can include assigning geolocation to each pixel or groups of pixels in an image in some embodiments.

[0072] As shown in FIG. 8, a method 800 includes the steps 502 to 508 of method 500 and additionally steps 702 to 710 of method 700 as well as steps 602 to 604 of method 600 instead of steps 514 to 516 of method 500. Instead of generally controlling the machine according to the map as provided by method 700, method 800 provides for more specifically controlling the machine according to a route derived from the map.

[0073] In some embodiments of the aforesaid methods and other methods described herein, the computing system is part of the agricultural machine to limit delay in the determining of the attributes. Delay is limited because the determination is performed locally at the agricultural machine. Also, in some examples, the captured images described herein are or include RGB images. In some embodiments, the agricultural machine is a combine harvester or another type of harvester. In some cases, the images are of images of the crop and the field in front of the machine. In some embodiments, the camera is mounted to a front and top portion of the agricultural machine (e.g., see FIGS. 3 and 4). In some embodiments, the camera is mounted such that a field of view (e.g., see field of view 401 shown in FIG. 4) is not limited by an implement (e.g., see feeder 315 shown in FIG. 3) attached to the front of the agricultural machine. In some embodiments, the camera is mounted to the cabin (e.g., see cabin 301) of the machine. And, in some examples, the camera is mounted to the hood (e.g., see hood 303) of the cabin. In some embodiments, the camera includes a wide field of view (e.g., see field of view 401). And, in some instances, the camera includes a wide-angle horizontal field of view (e.g., see field of view 401).

[0074] In some embodiments of the aforementioned methods, the determination of the attributes of the field and the crop in the field in the images (e.g., see images 1102 and 1202) at step 506 or step 706 is based at least on digital signal processing (e.g., see step 902 shown in FIG. 9). Also, in some embodiments, the determination of the attributes is based at least on a computer vision analysis (e.g., see steps 904 to 908 shown in FIG. 9 as well as data enhancement instructions 226 shown in FIG. 2). Further, in some of such examples, the determination of characteristics of the crop in the images is further based on digital signal processing in addition to the computer vision analysis (e.g., see all the steps of FIG. 9 as well as instructions 226). As shown in FIG. 9, in some examples, the digital signal processing (such as at step 902) occurs prior to the computer vision analysis as a pre-processing step to generate enhanced input for the computer vision analysis (such as the analysis at step 906 by computing scheme 907). In some embodiments, as shown in FIG. 9, the method step 506 or 706 includes, at step 906, processing, by the computing scheme 907, the determined attributes of step 506 or 706 or the enhance input of step 902. For example, at step 506 or706, as shown in FIG. 9, further determining attributes of the field and the crop in the images is based on digital signal processing at step 902. In some embodiments, the method includes inputting the enhanced input from the signal processing into an artificial neural network (e.g., see step 904) and the computer vision analysis includes inputting the enhanced input into an ANN (e.g., see step 904 as well as instructions 226). And, the determination of the attributes of the crop and the field is based at least on the output of the ANN (e.g., see step 908). In some cases, as shown in FIG. 9, at step 908, the step 506 or 706 continues with using an output of the computing scheme 907 or a derivative thereof to further determine the attributes of the field and the crop in the images. In some examples, the ANN includes or is part of a deep learning process that detects the different aspects of the crop or is a basis for the detection of the different aspects of the crop (e.g., see instructions 226). Also, in some examples, the deep learning process includes a convolutional neural network (CNN) (e.g., see instructions 226). Also, in some examples, the deep learning process includes a network of convolutional neural networks (CNNs) (e.g., see instructions 226).

[0075] In some embodiments, the computer vision analysis includes inputting aspects of the images or derivatives of aspects of the images into an artificial neural network (ANN) (e.g., see scheme 907), and the determination of the attributes of the crop and the field in the images is based at least on output of the ANN. In some examples, the ANN includes or is part of a deep learning process that determines attributes of the crop and the field or is a basis for the determination of the attributes of the crop and the field. Also, in some examples, the deep learning process includes a convolutional neural network (CNN). And, the deep learning process can include a network of convolutional neural networks (CNNs) and the CNN is one of multiple CNNs of the network.

[0076] Also, the ANN can include or be a part of a deep learning process that determines attributes of the crop and the field or is a basis for the determination of the attributes of the crop and the field, and the deep learning process includes a convolutional neural network (CNN) for semantic pixel-wise segmentation (e.g., see scheme 1104 shown in FIG. 11). In some examples, the use of pixel-wise semantic segmentation includes the use ofSEGNET or another type of a deep convolutional encoder-decoder architecture for image segmentation such as U-NET (which is a network for biomedical image segmentation), fully convolutional network (FCN), DEEPLABV2 or DEEPLABV3, and DECONVNET (e.g., see scheme 1104). In some examples, for an RGB input image (e.g., see images 1102 and 1202 of FIGS. 11 and 12 respectively), the determination provides an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background' (this can be broken down into more specific classes such as 'weeds', 'harvested area', 'swath', etc.) (e.g., see prediction views 1106 and 1206 shown in FIGS. 11 and 12 respectively). Also, in some case, the ANN includes or is part of a deep learning process that determines attributes of the crop and the field or is a basis for the determination of the attributes of the crop and the field, and the deep learning process includes a convolutional neural network (CNN) for crop lodging classification.

[0077] In some embodiments, the implementation of the aforesaid computing schemes or computer vision analysis can be performed by executing instructions 226.

[0078] In some embodiments of the aforesaid methods, values of crop lodging are recorded by the computing system as the machine moves through the field (not depicted). In such embodiments, such values can be compared against satellite images of crop lodging to validate the values.

[0079] In some embodiments, a system or method to detect and map crop lodging areas in an agricultural field is implemented through an agricultural machine or vehicle (e.g., a combine harvester) equipped with a machine location tracking system (e.g., one of many known real-time kinematic RTK or GPS systems, or a combination thereof) as well as equipped with a camera configured to capture images of an area in front of the machine (e.g., an RGB camera). Also, to implement the system or method a computing system is provided locally with the machine or remotely to process data captured by the camera and possibly from additional sensors to detect lodging and used the detected lodging to provide additional functionality such as a visual interface for an operator or input for a control system of the machine. Particular to this disclosure, the computing system also provides mapping of crop lodging areas in the field for the improvement of crop management, estimating yield loss andcounteracting the possible causes for the following year. Such mapping can include the generation of a FMIS map of the crop field and areas affected by crop lodging. For example, mapping may include a FMIS map with sectors and the sectors are scored with respect to an amount of lodging per sector.

[0080] In some embodiments of the aforesaid methods, the machine or agricultural vehicle includes an RGB camera mounted in the front of the vehicle with a wide-angle horizontal field of view (e.g., see camera 302 shown in FIG. 3 as well as view 401 depicted in FIG. 4). The possible mounting points can include the top of a cabin or on the hood of the cabin (e.g., see cabin 301 and hood 303 shown in FIGS. 3 and 4). The camera can capture the images of the crop in front of the vehicle or machine during operation. The images can be processed by a computing system (e.g., see any one of the computing systems shown in FIG. 1 as well as computing system 200 shown in FIG. 2). The processing can occur through a deep learning process for pixel-wise semantic segmentation. In some instances, SEGNET or another type of a deep convolutional encoder-decoder architecture for image segmentation is used in or as the deep-learning process (e.g., see scheme 1104 shown in FIGS. 11 and 12). For an RGB input image (e.g., see images 1102 and 1202 shown in FIGS. 11 and 12 respectively), the deeplearning process can provide an image where each pixel is labelled as either 'lodging area', 'standing crop' or 'background', for example. Based on the known field of view of the camera, its mounting position and rotation with respect to the vehicle, the detected areas can be transformed from camera frame to vehicle frame.

[0081] S ince the vehicle can be equipped with a location tracking system or a machine position tracking system (such as RTK-GPS), the data from the vehicle frame can then be transformed into input for a vehicle control system or other types of functionality such as map generation. In embodiments described herein, the input is used to generate a map providing an indication of lodging levels in the field. Also, since the same area of the field can be visible on several consecutive images, the same map areas can be updated as more information about them becomes available, such as by using Bayesian approaches. Furthermore, a simple visualization system for the machine operator can show the map of the field and displayresults of the deep-learning process or a derivative thereof on the map as the vehicle moves in the field. Such maps and the results of the deep-learning process can be stored for analysis purposes and preparation for the next season as well.

[0082] FIG. 10 illustrates a display 1002 of a user interface device 1000 (e.g., see user interface 216 shown in FIG. 2) displaying an FMIS map 1004 of levels of crop lodging in a crop field based on labeled and geotagged images or determined attributes derived from the images. In general, the map 1004 provides determined crop lodging levels associated with different locations of the field of crops. As shown in FIG. 10, each sector of the map 1004 includes a respective crop lodging level within a corresponding sector or area of the field. Specifically, lodging levels of "1", "2" and "3"are displayed in the map 1004. Levels "2" or above in the map 1004 indicate that lodging detected for a sector exceeds a threshold level. Threshing level per sector is calculated from the determined attributes provided by the methods described herein. The greater the level shown, the greater the amount of lodging in the sector. Sectors with levels of lodging exceeding the threshold are shown with the lodging level surrounded by a dashed-line box as well as graphical elements 1020 are provided in such sectors. Specifically, the FMIS map 1004 shows graphical elements 1020 in the sectors 1008, 1009, 1018, 1019, 1028, and 1029. Sectors 1006 and 1016, for example, do not include lodging levels that exceed "2" and thus, do not include such graphical elements.

[0083] The FMIS map 1004 corresponds to the geotagged images at a plurality of locations corresponding to geographic locations of the field where capturing of the images occurred, and when one of the graphical elements 1020 is selected (e.g., see the selected graphical element 1021 of sector 1018), a corresponding image (e.g., see the corresponding image 1022) of the geotagged images is displayed with the map. In some examples, an identification of a crop lodging area (e.g., see crop lodging area 1023) is included in the corresponding image and the identification includes a graphical identification such as a solid color representing lodging. And, the displaying the corresponding image includes displaying the corresponding image along with a respective RGB image (e.g., see RGB image 1024)corresponding to the area of the crop and the field that is displayed with the identification of the crop lodging area.

[0084] As shown in FIG. 10, map 1004 can show the determined levels of lodging per sector of a field and this can be important because being able to track lodging within a crop field provides a significant agronomic value. The map 1004 allows the operator of a planter or a harvester, or some other agricultural machine to manage farming processes accordingly. Also, in some embodiments, the systems can pull satellite imagery and identify the different levels of lodging from different color spectrums in the satellite imagery which can provide greater confidence in the generated maps described herein.

[0085] Also, in some embodiments, a crop lodging map (e.g., see map 1004) can be combined with a yield map. The advantage of the crop lodging map combined with the yield map over the yield map alone is that the crop lodging map provides additional information on the factors for the yields represented in a yield map. The crop variety map, with or without being combined with the yield map, can also be combined with different types of agriculture informational maps such as a soil quality map, a soil moisture map, a soil pH-level map, and / or a crop or carbon density map. Such combined maps can then be used to analyze a crop and its field and possibly improve farming practices or some other variance that may affect quality of a crop in different situations.

[0086] FIG. 11 illustrates an example RGB image 1102 of a crop field derived from image data captured by a camera of a harvester (such as camera 302) as well as a data processing scheme 1104 that receives input derived from the RGB image 1102 and processes the input to output data for a prediction view 1106 of the image showing determined attributes 1108 of the field (wherein the attributes shown include lodging areas 1110). Referring back to method 500, the images captured at step 502 can include RGB images such as image 1102. And, the determined attributes from step 506 can include attributes such as the attributes 1108 shown in FIG. 11. Also, as shown in FIG. 11, the attributes can include crop lodging areas 1110, the sky, weeds, unharvested crop, harvestedcrop, other attributes of a field or its surroundings and unlabeled or undetermined items of the images.

[0087] FIG. 12 illustrates another example RGB image 1202 of a field derived from image data captured by a camera of a harvester as well as the data processing scheme 1104 shown in FIG. 11 that receives and processes input derived from the RGB image 1202 to output data for a prediction view 1206 of the image showing determined attributes 1108 of the field (wherein the attributes shown do not include lodging areas).

[0088] In another embodiment of the invention the computing system is configured to determine a lodging pattern of at least one of the one or more lodging areas and to present on the map 1004 a graphical indicator of the lodging pattern. The lodging pattern can be determined using an artificial neural network trained to identify different patterns. One type of lodging, called wind lodging, results from wind causing the crop to lean. Wind lodging is typically characterized by a uniformly leaning crop pattern. It is often possible to harvest portions of crop that have sustained wind lodging damage if the harvester approaches the crop from a particular direction and / or using particular machine settings, such as moving into the direction of lean of the stalks at a reduced speed. Another type of lodging, called height lodging, results from the crop growing larger than the stem or roots can support such that the stem or roots yield to the weight of the plant. This type of lodging is typically characterized by a non-uniform leaning pattern of the crop. In other words, each of the plants leans in a different direction. Height lodging can be more problematic than wind lodging because, as the plants do not lean in a uniform direction, it is more difficult to harvest and may not be harvested.

[0089] When the artificial neural network model is trained to identify lodging areas of the field it is also trained to identify the lodging pattern of each lodging area such that the model not only identifies portions of the field where lodging exists but also identifies the type of lodging. When the computing system generates a map with graphical representations of the lodging areas it also includes a graphical indicator of the lodging pattern for each of one or more of the lodging areas. The graphical indicator may include arrows or a series of linesindicating a direction of the lodging or, in the case of height lodging, may simply present an outline of the lodging area with no arrows or lines or, alternatively, may present a particular color or pattern of shading. A heat map could be used to indicate the type of lodging pattern- different colors indicate different lodging patterns / angles. Identifying the lodging pattern of each lodging area enables the machine operator to determine whether he or she should avoid the area during a harvest operation or harvest it in a particular manner.

[0090] Figure 13 illustrates exemplary graphical representations of lodging areas on the map 1004 along with graphical indicators 1300 and 1302 of lodging patterns. The graphical indicators 1300, 1302 include arrows pointing in the direction of crop lodging. In other words, the arrows point in the direction the crop is leaning. Sectors 1018 and 1019 do not include graphical indicators because those sectors do not have a uniform lodging pattern, the lodging in those sectors is height lodging as explained above.

[0091] Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the ways used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, conceived to be a self- consistent sequence of operations leading to a predetermined result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like. It should be borne in mind, however, that these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure can refer to the action and processes of a computing system, or similar electronic computing device, which manipulates and transforms data represented as physical (electronic) quantities within the computing system's registers and memories into other data similarly represented as physicalquantities within the computing system memories or registers or other such information storage systems.

[0092] The present disclosure also relates to an apparatus for performing the operations herein. This apparatus can be specially constructed for the intended purposes, or it can include a general-purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program can be stored in a computer readable storage medium, such as any type of disk including floppy disks, optical disks, CD- ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, each coupled to a computing system bus.

[0093] The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct a more specialized apparatus to perform the methods. The structure for a variety of these systems will appear as set forth in the description herein. In addition, the present disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the disclosure as described herein.

[0094] The present disclosure can be provided as a computer program product, or software, which can include a machine-readable medium having stored thereon instructions, which can be used to program a computing system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some embodiments, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., a computer) readable storage medium such as a read only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory components, etc.

[0095] While the invention has been described in conjunction with the specific embodiments described herein, it is evident that many alternatives, combinations, modifications and variations are apparent to those skilled in the art. Accordingly, the example embodiments of the invention, as set forth herein are intended to be illustrative only, and not in a limiting sense. Various changes can be made without departing from the spirit and scope of the invention.

Claims

CLAIMSWhat is claimed is:

1. A method for mapping crop lodging areas (1110) in a field, comprising: capturing, by a camera (220, 302) mounted to an agricultural machine (300) moving through the field, images (1102, 1202) of the field and a crop in the field (step 502); capturing, by a location tracking system (215), geographic locations of the agricultural machine (300) while capturing the images (1102, 1202) (step 504); determining, by a computing system (200) communicatively coupled to the camera (220, 302), attributes (1108) of the crop and the field in the images (1102, 1202), wherein the determined attributes (1108) comprise crop lodging areas (1110) (step 506); associating, by the computing system (200), the determined attributes (1108) to the geographic locations based on timing of the capturing of the images (1102, 1202) and the geographic locations (step 508); and generating, by the computing system (200), a map (1004) based on the associations between the determined attributes (1108) and the geographic locations (step 510), wherein the map (1004) comprises levels of crop lodging at different sectors (1006, 1008, 1016, 1018) in the field.

2. The method as set forth in claim 1, further comprising - determining, by the computing system (200), a lodging pattern of at least one of the crop lodging areas, the step of generating the map including presenting a graphical indicator of the lodging pattern.

3. The method of claim 1, wherein the capturing of the images (1102, 1202) and the geographic locations as well as the determining of the attributes (1108) occur continuously as the agricultural machine (300) moves through a field.RECTIFIED SHEET (RULE 91 ) ISA / EP4. The method of claim 1, comprising rendering, by a mapping application (228) of the computing system (200), the FMIS map (1004) to be displayed in a graphical user interface (216) (step 512), wherein the display of the map (1004) shows the levels of crop lodging at different sectors (1006, 1008, 1016, 1018) in the field.

5. The method of claim 4, wherein the rendering of the FMIS map (1004) comprises rendering the FMIS map (1004) to be displayed along with a yield map of the field or another type of FMIS map of the field.

6. The method of claim 1, wherein the associations between the determined attributes (1108) and the geographic locations comprises geotagging the images (1102, 1202) which includes adding geographical identification metadata to respective items comprising the images(1102. 1202), and wherein the geographical identification metadata associates the images (1102, 1202) to the geographic locations based on timing of the capturing of the images(1102. 1202) and the geographic locations.

7. The method of claim 6, wherein the associations between the determined attributes (1108) and the geographic locations further comprises associating the determined attributes (1108) and the geotagged images (1102, 1202) in a database (224).

8. The method of claim 6, wherein the FMIS map (1004) shows at least a plurality of graphical elements (1020) corresponding to the geotagged images (1102, 1202) at a plurality of locations corresponding to geographic locations of the field where capturing of the images(1102, 1202) occurred, and wherein when one of the graphical elements (1020) is selected (1021), a corresponding image (1022) of the geotagged images (1102, 1202) is displayed with the map (1004).

9. The method of claim 1, comprising:RECTIFIED SHEET (RULE 91 ) ISA / EPcommunicating, by the computing system (200), the generated FMIS map (1004) over a network (218 or 104) to a controller (220, 230) of the agricultural machine (300) (step 514); and controlling, by the controller (220, 230), one or more operations of the agricultural machine (300) according to the generated FMIS map (1004) (step 516).

10. The method of claim 1, comprising: generating, by the computing system (200), a route based on the FMIS map (1004) (step 602); and controlling, by a controller (220, 230) of the agricultural machine (300), one or more operations of the agricultural machine (300) according to the generated route (step 604).

11. The method of claim 10, wherein the controlling of the one or more operations includes controlling direction and speed of the machine (300) according to the generated route.

12. The method of claim 9, wherein the controlling of the one or more operations includes controlling an implement (315) of the machine (300) according to the generated FMIS map (1004).

13. The method of claim 1, wherein the attributes are first attributes, and wherein the method further comprises: capturing, by a secondary sensor (304) mounted to the agricultural machine (300), secondary information of the field and the crop (step 702); capturing, by the location tracking system (215), geographic locations of the agricultural machine (300) while capturing the secondary information (step 704); determining, by the computing system (200), second attributes (1108) of the crop and the field in the secondary information (step 706), wherein the determined second attributes (1108) comprise crop lodging areas (1110);RECTIFIED SHEET (RULE 91 ) ISA / EPassociating, by the computing system (200), the determined second attributes (1108) to the geographic locations based on timing of the capturing of the secondary information and the geographic locations (step 708); and generating, by the computing system (200), the FMIS map (1004) based on the associations between the determined attributes (1108), the second determined attributes (1108), and the geographic locations (step 710).

14. The method of claim 13, wherein the secondary sensor (304) is a second optical sensor, an ultrasonic transducer, a radio frequency (RF) sensor, lidar, radar, or any type of sensor that senses structure in a crop field without contacting the crop field, or some combination thereof.

15. The method of claim 13, wherein the determination of the first attributes or the second attributes (1108) of the crop and the field in the images is based at least on a computer vision analysis (e.g., see FIG. 9).

16. The method of claim 15, wherein the determination of the first attributes or the second attributes (1108) of the crop and the field in the images is based at least on digital signal processing (step 902), and wherein the digital signal processing occurs prior to the computer vision analysis as a pre-processing step to generate enhanced input for the computer vision analysis.

17. The method of claim 15, wherein the computer vision analysis comprises inputting the enhanced input into an artificial neural network (ANN) (e.g., see scheme 907 and step 904), and wherein the determination of the first attributes or the second attributes (1108) of the crop and the field in the images is based at least on output of the ANN (step 906).

18. The method of claim 17,RECTIFIED SHEET (RULE 91 ) ISA / EPwherein the ANN (scheme 907) includes or is part of a deep learning process that determines the first attributes or the second attributes (1108) of the crop and the field or is a basis for the determination of the first attributes or the second attributes (1108) of the crop and the field, wherein the deep learning process includes a convolutional neural network (CNN) for semantic pixel-wise segmentation (e.g., see scheme 904).

19. The method of claim 1, wherein the determination of the attributes (1108) of the crop and the field in the images is based at least on a computer vision analysis (e.g., see FIG. 9).

20. A system for mapping crop lodging areas (1110) in a field, comprising: a camera (220, 302) mounted to an agricultural machine (300) and configured to capture images (1102, 1202) of the field and a crop in the field; a location tracking system (215) configured to capture geographic locations of the agricultural machine (300) while the camera (220, 302) captures the images (1102, 1202) and the machine (300) is moving through the field; and a computing system (200) communicatively coupled to the camera (220, 302) and the location tracking system (215) and configured to: determine attributes (1108) of the crop and the field in the images (1102, 1202) (instructions 222), wherein the determined attributes (1108) comprise crop lodging areas (1110); associate the determined attributes (1108) to the geographic locations based on timing of the capturing of the images (1102, 1202) and the geographic locations (instructions 224); and generate a farming management information system (FMIS) map (1004) based on the associations (instructions 228), wherein the FMIS map (1004) comprises levels of crop lodging at different sectors (1006, 1008, 1016, 1018) in the field.RECTIFIED SHEET (RULE 91 ) ISA / EP1. A system for mapping crop lodging areas (1110) in a field, comprising: a sensor mounted to an agricultural machine (300) and configured to capture data of the field and a crop in the field; a location tracking system (215) configured to capture geographic locations of the agricultural machine (300) while the sensor captures the data and the machine (300) is moving through the field; and a computing system (200) communicatively coupled to the sensor and the location tracking system (215) and configured to: determine attributes (1108) of the crop and the field in the data (instructions 222), wherein the determined attributes comprise crop lodging areas (1110); associate the determined attributes (1108) to the geographic locations based on timing of the capturing of the data and the geographic locations (instructions 224); and generate a farming management information system (FMIS) map (1004) based on the associations (instructions 228), wherein the FMIS map (1004) comprises levels of crop lodging at different sectors (1006, 1008, 1016, 1018) in the field.RECTIFIED SHEET (RULE 91 ) ISA / EP

Images