Agricultural header control

Abstract

Systems and methods are disclosed for controlling an agricultural header based on movement of crop relative to a portion of the agricultural header. Presence or movement of crop material, such as crop material representative of grain (e.g., ears, heads, or pods of the crop ("EHP")), relative to a harvester header or a portion of a harvester header can be detected, such as by analyzing image data representative of one or more images collected over time. Based on the position or movement of the crop material relative to the header, or both, one or more parameters of the harvester header can be adjusted, for example, to reduce the amount of grain loss.

Description

Technical Field

[0001] This disclosure generally relates to systems and methods for controlling agricultural headers. Background Technology

[0002] Agricultural headers (especially those used in conjunction with combine harvesters) are used to cut crops off the ground and guide the cut crop material to a collection location. Cutting the crop off the ground represents the first step in crop processing, ultimately separating the grain from the non-grain material (MOG). When a header is used in conjunction with a combine harvester, the combine harvester performs subsequent processing operations after the header has cut the crop. Summary of the Invention

[0003] A first aspect of this disclosure relates to a computer-implemented method, executed by one or more processors, for controlling an agricultural header during harvesting based on the movement of crop material relative to the header. The method may include analyzing one or more images containing at least a portion of the agricultural header to detect crop material present in the one or more images; classifying the detected crop material in the one or more images; generating measured distribution data based on the classified crop material; and adjusting the settings of the agricultural header using the measured distribution data.

[0004] A second aspect of this disclosure relates to an apparatus for controlling an agricultural header based on the movement of crop material at the header during harvesting. The apparatus may include one or more processors and a non-transitory computer-readable storage medium connected to the one or more processors and storing programming instructions for execution by the one or more processors. The programming instructions may instruct the one or more processors to analyze one or more images containing at least a portion of the agricultural header to detect crop material present in the one or more images; classify the detected crop material in the one or more images; generate measured distribution data based on the classified crop material; compare the measured distribution data with target distribution data; and adjust the settings of the agricultural header when the measured distribution data does not meet the target distribution data.

[0005] The various aspects may include one or more of the following features. Measured distribution data may be compared with target distribution data. Adjusting the settings of the agricultural header using the measured distribution data may include adjusting the settings of the agricultural header when the measured distribution data does not meet the target distribution data. One or more images of a region of the harvester may be generated during the harvesting operation. Generating the one or more images of the region of the harvester header generated during the harvesting operation may include capturing the one or more images using an image sensor. Analyzing one or more images containing at least a portion of the agricultural header to detect the presence of crop material in the one or more images may include: detecting the type of crop material present in the one or more images. Detecting the type of crop material present in the one or more images may include: detecting at least one of the crop grain component (CGC) of the crop being harvested or the material other than grain (MOG) of the crop being harvested. The measured distribution data may include detected traits of the crop material. It may be determined whether the detected traits of the crop material deviate from the selected conditions by a selected amount. Adjusting the settings of an agricultural header using measured distribution data may include adjusting the header settings when a detected trait of crop material deviates from a selected condition by a selected amount. Analyzing one or more images containing at least a portion of the agricultural header to detect crop material present in the one or more images may include detecting traits of the crop material in the one or more images. Detecting traits of the crop material in the one or more images may include determining the trajectory of the detected crop material relative to the agricultural header. Determining the trajectory of the detected crop material relative to the agricultural header may include determining a vector of the crop material relative to the agricultural header. Determining the vector of the crop material relative to the agricultural header may include determining features of the crop material; determining the boundaries of the harvester header; generating a line from the feature to a position along the boundary; and detecting how the position of the feature changes over time relative to the position along the boundary of the agricultural header based on how the length and position of the line change relative to the position. Analyzing one or more images containing at least a portion of the agricultural header to detect crop material present in the one or more images may include predicting whether the trajectory represents a loss of crop material leaving the header. It may be determined whether the trajectory-based loss exceeds a threshold. Detecting the characteristics of crop material in one or more images may include: determining the rotation of the crop material. At least a portion of the measured distribution data may be displayed. Analyzing one or more images containing at least a portion of an agricultural header to detect the crop material present in the one or more images may include: detecting the crop material based on a contrast between a first color associated with the crop material and a second color associated with the surrounding environment of the crop material.

[0006] Other features and aspects will become apparent by considering the detailed description and accompanying drawings. Attached Figure Description

[0007] Please refer to the accompanying drawings for a detailed description, in which:

[0008] Figure 1 This is a perspective view of an exemplary combine harvester that moves across a field and harvests crops according to some embodiments of this disclosure.

[0009] Figure 2 This is a schematic diagram of an exemplary harvester control system according to some embodiments of the present disclosure.

[0010] Figure 3 This is a flowchart of an exemplary method for controlling an agricultural header according to some embodiments of the present disclosure.

[0011] Figures 4 through 7 are a series of exemplary images taken from an image sensor pointing towards the area above and in front of the cutter, according to some embodiments of the present disclosure.

[0012] Figure 8 This is an enlarged view of the image shown in Figure 4, which illustrates an exemplary coordinate system with the origin located at the tip of the row cell cover.

[0013] Figures 9 to 11 These are a series of images of the area in front of and above an exemplary cutter table, obtained by an image sensor according to some embodiments of this disclosure.

[0014] Figure 12 This is another exemplary image showing an area adjacent to an agricultural header and crop material located near the agricultural header, according to some embodiments of this disclosure.

[0015] Figure 13 This is an exemplary graphical user interface according to some embodiments of the present disclosure.

[0016] Figure 14 This is another exemplary graphical user interface according to some embodiments of the present disclosure.

[0017] Figure 15A yes Figure 15B and Figure 15C A diagram showing how the various parts of a table, as represented by the table, are arranged according to their positions.

[0018] Figure 15B and Figure 15C These are the left and right sides of an exemplary table according to some embodiments of this disclosure, which contains exemplary mitigation actions based on image analysis of crop materials and other data for determining mitigation actions.

[0019] Figure 16 These are exemplary images captured by an image sensor according to some embodiments of the present disclosure, showing an area above an agricultural header.

[0020] Figure 17 This is a block diagram illustrating an exemplary computer system for providing computational functions associated with the algorithms, methods, functions, processes, flows, and programs described in this disclosure, according to some embodiments of the present disclosure.

[0021] Figure 18 It is a superposition of images showing the movement of corn ears from a first time T1 and a second time T2 according to some embodiments of the present disclosure. Detailed Implementation

[0022] To facilitate an understanding of the principles of this disclosure, reference will now be made to embodiments illustrated in the accompanying drawings, and specific language will be used to describe these examples. However, it should be understood that this disclosure is not intended to limit its scope. Any changes and further modifications to the described apparatus, system, or method, and any further application of the principles of this disclosure, are entirely contemplated, as would normally occur to those skilled in the art to which this disclosure pertains. In particular, it is entirely conceivable that features, components, and / or steps described for one embodiment may be combined with features, components, and / or steps described for other embodiments of this disclosure.

[0023] This disclosure relates to controlling a crop header based on the movement of crop material relative to a portion of the header or the position of one crop material relative to another crop material at a location near the header. Specifically, this disclosure describes detecting the presence or movement of crop material (such as crop material representing grain (e.g., ears, heads, or pods of a crop (“EHP”))) relative to the harvester header or a portion of the harvester header. Based on the position or movement of the crop material relative to the header, or both, one or more parameters of the harvester header can be adjusted, for example, to reduce grain loss. The detected crop material and its position relative to the harvester header can indicate grain loss or undesirable operation of the agricultural header. Therefore, adjustments to one or more parameters of the header are performed to improve the movement of the crop material, for example, to reduce grain loss, and thus increase the yield of the area being harvested.

[0024] Figure 1This is a perspective view of an exemplary combine harvester 100 moving across field 102 and harvesting crop 104. In this example, crop 104 is arranged in multiple rows 106. In this example, crop 104 is corn arranged in multiple rows 106. However, the scope of this disclosure covers many other types of crops, whether planted in rows or otherwise. The combine harvester 100 includes a corn header 108 that includes multiple row units 110, each row unit 110 being aligned with a specific row 106 to harvest the crop contained in that row 106.

[0025] Although a combine harvester 100 and a corn header 108 have been described, the scope of this disclosure includes other types of agricultural vehicles and other types of headers. For example, the invention covers self-propelled forage harvesters, stalk-trailer tractors, cotton harvesters, or other agricultural vehicles that transport or otherwise carry headers and other types of headers (such as belt headers) for harvesting crops. Furthermore, although corn, corn ears, and corn kernels are used in the context of the examples described herein, the scope of this disclosure is not limited to corn. Rather, the scope of this disclosure covers many other crop types and their associated kernels.

[0026] The combine harvester 100 includes a harvester control system 112 to control one or more aspects of the combine harvester 100 during harvesting operations. In some embodiments, the harvester control system 112 is a computer-implemented device that receives information, such as sensor data, analyzes the received data, and controls one or more aspects of the combine harvester 100 in response to the analysis. In the example shown, the harvester control system 112 includes one or more sensors 114 that sense the presence of crop material (such as EHP) relative to the header 108. In some embodiments, the area sensor 114 is an image sensor that captures images. The combine harvester 100, the corn header 108, or both may include other sensors. For example, in some embodiments, the corn header 108 includes an impact sensor that detects the force or sound of EHP interacting with the header 108 (e.g., an impact). Sensors 114 encompass sensors operable to detect a portion of the radiation spectrum (such as the visible spectrum, infrared spectrum, or radar spectrum). Thus, sensors include, for example, optical sensors (e.g., cameras, stereo cameras), infrared sensors, lidar, or radar. Sensor 114 is interchangeably referred to as a zone sensor because it captures images of a zone, such as an area near the header. This disclosure also covers other types of sensors operable to obtain images of crop material at a position relative to the header. Furthermore, different types of image sensors can be used in combination, and each sensor is operable to transmit sensed data to the harvester control system 112 for analysis, as described in more detail later. The sensed data is transmitted to the harvester control system 112 via a wired or wireless connection.

[0027] like Figure 1 As shown, one or more sensors 114 are positioned on the combine harvester 100 or header 108 to detect crop material, such as EHP or material other than grain (MOG), at different regions 116, 118, and 120 relative to the header 108. In the example shown, region 116 is the region defined above and in front of the header 108; region 118 is the region adjacent to the lateral side 122 of the header 108; and region 120 is the region below and adjacent to the tail end of the header 108. Regions 116, 118, and 120 represent the 3D space adjacent to the header 108. Therefore, region sensor 206 captures an image of the space represented by regions 116, 118, and 120 around the header 108. Furthermore, the shapes of regions 116, 118, and 120 shown are provided as examples only and are not intended to be limiting. Therefore, in other embodiments, the shape and size of one or more of regions 116, 118, and 120 may differ. Figure 1Examples are shown in the diagram. In some embodiments, the harvester control system 112 can be operated to detect crop material, such as EHP and MOG, using image data acquired by sensor 114 while the crop material is in motion during harvesting. Specifically, using image data from sensor 114, the harvester control system 112 can be operated to detect crop material as it travels relative to the header 108 during harvesting operations. For example, in some cases, one or more sensors in sensor 114 capture images of crop material moving through row unit 110, within the troughs of the cross-type auger conveyor of corn header 108, or at one or more locations contained within the boundaries of corn header 108. The sensed crop material information is used to determine the characteristics of the sensed crop material, such as whether the sensed crop material has escaped collection or may escape collection, resulting in material loss, such as material loss to the ground.

[0028] Image data collected by sensor 114 can be presented in a variety of ways. For example, in some embodiments, the image data forms a single or series of images with a 2D coordinate system. In other embodiments, the image data is a stereo image with a 3D coordinate system. In other embodiments, the image data is lidar point cloud data with a Cartesian or spherical coordinate system. Furthermore, in some cases, the image data is lidar point cloud data enhanced with camera pixel color data and including a 3D coordinate system. In some cases, the image data is radar 3D image data. In other embodiments, the image data can be these data types or a combination of one or more other data types.

[0029] Figure 2 This is a schematic diagram of an exemplary harvester control system 112. In some embodiments, the harvester control system 112 takes the form of a computer system, such as computer system 1700, which will be described in more detail below. Additional details of the harvester control system 112 (such as processor 202 and memory 204) are included below in the case of computer system 1700.

[0030] As shown, the harvester control system 112 includes a controller 200. The controller 200 includes a processor 202 communicatively connected to a memory 204. The memory 204 communicates with the processor 202 and is used to store programs and other software and information (such as in the form of data). The processor 202 is operable to execute programs and software and to receive and send information to the memory 204. Although a single memory 204 and a single processor 202 are shown, in other embodiments, multiple memories, processors, or both may be used. Although the processor 202 and memory 204 are shown as local components of the controller 200, in other embodiments, one or both of the processor 202 and memory 204 may be located in a remote location.

[0031] The harvester control system 112 also includes one or more area sensors 206 (which may be similar to sensor 114) that capture images or data representing material passing through one or more areas near the header. For example, the area sensor 206 may be located on the header (such as... Figure 1 On the header 108 shown in the figure, or on a combine harvester (such as...) connected to the header. Figure 1 On the combine harvester 100 shown in the figure, so as to capture the area relative to the header (such as also as Figure 1 Images of areas 116, 118, and 120 are shown. As explained above, area sensors (such as area sensors 114 or 206) include image sensors, such as cameras (e.g., monochrome and stereo cameras), radar (such as Doppler radar), lidar, and other devices and techniques operable to capture images or otherwise detect the presence of crop material (such as EHP and MOG), and particularly to detect crop material associated with a portion of the header. Image sensors include charge-coupled devices (CCDs) and active pixel sensors, such as complementary metal-oxide-semiconductor (CMOS) sensors. Data generated by area sensor 206 is collectively referred to herein as “image data,” even though data provided from some sensors within the scope of area sensor 206 (such as radar data and lidar data) may not conventionally be considered image data. Image data provides a representation of material (such as crop material) present in the area being sensed at the moment the data is collected. The one or more area sensors 206 communicate with controller 200 and transmit image data to controller 200.

[0032] The harvester control system 112 also includes a Global Navigation Satellite System (GNSS) antenna 208 or is communicatively connected to a GNSS antenna 208. The GNSS antenna 208 receives geospatial positioning data from satellite navigation systems such as the Global Positioning System (GPS), BeiDou Navigation Satellite System (BDS), Galileo, GLONASS, the Indian Regional Navigation Satellite System (IRNSS), Navigation Indian Constellation (NavIC), and the Quasi-Zenith Satellite System (QZSS). The received geospatial positioning data is used for georegistration, for example, of data received from one or more other sensors, such as area sensor 206. For example, georegistered data is used to georegulate image data received from area sensor 206, such that, for example, the location of a specific event represented in the image data is associated with the image data, enabling mapping of the image data. In some cases, georegistered data is also used to determine the location of agricultural vehicles such as combine harvesters, track their routes, or map the routes that agricultural vehicles will take.

[0033] The exemplary header control system 112 also includes a user input device 210 or connected to a user input device 210. The user input device 210 can be any of a keyboard, keypad, joystick, mouse, scanner, camera, microphone, button, knob, or other type of input device operable to receive user input. The harvester control system 112 also includes a display 212 or connected to a display 212. The display 212 is operable to display information to a user, such as one or more images received from the area sensor 206. In some embodiments, the display 212 also displays data, such as loss data obtained by processing the data received from the area sensor 206. The information displayed on the display 212 can be provided via a graphical user interface (GUI) 214. In some cases, the display 212 is a touchscreen display and operates as an input device in addition to displaying information. The information displayed in the GUI 214 is presented, for example, using text (e.g., numerically), graphics, symbols, colors, patterns, flashing objects or text, or text or objects with different intensities. However, information can also be presented in other ways. For example, in some cases, information such as one or more messages displayed on GUI 214 can be output as speech. Additional details of the GUI within the scope of this disclosure are provided below.

[0034] The harvester control system 112 also includes a database 216 or is communicatively connected to a database 216. In some cases, the database 216 is in the form of a mass storage device, which includes the mass storage devices described below. The database 216 may be located at a local location, such as on the combine harvester or other vehicle containing the harvester control system 112, or it may be located at a remote location. The database 216 stores data for later use, such as by the controller 200, and particularly by the processor 202 of the controller 200.

[0035] Memory 204 stores data, such as image data 218 received from the one or more area sensors 206. As explained above, image data 218 received from area sensors 206 includes images or other data indicating the presence or absence of objects (e.g., crop material) in specific areas (such as areas 116, 118, and 120 described above). Memory 204 also includes measured component distribution data 220, target component distribution data 222, and supplementary harvesting data 224. In some embodiments, memory 204 also includes other types of data. Examples of supplementary data include yield, grain cleanliness, grain quality, grain loss, grain damage, and component or vehicle operating speed. Exemplary uses of supplementary data will be described in more detail below. Memory 204 may also include other data, such as current and previous actuator settings and header parameters, and geospatial location data.

[0036] Processor 202 executes programs, such as image analyzer program 224. Controller 200 uses data to determine or predict crop material characteristics (e.g., crop loss) on or near the header during harvesting. For example, controller 200 uses image data 218, measured distribution data 220, and target distribution data 222 to determine whether grain loss exceeds a selected grain loss level. As described in more detail below, in some embodiments, processor 202 of controller 200 executes image analyzer program 224, which uses image data 218 to generate measured distribution data 220. The measured component distribution data 220 is compared with target distribution data 222 to determine whether one or more aspects of the crop material on the header meet criteria contained in the target distribution data 222 (e.g., whether grain loss detected from the header is at an acceptable level). For example, if grain loss is higher than a selected level, controller 200 generates one or more control signals to actuate one or more actuators of the header. Actuation of one or more header actuators alters header parameters, which in turn changes (e.g., reduces) the level of grain loss from the header. In some cases, if the determined grain loss is below the selected level, the current header parameters are maintained.

[0037] The exemplary header control system 112 also includes other sensors 226 or is communicatively connected to other sensors 226. Other sensors 226 include, for example, a grain loss sensor 228, a grain quality sensor 230, a grain yield sensor 232, a grain cleanliness sensor 234, a harvester under-harvest image sensor 236, and a rear harvester image sensor 238. The grain loss sensor 228 includes a plate impact sensor and a piezoelectric sensor that sense the impact of grain falling from the harvester onto a surface and uses this information to determine grain loss. The grain quality sensor 230 includes, for example, an image sensor that captures one or more images of the harvested grain. The one or more images are analyzed to detect properties of the harvested grain, such as color or size, thereby determining grain quality. The grain cleanliness sensor 234 includes, for example, an image sensor that captures one or more images of the harvested grain, and these images are analyzed to detect the presence of MOG or other materials in the harvested grain. The rear harvester sensor 238 includes, for example, an image sensor that captures one or more images of an area near the tail end of the harvester. The captured images are analyzed, for example, to detect the mass of the residue dispersion as the residue leaves the combine harvester, or parameters associated with the stockpile formed by the combine harvester.

[0038] For example, the grain cleanliness and grain quality information provided by the grain cleanliness sensor 234 and the grain quality sensor 230, respectively, may be affected by MOGs. That is, the presence of MOGs in the crop material sensed by the grain cleanliness sensor 234 or the grain quality sensor 230 affects the data output by these sensors. Therefore, these supplementary data output by the grain cleanliness sensor and the grain quality sensor can represent the amount of MOGs contained in the crop material captured by the header. This supplementary data can be used in conjunction with image data from sensor 114 (especially when image data from sensor 114 is being used to detect and measure MOGs) to change header settings (such as stalk roller speed or the cover plate spacing of the row units of the corn header), for example, to change the amount of MOGs held by the header.

[0039] In another example, the data captured by the rear harvester sensor 238 can also be a representation of the MOG held by the combine harvester and thus by the header. This data can be used in conjunction with image data from sensor 114 to change header settings (such as stalk roller speed or the cover plate spacing of the row units of the corn header) to, for example, change the amount of MOG held by the header.

[0040] The harvester control system 112 includes actuators 240 or is communicatively connected to actuators 240. Actuators 240 are associated with the header to change the state or parameters of the header, or actuators 240 are associated with a combine harvester or other agricultural vehicles connected to the header and similarly used to change the state or parameters of the header. Actuators 240 include a cover width actuator 242, a rotary actuator 244, a reciprocating actuator 246, a compressible component actuator 248, and a speed control actuator 250.

[0041] Rotary actuator 244 includes actuators operable to operate, for example, stalk rollers (e.g., stalk rollers of a corn header), augers (e.g., cross-augers), stalk shredders, reels (e.g., reels of a belt conveyor header), rotary drums, and belts (e.g., belts of a belt conveyor header). For example, reciprocating actuator 246 includes an actuator operable to reciprocate the cutting bar of a belt conveyor header. Compressible component actuator 248 includes, for example, a pneumatic cylinder or a hydraulic accumulator. Compressible components such as pneumatic cylinders or hydraulic accumulators are used as pneumatic springs or dampers, for example, to control the movement of the header or to control the momentary movement of the cover plates of the row unit, for example, when the cover plates move in response to engagement with incoming crop. Cover plate width actuator 242 changes the spacing between the cover plates of the stalk roller assembly of the corn header in response to a signal from controller 200. Speed ​​control actuator 250 includes actuators operable to control the speed of components, such as the speed of rotating or linear components. Speed ​​control actuator 250 includes actuators for changing the speed of the header reel, conveyor pulley, stalk roller, cutting bar, or another component. Speed ​​control actuator 250 also includes actuators for changing the speed of an agricultural vehicle (e.g., a combine harvester) through the field in response to a signal from controller 200. One or more of the speeds may be related. For example, the speed of the combine harvester may be related to the speed of the stalk roller. If the speed of the stalk roller decreases, the speed of the combine harvester through the field may also decrease to improve the flow of crop material through or along the header, for example, to prevent crop material from accumulating on the header or to reduce MOG intake. If the speed of the combine harvester through the field decreases, then the speed of the stalk roller may similarly decrease. Alternatively, if one of the speed of the combine harvester through the field or the speed of the stalk roller increases, then the other of the speed of the combine harvester or the speed of the stalk roller may similarly increase.

[0042] Figure 3This is a flowchart of an exemplary method 300 for controlling an agricultural header, such as by changing one or more settings of the header in response to sensed crop material properties. At 302, image data (such as image data 218) is collected from one or more sensors (such as area sensor 206) monitoring one or more areas near the header. In some embodiments, additional data is collected. For example, additional data may be received from one or more other sensors (such as one or more of sensor 226).

[0043] At 304, image data is analyzed, for example, by the harvester control system 112, to generate crop information associated with the crop material present in the image data. The crop information includes, for example, the type of crop material being detected and the characteristics of the crop material present in the image data.

[0044] Figures 4 through 7 are a series of images 400, 500, 600, and 700 taken from area sensors pointing towards the area above and in front of the header 402, respectively. This series of images is a sequential arrangement in chronological order (i.e., from the earliest time shown in Figure 4 to the latest time shown in Figure 7). Images 400 through 700 are received by a controller (such as controller 200) of the harvester control system, and an image analyzer (such as image analyzer 224) running on the processor (such as processor 202) of the controller analyzes the images. The image analyzer performs image analysis to determine crop material, such as EHP, MOG, or grain. Grain is a single seed that forms EHP in an aggregated manner. Specifically, the image analyzer identifies and tracks the EHP, portions of EHP, and grain (hereinafter referred to as the “crop grain component”, CGC) present in the collected images and generates metrics related to header performance, particularly header grain loss performance. For example, CGC is intended to cover cotton bolls as well as other parts of the crop that the harvester aims to capture.

[0045] Example metrics generated by the image analyzer include metrics for classifying crop material. For example, in some implementations, the image analyzer identifies a single kernel, ear of corn, broken ear of corn, pod, head, or MOG. The image analyzer uses image analysis techniques to determine crop material. For example, the image analyzer may use one or more of the following image analysis techniques: two-dimensional (2D) object recognition, three-dimensional (3D) object recognition, image segmentation, motion detection (e.g., single-particle tracking), video tracking, optical flow, 3D pose estimation, pattern recognition, and object recognition, to name just a few. These exemplary image analysis techniques are not exclusive. Therefore, other types of image analysis techniques may be employed to detect the presence of crop material in an image and the movement of crop material between images. Furthermore, in some implementations, classification methods using machine learning algorithms are also used to determine features, such as features of different types of crop material or headers, and the movement of detected objects between images. Exemplary machine learning algorithms include, but are not limited to, supervised learning algorithms, unsupervised learning algorithms, semi-supervised learning algorithms, and reinforcement learning algorithms.

[0046] In some implementations, neural networks (including those using deep learning) can also be used to identify and classify crop material present in image data. Exemplary neural networks include perceptual neural networks, feedforward neural networks, convolutional neural networks, recurrent neural networks, and autoencoders, to name just a few. Other types of neural networks are also within the scope of this disclosure.

[0047] Additionally, the image analyzer determines the location of the identified crop material within the image. For example, the image analyzer can be operated to detect whether the identified crop material is attached to a crop bead; located on a surface (such as the surface of a header); in the air; or on the ground. Further, the image analyzer can be operated to determine whether the crop material was present on the ground before harvesting or was brought onto the ground due to harvesting travel. This function will be described in more detail below. The image analyzer also determines the location (e.g., a position vector), motion (e.g., a motion vector), or both of the crop material within an image, between multiple images, or in a combination thereof. An example of a motion vector (e.g., in...) is... Figure 18 (As shown in the figure) it covers the speed and direction of motion, and is determinable within an image, for example, by using motion blur of an object; is determinable across multiple images by using changes in position between images, for example; or is determinable by a combination of these techniques.

[0048] Returning to Figures 4 through 7, the image analyzer has identified a type of crop material. Here, the image analyzer has identified corn ear 402. Crop material is determined using one or more characteristics, such as size, shape, color, pattern, or other characteristics associated with a particular type of crop material. Here, the image analyzer may determine corn ear 402 based on the elongated shape of the corn ear, the color of the corn ear (e.g., yellow), the pattern associated with the corn ear, a combination of these, or based on other criteria. Therefore, in some embodiments, features of other types of crop material using other types of image analysis techniques can be used to detect the crop material type, such as CGC or MOG. Furthermore, in other embodiments, the image analyzer may have features or a set of features for identifying other types of crop material present in the same image. For example, the image analyzer may have a set of features programmed to identify individual kernels present in the image. Those identified kernels may be indicated or emphasized in the image in a manner different from the corn ear in the image. Thus, an operable image analyzer can identify different types of crop material present in an image and label these different types of crop material in different ways to distinguish one type of crop material from another. In particular, in some implementations, an image analyzer can be operated to detect different types of crop material and to detect whether the identified crop material is collected by the header or falls to the ground, as described in more detail below.

[0049] Returning to this example, once the corn cob 402 is identified, an image analyzer can be operated to determine the characteristics of the corn cob 402. For example, in some embodiments, the image analyzer determines one or more of the following: size, shape, features (e.g., centroid), centroid, major axis, minor axis, or other properties of the corn cob 402. For example, in the example shown, the image analyzer determines the size of the corn cob 402 (e.g., length and width), the centroid of the corn cob 402, the major axis of the corn cob 402 (e.g., a line extending along the length of the corn cob 402), and the minor axis of the corn cob 402 (e.g., a line extending along the width of the corn cob 402). In some cases, the major axis, minor axis, or both can be represented by the centroid. In other embodiments, other properties may be used in addition to or excluding other properties. Furthermore, although the centroid of the corn cob 402 has been described, in other embodiments, one or more other features of the crop material, such as CGC, are identified. In some embodiments, the identified one or more features are used to define a reference marker. The reference marker can be used in a manner similar to that of a principal axis, such as to determine the rotation of the corn cob or other crop material, as described in more detail below. The examples in this disclosure describe the use of a spindle to detect rotation. However, the spindle is only used as an example for illustrative purposes. Therefore, other reference markers can be identified or defined and used to detect rotation or other properties of crop materials such as CGC.

[0050] As shown in Figures 4 through 7, an image analyzer, another application, or device generates a line 404 representing the main axis of the corn cob 402. In this example, the main axis 404 extends through the centroid 405 of the corn cob 402, although the centroid is not visually identified using markers. In the example shown, the main axis 404 corresponds to the longitudinal axis of the corn cob 402. The image analyzer also determines a reference location 406 corresponding to a portion or feature of the header 408. In this example, the reference location 406 is a static location corresponding to a identifiable feature, such as the tip 410 of the row unit cover 412 of the header 408. In other embodiments, other features of the header 408 may be used as location 406. In this example, the reference location 406 is presented on a display (such as display 212) in a circular shape. Other types of markers may be used to determine the reference location 406 in the presented image. Exemplary markers include markers with different shapes, colors, patterns, symbols, and characters. In some cases, text or objects are used as the mark type for determining reference position 406 on the display, and furthermore, text or objects with different intensities are used as the mark type for determining reference position 406 on the display.

[0051] The image analyzer generates a position vector line 414 extending from a reference position 406 to the centroid 405 of the corn cob 402. The position vector line 414 is used to determine how the corn cob 402 moves between a series of images from Figures 4 to 7. The position vector line 414 and... Figure 8 The coordinate system 800 shown is used in combination. The coordinate system 800 shown is provided by way of example only. Therefore, other coordinate systems (including radial, spherical, cylindrical, and Cartesian coordinate systems) are within the scope of this disclosure. Furthermore, the orientation of the described coordinate system 800 is also provided by way of example only. Other coordinate system orientations may also be used, and are within the scope of this disclosure. Furthermore, although coordinate system 800 is shown as a 2D coordinate system, the scope of this disclosure also covers the use of 3D coordinate systems to define the position and movement of objects within and across an image.

[0052] Figure 8Figure 4 is another view of the image shown, illustrating a coordinate system 800 with an origin 802 located at a reference position 406. The coordinate system 800 is used at least in part to determine the angle 803 of the position vector line 414. In the example shown, the 0° position is shown at 804 and at the top of the y-axis 806; the 90° position is located to the right of the x-axis 808; the 180° position is shown at the bottom of the y-axis 806; and the 270° position is shown to the left of the x-axis 808. In this way, the angle 803 of the position vector line 414 is determined, and this angle 803 is used to quantify how the corn cob 402 moves between images. As a result, the image analyzer is able to determine, for example, the change in the object's position between images, and thus determine the object's motion vector. The image analyzer can use this information, other information, or a combination of this information and other information to predict or detect whether the corn ear 402 will be captured by the header 408 and become part of the harvested kernels, or whether the corn ear 402 will eventually fall to the ground and become part of the kernel loss.

[0053] Figure 18 An exemplary coordinate system 1800, similar to coordinate system 800, is shown in a set of images. Object 1802 is positioned at two locations relative to coordinate system 1800. Each of these locations 1804 and 1806 corresponds to the time when the image of object 1802 was captured, i.e., T1 and T2, respectively. Therefore, in Figure 18 In this example, the images of object 1802 taken at times T1 and T2 are superimposed relative to coordinate system 1800. A first position vector 1808, extending from the origin 1810 to the centroid 1812 of object 1802, is the position vector at time T1. A second position vector 1814, extending from the origin 1810 to the centroid 1812 of object 1802, represents the position vector at time T2. In this example, the images of object 1802 taken at times T1 and T2 are consecutive images. The distal threshold 1816 of the chute 1820 is also... Figure 18 As shown in the image.

[0054] A first angle 1822 is measured relative to the 0° position 1824 at the top of the y-axis 1826 and the first position vector 1808. A second angle 1828 is measured relative to the 0° position 1818 and the second position vector 1814. The motion vector line 1830 extends between the centroid 1812 of the object 1802 at the position at time T1 and the centroid 1812 of the object 1802 at the position at time T2. Because the motion vector 1830 points towards the chute 1820, and because the object does not exceed the far-end threshold 1816 at time T2, the image analyzer predicts that the object 1802 is captured by the chute 1820.

[0055] Back Figure 8While a 2D coordinate system 800 is used in this example, a 3D coordinate system can be used in a similar manner in other examples to detect position and velocity, and to predict the outcome of an object's motion (e.g., whether the crop material is ultimately captured by the header). The third dimension of an object's position in an image can be estimated based on its relative size within the image. For example, if the object is larger in the image, it is closer to the image sensor. Conversely, if the object is smaller, it is farther from the image sensor. The image analyzer uses the relative size of objects within an image or across multiple images to detect the object's position in 3D space. Therefore, although the examples contained herein are made in a 2D coordinate system, the scope of this disclosure covers the use of, for example, a 3D coordinate system to determine the position of an object in 3D space. In the case of a 3D coordinate system, angle 803 would be the angle within 3D space. In other examples, one or more stereo cameras are used to determine the position of an object (such as crop material) in three-dimensional space.

[0056] like Figure 8 As shown, position vector line 414 defines an angle 803 of approximately 170° in coordinate system 800. Between images, the motion vector of the object is determined using changes in the length and angle 803 of position vector line 414. Furthermore, if the time between images is known, the velocity vector of the object can also be determined.

[0057] Preferably, the corn cob 402 moves toward the header 408, which will be indicated as a motion vector, and as explained above, the motion vector can be determined using the position vector line 414 and the angle 803. Figure 8 In this context, angle 803 is close to 180°. For example, an angle 803 in the range of 120° to 135° can be used to indicate crop material that will be captured by the header and thus interpreted as harvested grain. Other angle ranges of angle 803 can also be used. For example, in some embodiments, an angle range between 110° and 180° is used to identify CGCs as harvested grain. This angle range of angle 803 can vary depending on the location of the corn ear 402 in the image (e.g., due to the viewing angle associated with the placement of the area sensor). In this example case, an angle of 180° indicates a direction toward the header 408. Furthermore, by utilizing angles close to 180° (such as angles within ±20° of 180°) and as the position vector line 414 increases in length, the image analyzer can determine or predict that the corn ear will be successfully captured by the header and thus become part of the harvested grain, rather than a grain loss. In some cases, the value of angle 803 can be used as a confidence factor for predicting whether a CGC has been captured by the header. For example, in Figure 8In the case of coordinate system 800, angles in the range of 150 to 180 indicate crop material that may be captured by the header. Angles in the range of 150 to 120 still indicate that the crop material may be captured by the header, but represent a lower level of confidence. Similarly, as the angle decreases, the confidence level of crop material captured by the header continues to decrease. However, angle 803 is one of several factors that can be considered when determining whether crop material has been captured by the header.

[0058] However, in Figures 4 through 7, the angle 803 defined by the position vector line 414 decreases over time (indicated by the progression of the images from Figure 4 to Figure 7). In Figure 4, the angle is approximately 170°. However, this angle 803 decreases over time to approximately 50° (i.e., less than 90°) in Figure 7, indicating that the corn ear 402 is moving in front of the reference position 406. This indicates that the corn ear 402 may have fallen to the ground and is considered a kernel loss. Furthermore, the length of the position vector line 414 decreases in conjunction with the decreasing angle 803. As a result, the image analyzer determines that the corn ear 402 is moving away from the header 408 rather than towards it, and as the angle 803 decreases to less than 90°, the image analyzer considers the corn ear 402 as a kernel loss. In the example shown in Figure 7, since reference position 406 defines a portion of header 408 (which is the farthest point of header 408 shown in the image), an angle 803 greater than 90° indicates that the corn ear 402 extends further away from header 408 and thus away from header 408. In some embodiments, the angle range interpreted as kernel loss is 160° to 200°. Similarly, this angle range can vary depending on the position of the corn ear 402 within the image, due to, for example, the perspective view represented by the image and the position of the crop material within the image, or, for example, the 3D position of the corn ear 402 relative to header 408 when using a 3D coordinate system.

[0059] Other factors may also be considered when predicting whether crop material will be captured by the header. For example, the future position based on the determined trajectory (e.g., using a motion vector or velocity vector) may be used alone or in combination with angle 803 to predict whether corn ear 402 will be captured by header 408. Furthermore, the combine harvester's speed, direction, or both may also affect whether corn ear 402 is captured by header. Therefore, in some embodiments, the combine harvester's direction, speed, or both are used to predict whether crop material will be captured by header.

[0060] As explained above, an image analyzer or another application or device generates a line 404 along the long axis of the corn cob 402. In this example, line 404 passes through the centroid 405. Rotation of the corn cob 402 can be detected based on changes in length or orientation, or both, from one image to the next. Changes in the orientation of line 404 relative to coordinate system 800 can be used to indicate rotation of the corn cob 402, such as in the plane of the image. Changes in the length of line 404 provide an indication of rotation in a plane other than the plane of the image.

[0061] In some implementations, an operable image analyzer determines the location of different types of crop material based on the color present behind the crop material. For example, when the header has a specific color, such as green, the image analyzer can determine that the detected crop material is on the header or positioned above the header when the background color is green or another color associated with the header. Similarly, if the background color is brown or the same color as the ground, the image analyzer can determine that the crop material is positioned on or above the ground. If the background color is blue or the same color as the sky, the image analyzer can determine that the detected crop material is in the air. Other background colors associated with other objects can be similarly used to determine the location of the detected crop material in space. Furthermore, the relative size of the material within the image or the use of image shifting (where the area sensor is a stereo camera) can be used to determine how close the crop material is to the area sensor, and by extension, how close the crop material is to other objects present in the image. Using this location and color information, the operable image analyzer determines the position of the detected crop material relative to other objects or features represented in the image.

[0062] In some implementations, the image analyzer performs similar analysis on other types of crop material. Therefore, in some implementations, the image analyzer determines and tracks motion, and ultimately determines whether the crop material is captured by the header or falls onto the ground of various different types of crop material.

[0063] Additional capabilities of the image analyzer include determining whether crop material (such as CGCs (e.g., EHP)) is attached to the bead (i.e., its condition on the bead), or is in contact with or residing on the harvester, in the air, or on the ground. If the crop material is determined to be on the ground, the image analyzer can also determine whether the crop material was on the ground before harvesting or due to the harvesting operation. For example, if the crop material is determined to be on the ground and moves in a downward direction in the image over several image steps (e.g., from the top edge 418 to the bottom edge 420 in the image of Figure 4, and disappears from the image upon reaching the header 408), the image analyzer determines that this particular crop material was on the ground before the harvesting operation. The crop material disappearing upon reaching the header 408 is a result of the crop material passing under the header 408.

[0064] Using position vector lines (such as position vector line 414) and other information (such as the configuration of the header and agricultural vehicles (such as combine harvesters) connected to the header), an operable image analyzer can determine various details about crop material shown in an image or a series of images. For example, an image analyzer can be operated not only to determine the different types of crop material present in an image, but also to determine whether the crop material is stationary (e.g., the length and angle of the position vector line remain constant in two or more consecutive images); whether the crop material is moving toward the header or a part thereof (such as the cross-shaped helical conveyor of the header) (e.g., the length of the position vector line associated with the crop material is increasing, and the reference angle relative to the defined coordinate system is within a selected range of 180°); whether the crop material is moving away from the header (e.g., the length of the position vector line is decreasing (if not moving further away from the reference point), or the position vector line is increasing and the associated reference angle is between 0° and 90° or between 270° and 360°); the rotation and amount of rotation of the crop material (e.g., changes in the length or angle of a line extending along the axis of the crop material); or the impact angle of the crop material relative to another object. Impact can be determined based on changes in orientation between images, and the impact angle can be determined based on the position vector line and the reference angle, as explained above.

[0065] Figures 9 to 11Another sequence of images of the area in front of and above the header 1000, obtained by a region sensor (such as region sensor 206), is shown. The header 1000 is operating to harvest crop 1002 while being advanced through field 1004. As shown in the images, an image analyzer (such as image analyzer 224) determines the CGCs. Here, the CGCs are ears of corn 1006, 1008, 1010, and 1012. In this example, the image analyzer determines each of ears of corn 1006 to 1012 by generating a contour 1014 around each ear, and the image analyzer also generates a line 1016 extending along the main axis of each ear and passing through the centroid of each ear. The image analyzer is also operable to determine the motion of the ears of corn relative to the header 1000 using the principles discussed herein. Although the reference position is not in Figures 9 to 11 As shown in the figure, however, a reference position similar to reference position 406 can be used, for example, to detect the movement of corn ears.

[0066] In this sequence of images, the image analyzer detects the bouncing of the corn ear 1006 based on, for example, changes in a position vector line relative to a reference position, which may resemble position vector line 414. As explained above, in some embodiments, changes in length and angle relative to a reference position and coordinate system from one image to another are particularly used to determine changes in the position of the crop material relative to the header 1000, etc. Figure 9 and 10 The image shown allows the image analyzer to determine that the corn ear 1006 is moving upwards and toward the cover 1018 of the header 1000, as indicated by arrow 1020. Figure 10 Image to Figure 11 The image analyzer was similarly able to determine from the image that corn ear 1006 had moved downwards as indicated by arrow 1022. In contrast, from... Figures 9 to 10 The lines 1011 that determine the main axis of the corn ear 1010 maintain a similar orientation. This orientation represents the ideal characteristics of the harvested corn ear and causes the corn ear 1010 to move efficiently into the auger conveyor 1024 for subsequent movement along the header 1000. Therefore, the image analyzer is operable to detect the characteristics of the CGCs at the header 1000, including the bounce of the corn ear or other CGCs (such as EHP). A controller, such as controller 200, is operable to incorporate this bounce data as part of measured distribution data and use the measured distribution data to ultimately control one or more settings of the header 1000.

[0067] The bouncing of corn ears may indicate that excessive force was applied to the crop or CGC during harvesting. Excessive force (i.e., force exceeding the force required for successful harvesting of the CGC) can cause the corn ear to spill off the header, resulting in kernel loss. Excessive force can also cause pulverization losses (such as hulling), in which the CGC (such as the corn ear) impacts a portion of the header and causes one or more kernels to separate from the ear. These kernels typically fall to the ground as kernel loss. Furthermore, the bouncing of CGCs (such as the corn ear) increases the time the CGC remains on the header, and therefore, causes delays in the transport and, in some cases, disposal of the CGC. This delay can lead to the accumulation of crop material on the header, which can further degrade header performance and increase kernel loss.

[0068] The image analyzer can also identify and determine kernels from the ear, head, or pod. For example, the image analyzer can be operated to detect corn kernels 1200, such as kernels that have been separated from the ear during harvest, such as... Figure 12 As shown. In some embodiments, the number of separated grains is estimated from the discolored portion of the detected ear, which represents missing grains. For example, a portion 1202 of a corn ear 1204 has a color different from yellow. The image analyzer interprets the color change to identify missing grains. As a result, the number of missing grains is estimated by, for example, estimating the size of portion 1202 based on the average size of the corn kernels. Figure 12 Another ear of corn 1206 was also identified. In some embodiments, different crop components are identified differently. For example, in some embodiments, grains or other individual kernels are identified in one way (e.g., surrounded by a line pattern different from the line pattern used to identify another part of the crop), while the whole ear of corn is identified in another way, and a portion of the ear of corn is identified in yet another way. Thus, the ear of corn is identified using a line with a different color than the line used to identify individual grains. In addition to line color, different line patterns may also be used. Furthermore, any other type of method may be used to identify different crop materials, such as using flashing identifiers, symbols, or characters.

[0069] Therefore, the image analyzer is operable to identify different types of crop material and detect and monitor the position and movement of these crop materials in the image relative to the header or some other feature. For example, the image analyzer is operable to determine the position of the crop material (e.g., using position vector lines or background color), particularly its position relative to the header; direction of movement, particularly relative to the header (e.g., forward or backward movement relative to the header); bounce relative to the header; stationary position relative to the header (e.g., when the crop material is resting on the header); and velocity of the crop material (e.g., based on the degree of positional change of the crop material between one image and the next). Using information collected by the image analyzer from the captured images, the image analyzer 224 or some other software application (whether located on the header control system 112 or remotely communicatively connected to the header control system 112) characterizes this information, such as in the form of various parameters, such as Figure 3 As indicated at position 306 in the document.

[0070] Information can be categorized based on, for example, the type of crop material detected. For instance, crop material can be characterized based on its material type. In the case of maize, crop material can be identified by individual grains (such as...). Figure 12 The reference numeral 1200 shown in the figure indicates that the crop material is characterized by ears of corn, portions of ears of corn (e.g., determined by the length of the ear which is less than the selected length), husk material, or other MOGs. Therefore, in some cases, crop material can be classified by ear, pod, or head, and by individual kernels of the crop. In some embodiments, the detected crop material can also be characterized in various ways. For example, the crop material can be further classified based on parameters such as direction of travel, speed, whether an impact has occurred (which may indicate that the crop material bounced on the header), and whether the crop material was ultimately lost from or captured by the header.

[0071] In 308, the characterized information is transformed into a measurement distribution. For example, a measurement distribution is generated for each parameter of the characterized information. The measurement distribution can be generated as a percentage (e.g., the number of bounced EHPs in each defined area), a count (e.g., the number of EHPs moving away from the header), or in any other desired manner. For example, the distribution for each parameter can be determined based on area, such as hectares, time period, crop row, or based on a portion of the header, such as each row unit of a corn header. However, the distribution can be determined on other bases, and is within the scope of this disclosure. Reference Figure 2 The measured distribution data is stored as measured distribution data 220 in the memory 204 of the controller 200. However, the measured distribution data can be stored at any desired location.

[0072] In step 310, the measured distribution of each parameter is compared to the target distribution used for the corresponding parameter. The target distribution for each parameter is pre-selected, and a threshold can be defined for each parameter. (Reference) Figure 2 For example, target distribution data is stored as target distribution data 222 in the memory 204 of controller 200. However, target distribution data can be stored at any desired location. At 312, when the measured distribution value of the measured distribution data does not meet the criteria contained in the target distribution data, one or more settings of the header are changed. For example, when the measured distribution value of a parameter meets or exceeds a defined threshold, a change is made to a component or system applied to the header or an agricultural vehicle connected to the header. Therefore, when the measured distribution of a parameter contained in the measured distribution data does not meet the corresponding criteria in the target distribution data, a controller (such as controller 200) generates a signal, for example, to cause a change in the position of an actuator, thereby changing the header settings. In some cases, different actuators are actuated to adjust different settings of the header or an agricultural vehicle connected to the header, depending on the type of parameter.

[0073] Furthermore, in some embodiments, a standard provided in the target distribution data is selected such that changes to the header settings are implemented before grain loss occurs during harvesting by the header. Therefore, this disclosure provides active control of the header or agricultural vehicle to reduce or eliminate grain loss, for example, due to the contact between the crop and the header.

[0074] In other implementations, the measured distribution data is used to control one or more settings of the cutter head, for example, by adjusting one or more actuators, without comparison to target distribution data. For example, in some cases, the controller (such as controller 200) directly uses the measured distribution to control one or more settings of the cutter head.

[0075] In some cases, additional data can be used to adjust header settings in response to a comparison between measured distribution data and target distribution data. For example, data such as grain loss data, grain quality data, grain cleanliness data, rear image data, component performance data, and component usage data can be used. Different data types can be obtained using, for example, grain loss sensors, grain quality sensors, grain cleanliness sensors, and image sensors. Other data may be available at other times, such as during maintenance of the header or agricultural vehicles connected to it. Exemplary uses of other data types are provided above.

[0076] For example, in some implementations, machine performance data or usage data includes the number of actuation cycles experienced by a component of the header or the duration of operation of the component since installation. In this case, a controller (such as controller 200) uses the performance data or usage data to provide recommendations. For example, component wear information can be used to limit or reduce the amount of wear experienced by a component, and thus extend the component's service life. Components for which wear information can be utilized include, for example, stem rollers or sickles. In other cases, performance information is used to address material creep or component loosening. For example, over time, the collection chain may become slack due to loosening, and the controller can adjust the chain to reduce or eliminate loosening.

[0077] Figure 13 This is an exemplary graphical user interface (GUI) 1300. GUI 1300 may be similar to GUI 214 described above. GUI 1300 is provided on a display such as display 212 and displays crop information collected during the harvesting operation. GUI 1300 displays output, for example, a comparison between measured distribution data and target distribution data. For example, on display portion 1302, pre-harvest grain loss is represented. In the example shown, the percentage of crop identified as having pre-harvest grain loss is presented.

[0078] In some implementations, pre-harvest grain loss is determined using the predicted yield of the area being harvested and data representing grain loss identified as pre-harvest grain loss. In some implementations, actual yield data from the previous season is used as the predicted yield data. In other implementations, the predicted yield is determined by combining pre-existing data with sensor data acquired during the harvesting operation. In still other implementations, the predicted yield data is generated, for example, based on yield data expected to be acquired during the harvesting operation for the area being harvested.

[0079] In some cases, a pre-harvest grain loss sensor is used to detect pre-harvest grain loss. In some implementations, a zone sensor (such as zone sensor 206) is operable to sense pre-harvest grain loss, and an image analyzer is operable to determine the presence of pre-harvest grain loss in a captured image. As explained above, pre-harvest grain loss can be determined by how the detected grain moves in the image. For example, referring to Figure 4, if grain is first detected at the top edge 418, indicating that the grain was already present on the ground before the header engaged with the crop, the image analyzer (or other application) is operable to determine that the detected grain is pre-harvest grain loss. At 1302, using the grain loss determined to be pre-harvest grain loss and predicted yield information, the percentage of predicted yield determined to be pre-harvest harvest loss is determined and presented to the user.

[0080] The harvested crop (e.g., CGC) that has been identified as exhibiting abnormal or unwanted traits (such as those determined by comparing measured distribution data with target distribution data) is displayed on the display portion 1304 of the GUI 1300. Using the techniques described above, an image analyzer, such as image analyzer 224 or other application or device, is operable to detect how individual crop components (e.g., individual grains, EHP, or MOG) move relative to the header. As explained, this data is categorized and accumulated into measured distribution data.

[0081] Display section 1304 is divided into segments 1306, 1308, and 1310. Although three segments are provided in the example shown, additional or fewer segments may be included in other implementations. Each segment displays information corresponding to a different type of anomalous trait (which can be presented in various different ways, as described in more detail below). For example, segment 1308 represents the percentage of CGC moved forward relative to the cutter head. In some cases, in Figure 8 In this case, forward movement is indicated by a position vector line that increases in length and angle between 0° and 90° or between 270° and 360°. Forward movement indicates crop material with the potential to eventually fall to the ground. In the example shown, 1.8% of the predicted yield (determined using detected crop material (e.g., CGC) with known grain size) was determined as forward movement relative to the header.

[0082] Paragraph 1306 represents the percentage of CGCs identified as bouncing on the header. Here, 0.2% of the yield (e.g., CGCs) is identified as bouncing on the header. In some cases, the actual yield value is used, and in others, the predicted yield value is used. The bouncing crop material is identifiable, as described above. Paragraph 1310 represents the percentage of crop grain material identified as exhibiting undesirable behavior relative to the total predicted yield, and is the sum of the other paragraphs 1306 and 1308. In the example shown, the total percentage of grains identified as exhibiting undesirable or abnormal behavior is 2%. In some implementations, each paragraph represents a different type of undesirable crop trait, and the paragraph representing the sum of all types of undesirable traits is omitted.

[0083] Display section 1312 displays portions of the harvested crop that have been identified as exhibiting favorable traits. In some embodiments, portions exhibiting favorable traits are determined by comparing measured distribution data with target distribution data, as described above. For example, in some embodiments, crop gluten (CGCs) that do not bounce or move forward relative to the header or do not fall to the ground are considered to have favorable traits during harvesting. In the example shown, the portion representing favorable traits is shown as a percentage, such as 95.3%.

[0084] In some cases, display segments 1302, 1304, and 1312 are datasets associated with individual row units or portions of the header. Presenting data in this manner provides the ability to adjust various aspects of the header at a granular level, such as controlling individual aspects of row units to improve overall header performance. In other embodiments, the presented data may be presented as data associated with the number of passes through the field (e.g., a single pass). Furthermore, the data may be presented in any number of other ways or combinations of different ways.

[0085] Figure 14 This is another example GUI 1400 including a first display portion 1402 and a second display portion 1404. The first display portion 1402 includes a plurality of display segment groups 1406, wherein each display segment group 1406 contains display segments 1408, 1410, and 1412, respectively, similar to the display segments 1302, 1304, and 1312 described above. In some embodiments, display segments 1408, 1410, and 1412 in each display segment group 1406 represent a type of trait of crop material during harvesting operations. For example, in some cases, display segments 1408, 1410, and 1412 in each display segment group 1406 represent undesired traits, the composition of desired traits, and pre-harvest losses, as explained above. In other embodiments, different display segments may represent other types of information associated with header operations, including information associated with comparisons between measured distribution data and target distribution data.

[0086] In some implementations, different segment groups 1406 represent the performance of each row unit of the cutter. In this case, the performance of each row unit is monitored, and each row unit can be adjusted independently based on the monitored performance. In some implementations, multiple groups of rows are used to monitor the performance of the cutter. Therefore, in some cases, the performance of the cutter is monitored on a component-by-component basis (e.g., based on the performance of each row unit) or on a grouped component basis of the cutter (e.g., by group of row units).

[0087] The second display section 1404 contains cause and mitigation information. For example, information related to the causes of undesirable crop harvesting performance is displayed in the second display section 1404. In some cases, the causes of undesirable performance are determined based on analysis of measured distribution data, comparison between measured distribution data and target distribution data, or both. In some cases, the measured distribution data includes data from multiple rows of headers and within a selected time period. In some implementations, the analysis of this data is performed using numerical analysis, rules, one or more neural networks, or machine learning algorithms (alone or in combination). In some cases, other types of data are also used to determine cause and mitigation information. For example, local data (i.e., locally stored data), non-local data (e.g., remotely stored data), recent data, historical data, or combinations of these data types may be used in conjunction with measured distribution data and target distribution data, or both, to determine cause and mitigation information. In some cases, this data is based on, for example, design data, simulation data, digital twin data, field test data, machine data, fleet data, or crop data.

[0088] Based on the above analysis, one or more predicted causes of undesirable performance or traits of the crop material are identified. These one or more predicted causes are displayed on the second display portion 1404. Furthermore, one or more mitigation actions generated in response to these predicted causes are also presented in the second display portion 1404. In some embodiments, the mitigation action represents a change to the header (e.g., a change to one or more settings of one or more components of the header) to address one or more predicted causes, thereby correcting the identified performance defects or undesirable traits.

[0089] In some implementations, the mitigation action is performed automatically. In some cases, the user (such as a remote or local operator) is notified about the automatically performed mitigation action. The notification may be visual, auditory, or tactile. For example, in some cases, information such as one or more pieces of information related to the mitigation action may be presented as voice output. The output may also be presented in other ways. In other implementations, the mitigation action is performed upon user approval, such as through user input. In other implementations, the user (such as an onboard operator) is able to view predictive cause information and the determined mitigation action, and adjust machine settings via user input device 1414 (such as a switch, dial pad, touchscreen, microphone, or other type of device). In the example shown, input device 1414 is part of a touch-sensitive display, defined to perform the desired operation, such as receiving user input.

[0090] Figure 15A yes Figure 15B and Figure 15CA schematic diagram showing how the various parts of Table 1500, which are collectively represented, are arranged according to their positions. Figure 15B and Figure 15C A portion of Table 1500 is shown. Table 1500 contains exemplary mitigation actions based on image analysis of crop materials (such as CGC (e.g., EHP)) and other data used to determine mitigation actions. In some embodiments, other data may be omitted when determining mitigation actions. Thus, in some embodiments, mitigation actions are determined based on a comparison between measured distribution data and target distribution data, without using other data.

[0091] Column 1502 identifies the image analysis results obtained in determining one or more mitigation actions. Column 1504 identifies the distribution parameters of the detected crop material (e.g., CGC). That is, column 1504 identifies the distribution parameters describing the unfavorable traits of the detected crop material. Although not provided in Table 1500, in some embodiments, the causes associated with the unfavorable trait are predicted based on the analysis, as explained above. Column 1506 defines one or more mitigation actions determined based on the detected unfavorable trait. The mitigation action is determined using the predicted causes of the undesirable trait. For example, based on the prediction that a specific setting of a header component leads to an undesirable trait, a mitigation action is determined to change the setting of the header component to reduce or eliminate the undesirable trait. In some embodiments, the undesirable trait may be the result of multiple header settings, and a change in each of the multiple settings may be performed as a mitigation action to reduce or eliminate the undesirable trait. Column 1508 identifies other or supplementary data used in conjunction with the image analysis results data to determine one or more mitigation actions.

[0092] Referring to line 1510, the image analysis results identify undesirable traits as CGC material falling from one or more areas of the header, and specifically EHP crop material. For example, in some cases, material falling from the collection area of ​​a row unit of the header is identified as an undesirable trait. The associated distribution parameter that triggers the generation of mitigation actions is the selected amount of EHP falling into a designated area on the header. When the selected amount of EHP is detected falling into the designated area, the determined mitigation action is to reduce the speed of the agricultural vehicle traveling through the field. Additionally, supplementary data used to determine the mitigation action is the current speed of the agricultural vehicle traveling through the field. In line 1512, the image analysis results identify undesirable traits as ears of grain wedged onto the cover plate of the header. The distribution parameter is the ears of grain on the cover plate having a selected amount of movement thereon (e.g., a selected rate of movement or a selected amount of displacement). In this example, when the distribution parameter criterion is met, the mitigation action is to reduce the speed of the stem roller of the associated row unit and adjust the amount of separation of the cover plate. The supplementary data used to determine the relief action in line 1512 are the current stem roller speed and the current separation of the cover at opposite ends (i.e., front and rear) of the cover.

[0093] In line 1514, the image analysis results identify the undesirable trait as stalks entering the header via the cross-shaped auger conveyor. The associated distribution parameter criterion is the detection of MOGs entering the combine harvester connected to the header. The mitigation action is to increase the speed of the stalk rollers when a selected number of MOGs enter the combine harvester, or when a selected number of stalk material is detected at the cross-shaped auger conveyor. Supplementary data for determining the mitigation action is the current stalk roller speed of the row unit. In line 1516, the image analysis results detect unacceptable hulling at the hull. The distribution parameter is the detection of a selected number of individual grains on the header, and the mitigation action is to adjust the hull separation and the roller speed when a selected number of individual grains are detected on the header. Supplementary data for determining the mitigation action is the current hull separation, grain damage data, and grain loss data.

[0094] In line 1518, image analysis identifies the unwanted trait as the movement of an entire stalk into the combine harvester. The distribution parameter is the selected number of complete stalks detected moving from the row unit to the cross-shaped auger conveyor at the header. The mitigation action is to reduce the speed of the row unit's collection chain or the speed of the row unit's stalk roller, and the supplementary data used are the current collection chain speed and the current stalk roller speed. In line 1520, image analysis identifies the unwanted trait as unwanted vibration of one or more stalks or stems, and the distribution parameter is the selected number of EHP separations from the stalk or stem caused by the unwanted vibration. The mitigation action is to sharpen the sickle used to cut the stalk or stem from the ground, and the supplementary data used is the number of hours since the sickle blade was last sharpened or replaced.

[0095] The principles described herein also apply to harvesting platforms used in intercropping, where the field being harvested comprises two or more different crops arranged such that the harvesting platform encounters at least two of the different crop types during each pass of the field during harvesting. An exemplary type of intercropping within the scope of this disclosure is relay cropping, in which one or more crop types are harvested during a harvesting operation while one or more other intercropping types are not harvested during the same operation.

[0096] In the case of overlapping operations, image sensors (such as area sensor 206) collect image data, and the collected image data is analyzed, for example, by an image analyzer 224. The image analyzer identifies one or more crop types that will not be harvested, and in some cases, ignores image data associated with these crop types. Thus, in some cases, the image analyzer is used to act on image data associated with the crop types being harvested, while ignoring image data associated with the crop types not being harvested. Therefore, the image analyzer is operable to distinguish between crop material associated with one or more crop types being harvested and crop material of one or more crop types not being harvested, and in some cases, specifically acts on image data associated with one or more crop types being harvested.

[0097] In some implementations, the intercropping of the different crop types is arranged in adjacent rows in the field. Using this crop arrangement, Figure 14 One or more of the display segments 1406 shown in the diagram that are associated with rows of unharvested crops may be grayed out, blank, absent, or otherwise lack information.

[0098] In some implementations, an image analyzer assesses any damage suffered by unharvested crop types and adjusts one or more settings of the header to reduce or eliminate the damage. For example, image data obtained from an image sensor can detect unharvested crop material identified as being in the harvested crop material stream from the header. For example, soybeans and wheat can be intercropped together. During the harvesting operation, only wheat will be harvested. However, green soybean seed material from soybean plants can be detected in the harvested golden wheat crop material stream from the header by an image sensor (such as area sensor 206). Therefore, in some implementations, an image analyzer (such as image analyzer 224) uses, for example, color recognition and comparison of the colors of the detected crop material to distinguish unharvested crop material from the harvested crop type. If the amount of green soybean seed material meets a selected criterion (e.g., if the amount of green soybean seed material exceeds a threshold amount), a controller (such as controller 200) performs a mitigation action. In some cases, mitigation actions include, for example, raising the header, changing the header angle, changing the harvester speed, or altering the properties of the header reel. Other mitigation actions can be performed to reduce the harvesting or trapping of bead material associated with unharvested bead types.

[0099] Figure 16An exemplary image captured by an image sensor (such as area sensor 206) is shown. The captured image shows an area above header 1600. Header 1600 is harvesting a first crop type 1602. Here, the first crop type 1602 is wheat, and the wheat crop has a first color (e.g., golden yellow), and is engaged with and harvested by header 1600. A portion 1604 of header 1600 is aligned with a second crop type 1606, which is arranged in row 1608 between rows 1610 of the first crop type 1602. Here, the second crop type 1606 is soybean, and the soybean has a green color. Portion 1604 extends over the crop of the second crop type 1606 to prevent the crop of the second crop type 1606 from being engaged and damaged by header 1600. For example, portion 1604 may bend or press the crop of the second crop type 1606 away from header to prevent the crop of the second crop type 1606 from engaging with the cutting part of header 1600. Region 1612 of the captured image shows the flow of crop material across the header 1600. By analyzing region 1612, the presence of crop material from the second crop type 1606 in the crop material flow can be determined, for example, based on differences in color between the two different crop types. Mitigation actions are avoided if no crop material from the second crop type 1606 is detected. In some embodiments, supplementary data from other sensors (such as a crop impact sensor for detecting crop loss) are used in conjunction with the image sensor to detect damage to the second crop type 1606.

[0100] In other examples of intercropping, two or more crop types (e.g., corn and soybeans) are harvested simultaneously. During the simultaneous harvesting of different crop types, a single stream of harvested material can be generated. In the case of generating a single stream of crop material, an image analyzer (such as image analyzer 224) is configured to identify crop material for each crop type, such as EHP. The image analyzer analyzes each crop type and generates relevant information for each crop type. As described above, the generated information, such as in the form of measured distribution data, is analyzed (e.g., by comparison with target distribution data) to adjust one or more settings of the header.

[0101] In similar Figure 13 and Figure 14 In the case of the example GUIs 1300 and 1400, additional information, such as crop type, is also displayed. In some implementations, in the second display section (which may be similar to...),... Figure 14 The cause and mitigation information displayed in the second display section 1404 may include one or more suggestions on how to adjust the header to reduce or eliminate grain loss, or information on actions to be taken automatically.

[0102] In some implementations, remedial actions performed on the header (e.g., changes to one or more settings) take into account the market value of crop loss, the estimated amount or quality of crop loss, or other parameters. In some implementations, a GUI such as GUI 214 or GUI 1300 provides information to a user, such as an operator, allowing the user to define how and whether adjustments are applied to the header based on, for example, the type of information to be considered. For example, the information considered can change the corrective actions (e.g., setting adjustments) taken on the header for a detected operational defect. For example, a specific market price of the lost crop might lead to one type of corrective action for a detected defect, while different market prices might lead to different corrective actions based on the same detected operational defect. Similarly, for a particular detected operational defect, differences in the type of information considered by a controller (such as controller 200) might lead to different corrective actions. In some implementations, the GUI also provides controls that allow the user to select how adjustments associated with corrective actions are applied to the header, such as manual controls, automatic controls, consented automatic controls, etc. In some implementations, the GUI also includes controls for adjusting actuators of agricultural vehicles, such as actuators associated with grain separation, cleaning, and residue discharge of combine harvesters.

[0103] In some implementations, headers for simultaneously harvesting two or more crop types can generate separate material streams for each crop type. Image sensors (such as area sensor 206) collect image data including both material streams. An image analyzer, similar to image analyzer 224, analyzes the different material streams in the image data in a manner similar to that described above. In some cases, the different material streams are analyzed separately. In some cases, the image analyzer is also operable to detect the presence of a first crop material in the material stream associated with different crop materials. Target distribution data may include criteria for changing header settings based on a selected amount of material of the first crop type present in the material stream of the second crop type, and vice versa. Based on the material type of the crop type (e.g., EHP or MOG) and in cases where this material enters the material streams of different crop types, a controller (such as controller 200) can control one or more actuators (such as one or more actuators 240) to reduce the deposition of material of one crop type into the material streams of different crop types. For example, in the case where the first crop type is soybean and the second crop type is corn, a header operable to simultaneously harvest two crop types may experience, for example, corn ears being deposited in the soybean material stream. Image data of this phenomenon is captured by an image sensor (such as area sensor 206), and an image analyzer (such as image analyzer 224) analyzes the image data to determine where or how the corn ears are introduced into the soybean material flow, or vice versa. Corrective actions are determined based on, for example, a comparison between measured distribution data and target distribution data. The corrective actions may be actuation of one or more actuators in a portion of the crop type being processed, specific to the header.

[0104] Without limiting the scope, interpretation, or application of the appended claims in any way, the technical effects of one or more of the exemplary embodiments disclosed herein are to reduce grain loss and increase harvesting-related efficiency. In some cases, grain loss can be avoided by actively monitoring crop material properties associated with the header and adjusting the header to prevent grain loss.

[0105] Figure 17This is a block diagram of an exemplary computer system 1700, according to some embodiments of the present disclosure, for providing computational functionality associated with the algorithms, methods, functions, processes, flows, and programs described in the present disclosure. The computer 1702 shown is intended to encompass any computing device, such as a server, desktop computer, laptop / notebook computer, wireless data port, smartphone, personal data assistant (PDA), tablet computing device, or one or more processors of these devices, including physical instances, virtual instances, or both. Computer 1702 may include input devices capable of accepting user information, such as a keypad, keyboard, and touchscreen. Furthermore, computer 1702 may include output devices that can transmit information associated with the operation of computer 1702. The information may include digital data, visual data, audio information, or a combination of information. The information may be presented in a graphical user interface (UI) (or GUI).

[0106] Computer 1702 may act as a client, network component, server, database, persistence, or component of a computer system for performing the subjects described in this disclosure. The illustrated computer 1702 is communicatively connected to network 1730. In some embodiments, one or more components of computer 1702 may be configured to operate within different environments, including cloud-based environments, local environments, global environments, and combinations thereof.

[0107] At a higher level, computer 1702 is an electronic computing device operable to receive, transmit, process, store, and manage data and information associated with the described subject matter. According to some embodiments, computer 1702 may also include, or be communicatively connected to, an application server, an email server, a web server, a cache server, a streaming data server, or a combination of servers.

[0108] Computer 1702 can receive requests from client applications (e.g., those executing on another computer 1702) via network 1730. Computer 1702 can respond to received requests by processing them using software applications. Requests can also be sent to computer 1702 from internal users (e.g., from a command console), external parties (or third parties), automation applications, entities, individuals, systems, and computers.

[0109] Each component of computer 1702 can communicate using system bus 1703. In some implementations, any or all components of computer 1702 (including hardware or software components) can interface with each other or with interface 1704 (or a combination of both) via system bus 1703. The interface can use application programming interface (API) 1712, service layer 1713, or a combination of API 1712 and service layer 1713. API 1712 may include specifications for routines, data structures, and object classes. API 1712 may be language-independent or language-dependent. API 1712 may refer to a complete interface, a single function, or a collection of APIs.

[0110] Service layer 1713 can provide software services to computer 1702 and other components (whether shown or not) communicatively connected to computer 1702. All service consumers using this service layer can access the functionality of computer 1702. Software services such as those provided by service layer 1713 can provide reusable defined functionality through defined interfaces. For example, the interface can be software written in JAVA, C++, or a language that provides data in Extensible Markup Language (XML) format. Although shown as an integrated component of computer 1702, in alternative embodiments, API 1712 or service layer 1713 can be a separate component associated with other components of computer 1702 and other components communicatively connected to computer 1702. Moreover, without departing from the scope of this disclosure, any or all portions of API 1712 or service layer 1713 can be implemented as a successor or submodule of another software module, enterprise application, or hardware module.

[0111] Computer 1702 includes interface 1704. Although in Figure 17 While shown as a single interface 1704, two or more interfaces 1704 may be used depending on the specific needs, expectations, or particular implementation and described functionality of computer 1702. Computer 1702 may use interface 1704 to communicate with other systems connected to network 1730 (whether shown or not) in a distributed environment. Generally, interface 1704 may include, or be implemented using, software or hardware (or a combination of software and hardware) logic operable to communicate with network 1730. More specifically, interface 1704 may include software supporting one or more communication protocols associated with the communication. Thus, the hardware of network 1730 or the interface is operable to transmit physical signals for communication both within and outside of the illustrated computer 1702.

[0112] Computer 1702 includes processor 1705. Although in Figure 17 While shown as a single processor 1705, two or more processors 1705 may be used depending on the specific needs, expectations, or particular implementation and functions described for computer 1702. Generally, processor 1705 can execute instructions and manipulate data to perform operations of computer 1702, including operations using algorithms, methods, functions, processes, flows, and procedures as described in this disclosure.

[0113] Computer 1702 also includes database 1706, which can store data for computer 1702 and other components (whether shown or not) connected to network 1730. For example, database 1706 may be an in-memory database, a conventional database, or a database storing data consistent with this disclosure. In some embodiments, database 1706 may be a combination of two or more different database types (e.g., a hybrid in-memory database and a conventional database), depending on the specific needs, expectations, or particular implementation of computer 1702 and the functions described. Although in Figure 17 While shown as a single database 1706, two or more databases (of the same type, different types, or combinations thereof) may be used depending on the specific needs, expectations, or particular implementation and functionality of the computer 1702. Although database 1706 is shown as an internal component of the computer 1702, in alternative implementations, database 1706 may be external to the computer 1702.

[0114] Computer 1702 also includes memory 1707, which can store data for a combination of components (whether shown or not) used by computer 1702 or connected to network 1730. Memory 1707 can store any data consistent with this disclosure. In some embodiments, depending on the specific needs, expectations, or particular implementation of computer 1702 and the functions described, memory 1707 may be a combination of two or more different types of memory (e.g., a combination of semiconductor memory devices and magnetic memory devices). Although in Figure 17 While shown as a single memory 1707, two or more memories 1707 (of the same type, different types, or combinations thereof) may be used depending on the specific needs, expectations, or particular implementation and functionality of the computer 1702. Although memory 1707 is shown as an internal component of the computer 1702, in alternative embodiments, memory 1707 may be external to the computer 1702.

[0115] Application 1708 may be an algorithmic software engine that provides functionality based on the specific needs, expectations, or particular implementation of computer 1702 and the described functions. For example, application 1708 may act as one or more components, modules, or applications. Furthermore, although shown as a single application 1708, application 1708 may be implemented as multiple applications 1708 on computer 1702. Additionally, although shown as being inside computer 1702, in alternative implementations, application 1708 may be outside computer 1702.

[0116] Computer 1702 may also include power supply 1714. Power supply 1714 may include a rechargeable or non-rechargeable battery that can be configured to be user-replaceable or user-non-replaceable. In some embodiments, power supply 1714 may include power conversion and management circuitry, including recharging, standby, and power management functions. In some embodiments, power supply 1714 may include a power plug to allow computer 1702 to be plugged into a wall outlet or power source to, for example, power computer 1702 or recharge a rechargeable battery.

[0117] There may be any number of computers 1702 associated with or outside the computer system containing computer 1702, wherein each computer 1702 communicates via network 1730. Furthermore, without departing from the scope of this disclosure, the terms "client," "user," and other suitable terms may be used interchangeably. Moreover, this disclosure contemplates that many users may use one computer 1702, and that one user may use multiple computers 1702.

[0118] Implementations of the described subject matter may include one or more features individually or in combination.

[0119] For example, in a first embodiment, the computer-implemented method includes: analyzing multiple images containing at least a portion of an agricultural header to detect crop material present in the images; classifying the detected crop material in the multiple images; generating measured distribution data based on the classified crop material; comparing the measured distribution data with target distribution data; and adjusting the settings of the agricultural header when the measured distribution data does not meet the target distribution data.

[0120] The aforementioned and other described embodiments may each optionally include one or more of the following features:

[0121] The first feature, which may be combined with any of the following features, also includes one or more instructions that can be executed by a computing system to generate multiple images of an area of ​​the harvester header during a harvesting operation.

[0122] The second feature, which can be combined with any of the preceding or following features, wherein generating multiple images of an area of ​​the harvester header during the harvesting operation includes capturing the multiple images using an image sensor.

[0123] The third feature, which can be combined with any of the preceding or following features, wherein analyzing multiple images containing at least a portion of an agricultural header to detect crop material present in the images includes: detecting the type of crop material present in the images.

[0124] The fourth feature, which may be combined with any of the preceding or following features, wherein detecting the type of crop material present in the image includes detecting at least one of crop grain component (CGC) of the crop being harvested or material other than grain (MOG) of the crop being harvested.

[0125] The fifth feature, which can be combined with any of the preceding or following features, wherein analyzing multiple images containing at least a portion of an agricultural header to detect crop material present in the images includes: detecting the characteristics of the crop material in the multiple images.

[0126] The sixth feature, which may be combined with any of the preceding or following features, wherein detecting the characteristics of crop material in the image includes determining the trajectory of the detected crop material relative to the agricultural header.

[0127] The seventh feature, which may be combined with any of the preceding or following features, wherein determining the trajectory of the detected crop material relative to the agricultural header includes: determining the vector of the crop material relative to the agricultural header.

[0128] The eighth feature, which may be combined with any of the preceding or following features, wherein determining the vector of crop material relative to the agricultural header includes: determining the centroid of the crop material; determining the boundary of the harvester header; generating a line from the centroid to a position along the boundary; and detecting how the position of the centroid relative to the position along the boundary of the agricultural header changes over time based on how the length and position of the line change relative to the position.

[0129] The ninth feature, which can be combined with any of the preceding or following features, includes detecting the traits of crop material in the image by determining the rotation of the crop material.

[0130] The tenth feature, which may be combined with any of the preceding or following features, wherein determining the rotation of the crop material includes: determining the principal axis of the crop material and detecting changes in the length or orientation of the principal axis from one image to the next image to define the rotation of the crop material.

[0131] The eleventh feature, which may be combined with any of the preceding or following features, wherein analyzing the plurality of images containing at least a portion of the agricultural header to detect crop material present in the images includes: predicting whether a trajectory represents a loss of crop material leaving the header, and wherein comparing the measured distribution data with target distribution data includes: determining whether the trajectory-based loss exceeds a threshold defined in the target distribution data.

[0132] The twelfth feature, which may be combined with any of the preceding or following features, wherein the measured distribution data includes the detected traits of the crop material, and wherein comparing the measured distribution data with target distribution data includes determining whether the detected traits of the crop material deviate from the selected amount of conditions defined in the target distribution data.

[0133] The thirteenth feature, which can be combined with any of the preceding or following features, includes adjusting the settings of the agricultural header when the measured distribution data does not meet the target distribution data, such that the settings of the agricultural header are adjusted when the detected traits of the crop material deviate from the selected amount defined in the target distribution data.

[0134] The fourteenth feature, which may be combined with any of the preceding features, wherein analyzing multiple images containing at least a portion of an agricultural header to detect crop material present in the images includes: detecting crop material based on a contrast between a first color associated with the crop material and a second color associated with the surrounding environment of the crop material.

[0135] In a second embodiment, a non-transitory computer-readable medium storing one or more instructions executable by a computer system to perform operations including: analyzing a plurality of images containing at least a portion of an agricultural header to detect crop material present in the images; classifying the detected crop material in the plurality of images; generating measured distribution data based on the classified crop material; comparing the measured distribution data with target distribution data; and adjusting the settings of the agricultural header when the measured distribution data does not meet the target distribution data.

[0136] The aforementioned and other described embodiments may each optionally include one or more of the following features:

[0137] The first feature, which may be combined with any of the following features, also includes one or more instructions executable by a computer system to generate multiple images of an area of ​​the harvester header during a harvesting operation.

[0138] The second feature, which can be combined with any of the preceding or following features, wherein generating multiple images of an area of ​​the harvester header during the harvesting operation includes capturing the multiple images using an image sensor.

[0139] The third feature, which can be combined with any of the preceding or following features, wherein analyzing multiple images containing at least a portion of an agricultural header to detect crop material present in the images includes: detecting the type of crop material present in the images.

[0140] The fourth feature, which may be combined with any of the preceding or following features, wherein the type of crop material present in the detected image includes at least one of the following: the crop grain component (CGC) of the crop being harvested or the material other than grain (MOG) of the crop being harvested.

[0141] The fifth feature, which may be combined with any of the preceding or following features, wherein analyzing the plurality of images containing at least a portion of an agricultural header to detect crop material present in the images includes: detecting the characteristics of the crop material in the plurality of images.

[0142] The sixth feature, which can be combined with any of the preceding or following features, includes detecting the characteristics of crop material in the image by determining the trajectory of the detected crop material relative to the agricultural header.

[0143] The seventh feature, which may be combined with any of the preceding or following features, wherein determining the trajectory of the detected crop material relative to the agricultural header includes: determining the vector of the crop material relative to the agricultural header.

[0144] The eighth feature, which may be combined with any of the preceding or following features, wherein determining the vector of crop material relative to the agricultural header includes: determining the centroid of the crop material; determining the boundary of the harvester header; generating a line from the centroid to a position along the boundary; and detecting how the position of the centroid relative to the position along the boundary of the agricultural header changes over time based on how the length and position of the line change relative to said position.

[0145] The ninth feature, which can be combined with any of the preceding or following features, includes detecting the traits of crop material in the image by determining the rotation of the crop material.

[0146] The tenth feature, which may be combined with any of the preceding or following features, wherein determining the rotation of the crop material includes: determining the principal axis of the crop material and detecting changes in the length or orientation of the principal axis from one image to the next image to define the rotation of the crop material.

[0147] The eleventh feature, which may be combined with any of the preceding or following features, wherein analyzing multiple images containing at least a portion of an agricultural header to detect crop material present in the images includes: predicting whether a trajectory represents a loss of crop material leaving the header, and wherein comparing the measured distribution data with target distribution data includes: determining whether the trajectory-based loss exceeds a threshold defined in the target distribution data.

[0148] The twelfth feature, which may be combined with any of the preceding or following features, wherein the measured distribution data includes the detected traits of the crop material, and wherein comparing the measured distribution data with target distribution data includes determining whether the detected traits of the crop material deviate from the selected amount of conditions defined in the target distribution data.

[0149] The thirteenth feature, which can be combined with any of the preceding or following features, includes adjusting the settings of the agricultural header when the measured distribution data does not meet the target distribution data, such that the settings of the agricultural header are adjusted when the detected traits of the crop material deviate from the selected amount defined in the target distribution data.

[0150] The fourteenth feature, which may be combined with any of the preceding features, wherein analyzing multiple images containing at least a portion of an agricultural header to detect crop material in the images includes: detecting crop material based on a contrast between a first color associated with the crop material and a second color associated with the surrounding environment of the crop material.

[0151] In a third embodiment, a computer-implemented system includes one or more processors and a non-transitory computer-readable storage medium connected to the one or more processors and storing programming instructions for execution by the one or more processors. These programming instructions instruct the one or more processors to: analyze a plurality of images containing at least a portion of an agricultural header to detect crop material present in the images; classify the detected crop material in the plurality of images; generate measured distribution data based on the classified crop material; compare the measured distribution data with target distribution data; and adjust the settings of the agricultural header when the measured distribution data does not meet the target distribution data.

[0152] The aforementioned and other described embodiments may each optionally include one or more of the following features:

[0153] The first feature may be combined with any of the following features, wherein the programming instructions include programming instructions that instruct the one or more processors to generate multiple images of an area of ​​the harvester header during the harvesting operation.

[0154] The second feature, which can be combined with any of the preceding or following features, includes programming instructions that instruct the one or more processors to generate multiple images of an area of ​​the harvester header during harvesting operations, including programming instructions that instruct the one or more processors to capture the multiple images using an image sensor.

[0155] The third feature, which can be combined with any of the preceding or following features, wherein the programming instructions instructing the one or more processors to analyze multiple images containing at least a portion of an agricultural header to detect crop material present in the images include: instructing the one or more processors to detect the type of crop material present in the images.

[0156] The fourth feature, which may be combined with any of the preceding or following features, includes programming instructions that indicate the type of crop material present in the image detected by the one or more processors, including programming instructions that indicate the one or more processors to detect at least one of the crop grain component (CGC) of the crop being harvested or the material other than grain (MOG) of the crop being harvested.

[0157] The fifth feature, which may be combined with any of the preceding or following features, wherein the programming instructions instructing the one or more processors to analyze the plurality of images containing at least a portion of an agricultural header to detect crop material present in the images include: programming instructions instructing the one or more processors to detect traits of the crop material in the plurality of images.

[0158] The sixth feature, which may be combined with any of the preceding or following features, wherein the programming instructions instructing the one or more processors to detect the traits of crop material in the image include: programming instructions instructing the one or more processors to determine the trajectory of the detected crop material relative to the agricultural header.

[0159] The seventh feature, which may be combined with any of the preceding or following features, wherein the programming instructions instructing the one or more processors to determine the trajectory of the detected crop material relative to the agricultural header include: programming instructions instructing the one or more processors to determine the vector of the crop material relative to the agricultural header.

[0160] The eighth feature, which may be combined with any of the preceding or following features, wherein the programming instructions instructing the one or more processors to determine the vector of crop material relative to the agricultural header include: instructing the one or more processors to: determine the centroid of the crop material; determine the boundary of the harvester header; generate a line from the centroid to a position along the boundary; and detect how the position of the centroid relative to the position along the boundary of the agricultural header changes over time based on how the length and position of the line change relative to the position.

[0161] The ninth feature, which may be combined with any of the preceding or following features, wherein the programming instructions instructing the one or more processors to detect the traits of crop material in the image include: programming instructions instructing the one or more processors to determine the rotation of the crop material.

[0162] The tenth feature, which may be combined with any of the preceding or following features, includes programming instructions that instruct the one or more processors to determine the rotation of the crop material, including programming instructions that instruct the one or more processors to determine the principal axis of the crop material and detect changes in the length or orientation of the principal axis from one image to the next to define the rotation of the crop material.

[0163] The eleventh feature, which may be combined with any of the preceding or following features, wherein the programming instructions instructing the one or more processors to analyze the plurality of images containing at least a portion of an agricultural header to detect crop material present in the images include: programming instructions instructing the one or more processors to predict whether a trajectory represents a loss of crop material at the header, and wherein the programming instructions instructing the one or more processors to compare measured distribution data with target distribution data include: programming instructions instructing the one or more processors to determine whether the trajectory-based loss exceeds a threshold defined in the target distribution data.

[0164] The twelfth feature, which may be combined with any of the preceding or following features, wherein the measured distribution data includes detected traits of crop material, and wherein the programming instructions instructing the one or more processors to compare the measured distribution data with target distribution data include: programming instructions instructing the one or more processors to determine whether the detected traits of crop material deviate from the conditions defined in the target distribution data by a selected amount.

[0165] The thirteenth feature, which may be combined with any of the preceding or following features, includes programming instructions that instruct the one or more processors to adjust the settings of the agricultural header when the measured distribution data does not meet the target distribution data, including programming instructions that instruct the one or more processors to adjust the settings of the agricultural header when the detected traits of the crop material deviate from the conditions defined in the target distribution data by a selected amount.

[0166] The fourteenth feature, which may be combined with any of the preceding features, includes programming instructions that instruct the one or more processors to analyze multiple images containing at least a portion of an agricultural header to detect crop material present in the images, including programming instructions that instruct the one or more processors to detect crop material based on a contrast between a first color associated with the crop material and a second color associated with the surrounding environment of the crop material.

[0167] The embodiments of the subject matter and functional operation described in this specification may be implemented in digital electronic circuit systems, in tangibly implemented computer software or firmware, in computer hardware including the structures disclosed in this specification and their structural equivalents, or in a combination of one or more of these. Software embodiments of the described subject matter may be implemented as one or more computer programs. Each computer program may include one or more modules of computer program instructions encoded on a tangible, non-transitory, computer-readable computer storage medium for execution by or control of the operation of a data processing device. Alternatively or additionally, the program instructions may be encoded in / on artificially generated propagated signals. For example, the signal may be a machine-generated electrical, optical, or electromagnetic signal generated to encode information for transmission to a suitable receiver device for execution by the data processing device. The computer storage medium may be a machine-readable storage device, a machine-readable storage matrix, a random or serial access storage device, or a combination of computer storage media.

[0168] The terms “data processing apparatus,” “computer,” and “electronic computer equipment” (or their equivalents as understood by one of ordinary skill in the art) refer to data processing hardware. For example, a data processing apparatus can encompass all kinds of devices, apparatuses, and machines for processing data, including, for example, programmable processors, computers, or multiple processors or computers. The apparatus may also include a dedicated logic circuit system, including, for example, a central processing unit (CPU), a field-programmable gate array (FPGA), or an application-specific integrated circuit (ASIC). In some embodiments, the data processing apparatus or dedicated logic circuit system (or a combination of data processing apparatus or dedicated logic circuits) may be hardware-based or software-based (or a combination of both). The apparatus may optionally include code that creates an execution environment for computer programs, such as code constituting a combination of processor firmware, protocol stack, database management system, operating system, or execution environment. This disclosure contemplates the use of data processing apparatuses with or without conventional operating systems (e.g., LINUX, UNIX, WINDOWS, MAC OS, ANDROID, or IOS).

[0169] Computer programs (which may also be referred to or described as programs, software, software applications, modules, software modules, scripts, or code) can be written in any form of programming language. Programming languages ​​can include, for example, compiled languages, interpreted languages, declarative languages, or procedural languages. Programs can be deployed in any form, including as standalone programs, modules, components, subroutines, or units for use in a computing environment. Computer programs may, but are not required to, correspond to files in a file system. Programs may be stored as a part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), as a single file dedicated to the program in question, or as multiple co-located files storing portions of one or more modules, subroutines, or code. Computer programs can be deployed for execution on one or more computers, such as those located at a site or distributed across multiple sites interconnected by a communication network. While portions of programs shown in the various figures may be represented as separate modules implementing various features and functions through various objects, methods, or processes, these programs may alternatively include multiple submodules, third-party services, components, and libraries. Conversely, the features and functions of various components may be appropriately combined into a single component. The threshold used for calculation can be determined statically, dynamically, or both.

[0170] The methods, processes, or logical flows described in this specification can be executed by one or more programmable computers that execute one or more computer programs to perform functions by manipulating input data and generating output. The methods, processes, or logical flows can also be executed by a dedicated logic circuit system (e.g., a CPU, FPGA, or ASIC), and the device can also be implemented as a dedicated logic circuit.

[0171] A computer suitable for executing computer programs can be based on one or more of general-purpose and special-purpose microprocessors, as well as other types of CPUs. The components of a computer are a CPU for executing or running instructions and one or more memory devices for storing instructions and data. Generally, the CPU can receive instructions and data from memory (and write data to memory). A computer may also include or be operatively connected to one or more mass storage devices for storing data. In some embodiments, the computer can receive data from and transfer data to a mass storage device, such as a magnetic disk, magneto-optical disk, or optical disk. Furthermore, the computer can be embedded in another device, such as a mobile phone, a personal digital assistant (PDA), a mobile audio or video player, a game console, a global positioning system (GPS) receiver, or a portable storage device such as a universal serial bus (USB) flash drive.

[0172] Computer-readable media (appropriately, transient or non-transient) suitable for storing computer program instructions and data can include all forms of permanent / non-permanent and volatile / non-volatile memory, media, and memory devices. Computer-readable media can include, for example, semiconductor memory devices such as random access memory (RAM), read-only memory (ROM), phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), and flash memory devices. Computer-readable media can also include, for example, magnetic devices such as magnetic tape, cassette tape, card-type cassette tape, and internal / removable disks. Computer-readable media can also include magneto-optical and optical storage devices and technologies, including, for example, digital video discs (Digital Video Discs, DVDs), CD-ROMs, DVD+ / -Rs, DVD-RAMs, DVD-ROMs, HD-DVDs, and BLURAYs. Memory can store various objects or data, including caches, classes, frames, applications, modules, backup data, jobs, web pages, web page templates, data structures, database tables, repositories, and dynamic information. The types of objects and data stored in memory can include parameters, variables, algorithms, instructions, rules, constraints, and references. Additionally, internal memory can include logs, policies, security or access data, and report files. Processors and memory can be supplemented by or incorporated into a dedicated logic circuit system.

[0173] Embodiments of the subject matter described in this disclosure can be implemented on a computer having a display device for providing interaction with a user, including displaying information to the user (and receiving input from the user) . Types of display devices may include, for example, cathode ray tubes (CRTs), liquid crystal displays (LCDs), light-emitting diodes (LEDs), and plasma monitors. Display devices may include keyboards and pointing devices, including, for example, mice, trackballs, or trackpads. User input can also be provided to the computer using touchscreens, such as pressure-sensitive tablet computer surfaces or multi-touch screens using capacitive or inductive touchscreens. Other types of devices can be used to provide interaction with the user, including for receiving user feedback, including, for example, sensory feedback including visual, auditory, or tactile feedback. Input from the user can be received in the form of sound, speech, or tactile input. Furthermore, the computer can interact with the user by sending documents to and receiving documents from devices used by the user. For example, the computer can send a webpage to a web browser on a user's client device in response to a request received from a web browser.

[0174] The term "graphical user interface" or "GUI" can be used in the singular or plural to describe one or more graphical user interfaces and each of the displays of a particular graphical user interface. Therefore, a GUI can represent any graphical user interface, including but not limited to a web browser, a touchscreen, or a command-line interface (CLI) that processes information and effectively presents the results to the user. Generally, a GUI may include multiple user interface (UI) elements, some or all of which are associated with a web browser, such as interactive fields, dropdown lists, and buttons. These and other UI elements may be related to or represent the functionality of a web browser.

[0175] Implementations of the subject matter described in this specification can be implemented in computing systems that include back-end components (e.g., as data servers) or middleware components (e.g., application servers). Furthermore, the computing system may include front-end components, such as client computers having one or both of a graphical user interface or a web browser through which a user can interact with the computer. Components of the system can be interconnected via wired or wireless digital data communication (or a combination of data communication) of any form or medium in a communication network. Examples of communication networks include local area networks (LANs), radio access networks (RANs), metropolitan area networks (MANs), wide area networks (WANs), Worldwide Interoperability for Microwave Access (WIMAX), wireless local area networks (WLANs) (e.g., using 802.11a / b / g / n or 802.20 or a combination of protocols), all or part of the Internet, or any other communication system (or combination of communication networks) in one or more locations. A network can communicate with a combination of communication types, such as Internet Protocol (IP) packets, Frame Relay frames, asynchronous transfer mode (ATM) cells, voice, video, data, or network addresses.

[0176] A computing system may include clients and servers. Clients and servers can typically be geographically separated and can usually interact through a communication network. The client-server relationship can be established by computer programs running on their respective computers and having a client-server relationship.

[0177] A clustered file system can be any file system that can be accessed from multiple servers for reading and updating. Locking or consistency tracking may not be necessary, as locking of the swap file system can be done at the application layer. Additionally, Unicode data files can differ from non-Unicode data files.

[0178] While this specification contains numerous specific implementation details, these should not be construed as limiting the scope of the claims, but rather as descriptions of features that may be specific to particular embodiments. Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately in multiple embodiments, or in any suitable sub-combination. Furthermore, although previously described features may be described as functioning in certain combinations, and even initially claimed in this way, in some cases, one or more features from the claimed combination may be removed from that combination, and the claimed combination may be for sub-combinations or variations thereof.

[0179] Specific embodiments of the subject matter have been described. Other embodiments, modifications, and substitutions of the described embodiments are within the scope of the following claims, as will be apparent to those skilled in the art. Although operations are described in a specific order in the drawings or claims, this should not be construed as requiring these operations to be performed in the specific order shown or in sequential order, or requiring the performance of all shown operations (some operations may be considered optional) to achieve the desired result. In some cases, multitasking or parallel processing (or a combination of multitasking and parallel processing) may be advantageous and is considered appropriate.

[0180] Furthermore, the separation or integration of various system modules and components in the embodiments described above should not be construed as requiring such separation or integration in all embodiments, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

[0181] Therefore, the exemplary embodiments described above do not define or limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure.

[0182] Furthermore, any claimed implementation is considered to be applicable at least to computer-implemented methods, non-transitory computer-readable media storing computer-readable instructions to perform computer-implemented methods, and computer systems including computer memory operably connected to a hardware processor configured to perform computer-implemented methods or instructions stored on a non-transitory computer-readable medium.

[0183] While exemplary embodiments of the present disclosure have been described above, these descriptions should not be construed as restrictive or limiting. Rather, other changes and modifications may be made without departing from the scope and spirit of the disclosure as defined in the appended claims.

Claims

1. A computer-implemented method, executed by one or more processors, for controlling an agricultural header (108, 408, 1000, 1600, 1820) during harvesting based on the movement of crop material relative to the header (108, 408, 1000, 1600, 1820), the method comprising: Analyze one or more images containing at least a portion of the agricultural header to detect crop material present in the one or more images, wherein the portion of the agricultural header includes at least one static position, and the analysis includes detecting movement of the crop material relative to the static position; Classify the detected crop materials in the one or more images; Distribution data based on the generation measurement of crop materials classified (220). as well as The settings of the agricultural header are adjusted using the measured distribution data.

2. The computer-implemented method according to claim 1 further includes comparing the measured distribution data (220) with the target distribution data (222).

3. The computer-implemented method according to claim 2, wherein, Adjusting the settings of the agricultural header (108, 408, 1000, 1600, 1820) using the measured distribution data (220) includes: adjusting the settings of the agricultural header when the measured distribution data does not meet the target distribution data (222).

4. The computer-implemented method according to any one of claims 1 to 3, further comprising: During harvesting operations, one or more images of a region of the agricultural header (108, 408, 1000, 1600, 1820) are generated using image sensors (114, 206).

5. The computer-implemented method according to any one of claims 1 to 3, wherein, Analyzing one or more images containing at least a portion of the agricultural header (108, 408, 1000, 1600, 1820) to detect crop material present in the one or more images includes: detecting the type of crop material present in the one or more images.

6. The computer-implemented method according to claim 5, wherein, Detecting the type of crop material present in the one or more images includes detecting at least one of the crop grain component (CGC) of the crop being harvested or the material other than grain (MOG) of the crop being harvested.

7. The computer-implemented method according to claim 5, wherein, The measured distribution data (220) includes the detected traits of the crop material, and the method further includes determining whether the detected traits of the crop material deviate from the selected conditions by a selected amount.

8. The computer-implemented method according to claim 7, wherein, Adjusting the settings of the agricultural header (108, 408, 1000, 1600, 1820) using the measured distribution data (220) includes adjusting the settings of the agricultural header when the detected traits of the crop material deviate from the selected conditions by the selected amount.

9. The computer-implemented method according to any one of claims 1 to 3, wherein, Analyzing one or more images containing at least a portion of the agricultural header (108, 408, 1000, 1600, 1820) to detect crop material present in the one or more images includes: detecting the characteristics of the crop material in the one or more images.

10. The computer-implemented method according to claim 9, wherein, Detecting the characteristics of crop material in one or more images includes determining the trajectory of the detected crop material relative to the static position of the agricultural header (108, 408, 1000, 1600, 1820).

11. The computer-implemented method according to claim 10, wherein, Analyzing one or more images containing at least a portion of the agricultural headers (108, 408, 1000, 1600, 1820) to detect crop material present in the one or more images includes: predicting whether the trajectory represents a loss of crop material leaving the agricultural header, and The method also includes determining whether the loss based on the trajectory exceeds a threshold.

12. The computer-implemented method according to claim 9, wherein, Detecting the characteristics of the crop material in the one or more images includes determining the rotation of the crop material.

13. The computer-implemented method according to any one of claims 1 to 3, wherein, Analyzing one or more images containing at least a portion of the agricultural header (108, 408, 1000, 1600, 1820) to detect crop material present in the one or more images includes detecting the crop material based on a contrast between a first color associated with the crop material and a second color associated with the surrounding environment of the crop material.

14. An apparatus for controlling an agricultural header (108, 408, 1000, 1600, 1820) during harvesting based on the movement of crop material at the header (108, 408, 1000, 1600, 1820), the apparatus comprising: One or more processors (202, 1705); as well as A non-transitory computer-readable storage medium (204, 1707) is connected to the one or more processors and stores programming instructions for execution by the one or more processors, the programming instructions instructing the one or more processors to: Analyze one or more images containing at least a portion of the agricultural header to detect crop material present in the one or more images, wherein the portion of the agricultural header includes at least one static position, and the analysis includes detecting movement of the crop material relative to the static position; Classify the detected crop materials in the one or more images; Distribution data based on the generation measurement of crop materials classified (220). The measured distribution data is compared with the target distribution data (222); as well as When the measured distribution data does not meet the target distribution data, the settings of the agricultural header are adjusted.

15. The device according to claim 14, wherein, The programming instructions operable to instruct the one or more processors (202, 1705) to adjust the settings of the agricultural header (108, 408, 1000, 1600, 1820) when the measured distribution data (220) does not meet the target distribution data (222) include: programming instructions operable to instruct the one or more processors to adjust the settings of the agricultural header when the detected traits of the crop material deviate from the conditions defined in the target distribution data by a selected amount.

Images