SYSTEMS, METHODS AND NON-VOID COMPUTER-READABLE MEDIA FOR CONTROLLING A MACHINE BASED ON A CROSS ERROR

Abstract

Systems, methods, and non-volatile, computer-readable media for controlling a machine based on a transverse error. A system includes a control system or user interface (UI) and a processing circuit arrangement designed to cause the system to detect a machine detection boundary and a crop detection boundary, where the machine detection boundary pertains to a supporting structure of a machine and the crop detection boundary pertains to a set of plant material, to determine a transverse error represented by a first distance between the machine detection boundary and the crop detection boundary, and to control the control system or UI based on the transverse error.

Description

AREA OF REVELATION

[0001] Some embodiments provide systems, methods and non-volatile computer-readable media for controlling a machine based on a transverse error determined by means of perceptual acquisitions relative to parts of the machine and the crop. STATE OF THE ART

[0002] Crop fields are planted in rows with gaps between them to allow machinery to move through the fields without damaging the emerging crop. For example, the wheels of the machines can touch the spaces between the rows to support the machinery without driving over the crop. SUMMARY

[0003] Some embodiments provide improved systems, methods, and non-volatile computer-readable media for controlling a machine based on a transverse error determined by perceptual acquisitions relative to parts of the machine and the crop.

[0004] Some embodiments provide a system comprising a control system or user interface (UI) and a processing circuit arrangement, wherein the processing circuit arrangement is designed to cause the system to detect a machine detection boundary and a crop detection boundary, wherein the machine detection boundary pertains to a supporting structure of a machine and the crop detection boundary pertains to a set of plant material, to determine a transverse error represented by a first distance between the machine detection boundary and the crop detection boundary, and to control the control system or UI based on the transverse error between these detections.

[0005] Some embodiments provide a method that includes: detecting a machine detection boundary and a crop detection boundary, wherein the machine detection boundary belongs to a supporting structure of a machine and the crop detection boundary belongs to a set of plant material; determining a transverse error represented by a first distance between the machine detection boundary and the crop detection boundary; and controlling a control system or user interface (UI) based on the transverse error.

[0006] Some embodiments provide a non-volatile, computer-readable medium that stores instructions which, when executed by at least one processor, cause the at least one processor to perform a method comprising: detecting a machine detection boundary and a crop detection boundary, wherein the machine detection boundary pertains to a supporting structure of a machine and the crop detection boundary pertains to a set of plant material; determining a transverse error represented by a first distance between the machine detection boundary and the crop detection boundary; and controlling a control system or user interface (UI) based on the transverse error. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The various features and advantages of the non-restrictive embodiments discussed herein will become clearer upon review of the detailed description in conjunction with the accompanying drawings. The accompanying drawings are provided for illustrative purposes only and should not be interpreted as limiting the scope of protection of the claims. The accompanying drawings are not to be considered to scale unless expressly stated otherwise. For the sake of clarity, various dimensions may be exaggerated in the drawings. Fig. Figure 1 shows a machine according to several embodiments, which travels through rows of crops in a field; Fig. 2 shows a diagram of a system according to some exemplary embodiments; Fig. Figure 3A shows an example scenario of a machine according to some embodiments, which travels along rows of a field; Fig. Figure 3B shows another example scenario of a machine according to some embodiments, which travels along rows of a field; and Fig. Figure 4 shows a method for controlling a machine according to some exemplary embodiments. DETAILED DESCRIPTION

[0008] Some embodiments described herein relate to detecting whether the wheels of a machine moving through rows of crops in a field are touching or running over the crop. The machine may be a vehicle, such as a sprayer, tractor, etc. The machine may be self-propelled, although some embodiments are not limited to this. The machine may be described herein as being wheel-supported, although some embodiments are not limited to this. For example, according to some embodiments, the machine may be supported by chains, etc. In such cases, the detection may involve detecting whether the chains (or other support structures) of the machine are touching or running over the crop.

[0009] Fig. Figure 1 shows a machine according to several embodiments, which travels through rows of crops in a field.

[0010] With reference to Fig. 1. A machine 100 travels through a field 110. The machine 100 can have four wheels, with wheels located at the front and rear on both sides of the machine 100; however, some embodiments are not limited to this. The field 110 contains rows 112 of plant material and spaces 114 between the rows 112.

[0011] According to some embodiments, the plant material can be (or include) a crop (which may also be referred to herein as crop rows 112). The crop may include, for example, grain, corn, soybeans, pulses, nuts, vegetables, fruit, potatoes, tubers, etc. The crop may be planted in the rows 112 with the spaces 114 between them, such that the respective widths of the rows 112 and spaces 114 allow the machine 100 and / or other machines to move through the field without the wheels of the machine 100 touching the rows 112. For example, the width of each of the rows 112 may be less than the width of a gap between the wheels on each side of the machine 100. The width of each of the spaces 114 may also be greater than the width of each of the wheels.

[0012] While the machine 100 travels through the rows 112, it can perform one or more agricultural operations. For example, the machine 100 can spray the crop while traveling through the rows 112 (e.g., with pesticides, herbicides, fertilizers, water, etc.). When navigating through the rows 112, a guidance system on the machine 100 or a driver of the machine 100 attempts to steer the machine 100 so that its wheels touch (and follow) the spaces 114 without touching the rows 112. In this way, the machine 100 can span one of the rows 112 on each pass through the field 110.

[0013] For example, conventional guidance systems rely on a forward-facing camera at the front of the machine 100 to detect the rows 112 and control the machine's path so that its wheels do not drive over the crop. Despite the aforementioned attempts to steer the machine 100 so that its wheels do not touch the rows 112, the wheels may still occasionally touch and damage the crop in the rows 112. For example, a wheel of the machine 100 may deviate from the spacing 114 into the rows 112 and drive over the crop, due to a guidance system error or operator error. However, conventional guidance systems are unable to detect whether the wheels of the machine 100 have driven over the crop.Accordingly, conventional guidance systems are unable to use such information when adjusting the control of machine 100 based on the results of previous guidance, or to distinguish between crop yield reductions resulting from machine 100 driving over the crop and reductions due to other reasons (e.g., seed variety). Some embodiments provided herein overcome the limitations of conventional guidance systems in order to at least detect whether the wheels of machine 100 have touched and / or driven over the crop in rows 112.

[0014] Fig. Figure 2 shows an overview of System 200 according to some exemplary embodiments.

[0015] With reference to Fig. 2. The system 200 may include a perception processing unit 202, a memory 204, a perception device 206, a position determination device 208, a control system 210, and / or a user interface (UI) 212. The perception processing unit 202 may control the overall operation of the system 200 and may be implemented using a processing circuit arrangement. The term "processing circuit arrangement," as used in this disclosure, may refer, for example, to hardware including logic circuits, to a hardware / software combination such as a processor executing software, or to a combination thereof.For example, the processing circuit arrangement may more specifically include, but is not limited to, a central processing unit (CPU), an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field-programmable gate array (FPGA), a system-on-a-chip (SoC), a programmable logic unit, a microprocessor, an application-specific integrated circuit (ASIC). According to some embodiments, the system 200 may be partially or completely contained within the machine 100; however, some embodiments are not limited to this, and at least one or more elements of the system 200 may be external to the machine 100.

[0016] The perception processing unit 202 can store data in and / or retrieve data from the memory 204 (e.g., programming instructions for execution by the perception processing unit 202, operating data generated by the perception processing unit 202, etc.). The perception processing unit 202 can communicate with and / or control the perception device 206, the position determination device 208, the control system 210, and / or the UI 212.

[0017] Memory 204 can be a tangible, non-volatile, computer-readable medium, such as random-access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, a hard disk, a removable disk, a compact disk (CD) ROM, any combination thereof, or any other form of storage medium known in the art. Memory 204 can store data and / or instructions for retrieval by, for example, the perception processing unit 202.

[0018] The sensing device 206 can collect sensing data representing the position of at least one sensing structure of the machine 100 (e.g., a wheel, a chain, and / or another physical structure) and the position of at least one element from a row 112 and / or a space 114. The sensing device 206 can be an imaging device, but some embodiments are not limited to this. For example, the sensing device 206 can be a distance measuring device (e.g., a lidar system, a radar system, etc.) that can collect sensing data representing the position of at least one sensing structure of the machine 100 (e.g., a wheel, a chain, and / or another physical structure) and the position of at least one element from a row 112 and / or a space 114, based on distances measured by lidar, radar, etc.For the sake of clarity, the perceiving device 206 can be described here primarily in the context of an implementation based on an imaging device. The imaging device can capture images of the field 110. Each of the images can include at least one perceiving structure of the machine 100 and at least one from a row 112 and / or a space 114. The perceiving processing unit 202 can process the captured images to identify the relative position of the at least one perceiving structure with respect to the row 112 and / or the space 114.

[0019] The imaging device may be or include a camera (e.g., a visible light camera). According to some embodiments, the camera may use a charged-coupled device (CCD), a complementary metal-oxide semiconductor (CMOS), or another sensor that produces color image data, RGB color data, CMYK color data, HSV color data, or image data in another color space. RGB color data refers to a color model in which red, green, and blue light (or signals or data representing them) are combined to represent other wavelengths or disparity information. Each pixel or group of pixels in the captured image data may be associated with an intensity level (e.g., intensity level data), a corresponding pixel value, or an aggregated pixel value.In some embodiments, the intensity level is a measure of the amount of visible light energy, infrared radiation, near-infrared radiation, ultraviolet radiation, or other electromagnetic radiation (e.g., wavelengths) observed, reflected, and / or emitted by one or more objects or part of one or more objects within a scene or within an image (e.g., a raw or processed image) that represents the scene or part of it.The intensity level can be assigned to or derived from one or more of the following: an intensity level of a red, green, or blue component in the RGB color space; an intensity level of multiple components in the RGB color space; a value or brightness in the HSV color space; a brightness or luminance in the HSL color space; an intensity, magnitude, or power of the observed or reflected light in the green visible light spectrum or for another plant color; an intensity, magnitude, or power of the observed or reflected light with a specific green tint or another plant color; and an intensity, magnitude, or power of the observed or reflected light in multiple spectra (e.g., green light and infrared or near-infrared light). For RGB color data, each pixel can be represented by independent values ​​of red, green, and blue components and corresponding intensity level data.CMYK color data mixes cyan, magenta, yellow, and black (or signals or data representing these) to subtractively create other colors. HSV (hue, saturation, value) color data defines the color space in terms of hue (e.g., color type), saturation (e.g., vibrancy or purity of the color), and value (e.g., brightness of the color). For HSV color data, the value or brightness of the color can represent the intensity level. HSL color data defines the color space in terms of hue, saturation, and luminance (e.g., brightness). Brightness or luminance for HSL color data can cover the entire range between black and white. The intensity level can be associated with a specific color.Such as green, or a particular hue or shade within the visible light spectrum associated with green, or other visible colors, infrared radiation, near-infrared radiation, or ultraviolet radiation associated with the plant world.

[0020] According to some embodiments, the sensing device 206 can be positioned under (and attached to) the machine 100; however, some embodiments are not limited to this. For example, the sensing device 206 can be positioned elsewhere on the machine 100 or on another machine (e.g., another machine 100, a drone, etc.). According to some embodiments, the imaging device can be oriented to capture images of at least one sensing structure of the machine 100 (e.g., a wheel, a chain, and / or another physical structure) and at least one of the row 112 and / or the space 114.For example, the camera can be oriented to capture images of only a single wheel (or sensing structure) of the machine 100, of the two rear wheels (or two sensing structures) of the machine 100, or of all four wheels (or more than two sensing structures) of the machine 100; however, some embodiments are not limited to this. A camera's field of view can encompass the entire width of each of the at least one wheel of the machine 100 or it can encompass only one edge of each of the at least one wheel. According to some embodiments, the imaging device can be directed rearward (relative to the machine 100); this orientation is likely to result in less dust obscuring the captured images. However, some embodiments are not limited to this, and the imaging device can be directed forward (relative to the machine 100).

[0021] According to some embodiments, the imaging device can capture an image that includes a perceiving structure of the machine 100 (e.g., a wheel, a chain, and / or another physical structure), the row 112, and / or the space 114. The imaging device can provide the image to the perceiving processing unit 202, which can process the image and / or store it in memory 204. For example, the perceiving processing unit 202 can process the image to determine the width of the space 114 between an edge of a supporting structure (e.g., a wheel) and the row 112 according to a method discussed below. However, some embodiments are not limited to this, and as discussed above, in examples where the perceiving device 206 uses a distance-measuring device (e.g., the lidar system, the radar system, etc.),In one embodiment, the distance measuring device determines a crop distance from the distance measuring device to the row 112 and / or to the space 114 and provides the crop distance to the perception processing unit 202. According to some embodiments, the distance measuring device can also determine a machine distance from the distance measuring device to the perception structure of the machine 100 and provide the machine distance, along with the crop distance, to the perception processing unit 202; however, some embodiments are not limited to this. For example, the machine distance from the distance measuring device to the perception structure of the machine 100 can be fixed (e.g., based on a configuration of the machine 100), and a value of the machine distance can be stored in memory 204 for access by the perception processing unit 202.According to some embodiments, in scenarios where the sensing structure is a support structure (e.g., a wheel, a chain, etc.), the perception processing unit 202 can determine the width of the gap 114 based on a difference between the crop spacing and the machine spacing. According to some embodiments, when determining the difference between the crop and machine spacing, the perception processing unit 202 can take into account the respective angles between the distance measuring device and the row 112, the gap 114, and / or the support structure of the machine 100; however, some embodiments are not limited to this.

[0022] According to some embodiments, the imaging device can be a stereo camera that captures one or more image pairs, each of which includes a perceiving structure of the machine 100 (e.g., a wheel, a chain, and / or another physical structure), the row 112, and / or the space 114. While the stereo camera is described herein as capturing image pairs, some embodiments are not limited to this, and the stereo camera can capture a series of individual images or capture more than two images simultaneously (or concurrently). The perceiving processing unit 202 can apply a stereo fitting algorithm, such as a sum-of-absolute-differences algorithm, a sum-of-squares-differences algorithm, a consensus algorithm, etc., to determine one or more disparity values ​​(e.g., difference values) between each image pair.According to some embodiments, the perception processing unit 202 can generate a disparity map based on one or more disparity values. The perception processing unit 202 can estimate a distance (e.g., a reach) to one or more of the machine 100's perception structure (e.g., a wheel, chain, and / or other physical structure), the row 112, and / or the space 114 based on the one or more disparity values ​​(and / or the disparity map). For example, the perception processing unit 202 can estimate the crop distance and / or the machine distance based on the one or more disparity values ​​(and / or the disparity map). According to some embodiments, the perception processing unit 202 can generate a point cloud (e.g.,a 2D point cloud or a 3D point cloud) based on one or more disparity values ​​(and / or the disparity map) and estimate a distance between the supporting structure of the machine 100 (e.g., a wheel, a chain, etc.) and one or more from the series 112 and / or the space 114 based on a number of points in the point cloud. According to some embodiments, the perception processing unit 202 can generate the point cloud using any algorithm that would be known to persons with ordinary technical skills. According to some embodiments, the point cloud can include a model and / or a representation of the one or more disparity values ​​(and / or the disparity map).

[0023] As mentioned above, the sensing device 206, according to exemplary embodiments, can be a distance measuring device (e.g., a lidar system, a radar system, etc.) capable of collecting sensing data representing the position of at least one sensing structure of the machine 100 (e.g., a wheel, a chain, and / or another physical structure) and the position of at least one element in a row 112 and / or an intermediate space 114, based on distances measured by lidar, radar, etc. For example, the sensing device 206 can include a transmitter and a receiver. The transmitter of the sensing device 206 can output a signal in the direction of one or more elements below the sensing structure of the machine 100 (e.g., a wheel, a chain, and / or another physical structure), the row 112, and / or the intermediate space 114.The receiver of the sensing device 206 can receive a reflection of the output signal from one or more of the sensing structures of the machine 100 (e.g., a wheel, a chain, and / or another physical structure), the array 112, and / or the space 114. According to some embodiments, the transmitted signal can be a laser signal emitted by a laser transmitter; however, some embodiments are not limited to this, and the transmitted signal can be of any type usable in a radar system, lidar system, etc., that would be known to persons with ordinary technical skills. According to some embodiments, the sensing device 206 (and / or the sensing processing unit 202) can determine a distance (e.g., a range) to one or more of the sensing structures of the machine 100 (e.g., a wheel, a chain, and / or another physical structure), the array 112, and / or the space 114.a wheel, a chain, and / or another physical structure), the row 112, and / or the space 114 based on the reflected signal. For example, the sensing device 206 (and / or the sensing processing unit 202) can determine one or more distance values ​​(e.g., range values) based on a propagation time (or flight time) of each pair of transmitted and reflected signals. According to embodiments, the sensing processing unit 202 can estimate the crop distance and / or the machine distance based on the reflected signal(s). According to some embodiments, the sensing processing unit 202 can generate a point cloud (e.g., a 2D point cloud or a 3D point cloud) based on one or more distance values ​​(e.g., ranges) and determine a distance between the supporting structure of the machine 100 (e.g., a wheel, a chain, etc.).) and one or more from the series 112 and / or the space 114 based on a number of points in the point cloud. According to some embodiments, the perceptual processing unit 202 can generate the point cloud using any algorithm that would be known to persons with ordinary technical skills. According to some embodiments, the point cloud can include a model and / or a representation of the one or more disparity values ​​(and / or the disparity map).

[0024] The positioning device 208 can receive a positioning signal from an external source that represents, or can be processed to determine, the current position of the machine 100. For example, the positioning device 208 can be a receiver that receives a signal from global positioning satellites (e.g., a Global Positioning System (GPS) receiver), although some embodiments are not limited to this. In another example, the positioning device 208 can be a receiver that receives the positioning signal from a terrestrial source (e.g., a base station, an access point, another machine 100, etc.). The positioning device 208 can provide the positioning signal to the perception processing unit 202, which can process and / or store the position of the machine 100 in the memory 204.

[0025] For example, the Perception Processing Unit 202 can obtain the position of machine 100 directly from the positioning signal or determine the machine's position based on the positioning signal. The Perception Processing Unit 202 can store the position of machine 100 in memory 204 in conjunction with at least one of (1) a time at which that position represented the current position of machine 100, and / or (2) an image received from the imaging device that was taken at the time when the position represented the current position of machine 100 (or a crop distance and / or a machine distance measured by the distance measuring device). The Perception Processing Unit 202 can use the position received from the positioning device 208 to generate a crop damage map in field 110 according to a procedure discussed below.

[0026] The control system 210 can include one or more mechanical systems for controlling the movement and / or position of the machine 100. For example, the control system 210 can include a steering mechanism and / or a speed control mechanism (e.g., a drive mechanism (e.g., a motor), a braking mechanism, etc.). Each relevant mechanical system of the control system 210 can be controlled according to corresponding control signals received from the perception processing unit 202. For example, in response to the detection that the machine 100 is getting too close to or has crossed the row 112, the perception processing unit 202 can generate a control signal to control the steering mechanism to steer the machine 100 away from the row 112.In such a situation, the perception processing unit 202 can additionally or alternatively generate a control signal to control the speed control mechanism in order to reduce the speed of machine 100. The speed of machine 100 can be considered an error range in the control of the steering of machine 100, since the faster the machine 100 travels, the less time is available for steering corrections to avoid or reduce crop damage. Accordingly, the perception processing unit 202 can control the control system 210 to reduce the speed of machine 100 in order to reduce the overall crop damage caused by the crop being run over by the wheel of machine 100.

[0027] The UI 212 can include one or more devices for communicating information to a driver of the machine 100. For example, the UI 212 can include one or more display screens for showing visual information, audio speakers for outputting an audio signal, signal lights for displaying a visual signal, etc. Each relevant device of the UI 212 can be controlled according to appropriate control signals received from the perception processing unit 202. For example, in response to the determination that the machine 100 is too close to or has driven over the row 112, the perception processing unit 202 can generate a control signal to control one or more of the display screens, audio speakers, signal lights, etc., to issue a notification.The notification can contain information that can be used by the operator of machine 100 to adjust the position and / or speed of machine 100 to avoid or reduce crop damage caused by the crop being run over by the machine's wheel. For example, the notification can include an indication that machine 100 has deviated too far to one side, an indication of the side to which machine 100 has deviated too far, an indication of a direction in which machine 100 should be steered to avoid or reduce crop damage, an instruction to steer machine 100 in that direction, etc.

[0028] According to some embodiments, the machine 100 may include only one of the control system 210 and / or the UI 212. For example, in cases where the machine 100 is unmanned and / or autonomous, it may include the control system 210 without the UI 212. Alternatively, in cases where the machine 100 is manned without an automatic guidance system, it may include the UI 212 without the control system 210. Some embodiments are not limited to these examples, and the machine 100 may include both the control system 210 and the UI 212.

[0029] Fig. Figure 3A shows an example scenario of a machine 100 according to some embodiments, which travels along rows 112 of a field 110.

[0030] With reference to Fig. In example scenario 3A, a support structure 302 is positioned in a gap 114 next to a row 112. The support structure 302 can be a wheel of the machine 100; however, some embodiments are not limited to this, and the support structure 302 can, as mentioned above, be, for example, a chain, etc. In this example scenario, the machine 100, with the support structure 302, travels through the field 110 approximately parallel to the row 112, with the support structure 302 following the gap 114 next to the row 112.

[0031] Box 304 can represent a field of view of the perception device 206, which can include a perception structure of the machine 100, the row 112, and the space 114. The perception processing unit 202 can receive perception data captured in the field of view of the imaging device (one or more images, a crop distance, and / or a machine distance measured by the distance measuring device). Fig. In Figure 3A, the box 304 is shown to include the support structure 302 as a sensing structure; however, some embodiments are not limited to this. The sensing processing unit 202 can determine a distance 306 between an edge of the support structure 302 and the row 112 using the sensing data. For example, in implementations where the sensing device 206 is the imaging device, the sensing processing unit 202 can detect pixel boundaries of the sensing structure, the row 112, and / or the space 114 based on color information (or other distinguishing wavelength information) corresponding to the pixels of the image.According to some embodiments, in implementations where the sensing device 206 is the distance measuring device and / or the stereo camera, the perception processing unit 202 can determine the boundaries of the perception structure, the row 112, and / or the gap 114 based on the crop spacing and / or the machine spacing. According to some embodiments, the perception structure can be the support structure 302; however, some embodiments are not limited to this. For example, the perception structure of the machine 100 can be any physical structure of the machine 100 that is within the field of view of the perception device 206 and that can be used to determine a distance between the support structure 302 and the row 112.In scenarios where the perceiving structure is a physical structure of machine 100 that is not the support structure 302, the perceiving processing unit 202 can determine the distance between the support structure 302 and the row 112 based on a known distance or distances (in one or more dimensions) between the perceiving structure and the support structure 302. This known distance can, for example, be stored in memory 204.

[0032] The examples discussed here (e.g. in Fig. Figures 3A-3B refer to the box 304 containing a single side of a single support structure 302 of the machine 100; however, some embodiments are not limited to this. According to some embodiments, the box 304 may contain only the opposite side of the single support structure 302, two (or both) sides of the single support structure 302, one side each of two or more support structures 302, two (or both) sides of two or more support structures 302, and so on. Likewise, in scenarios where the sensing structure is not the support structure 302, the box 304 may contain two or more sensing structures and / or more than one side of a sensing structure. The following discussion can be applied to each of these examples to determine the proximity between the support structure(s) 302 and one or more rows 112.

[0033] According to some embodiments, in implementations where the perceiving device 206 is the imaging device, the perceiving processing unit 202 can detect the pixel boundaries by determining pixels of the image that correspond to colors (or other distinguishing wavelength information) corresponding to those of the perceiving structure (in scenarios where the perceiving structure is the support structure 302, this pixel boundary may also be referred to herein as the "machine detection boundary"), the array 112 (may also be referred to herein as the "crop detection boundary"), and / or the space 114. For example, the perceiving structure may correspond to black or gray colors, the array 112 may correspond to green colors, and the space 114 may correspond to brown colors; however, some embodiments are not limited to these.According to some embodiments, a table of color value ranges (or other wavelength ranges) can be stored in memory 204, in which each color value range corresponds to one from the perceptual structure, the row 112, and / or the space 114. The perceptual processing unit 202 can compare color values ​​(or other wavelength values) of pixels in the image with the color value ranges (or other wavelength ranges) and assign each pixel to one from the perceptual structure, the row 112, and / or the space 114, depending on which color value range (or other wavelength range) contains the color value (or other wavelength value) of the pixel.

[0034] According to some embodiments, the perception processing unit 202 can detect the pixel boundaries by segmenting at least some pixels of the image based on the color value ranges (or other wavelength ranges) to obtain perception structure, row 112 and / or interspace segments 114.

[0035] According to some embodiments, in implementations where the perceiving device 206 is the distance measuring device and / or the stereo camera, the perception processing unit 202 can detect boundaries corresponding to those of the perceiving structure (in scenarios where the perceiving structure is the support structure 302, this boundary may also be referred to herein as the “machine detection boundary”), the row 112 (may also be referred to herein as the “crop detection boundary”) and / or the space 114 based on the crop distance and / or machine distance received from the distance measuring device (and / or the stereo camera).According to some embodiments, in scenarios where the perceiving structure is the support structure 302, the perception processing unit 202 can detect the machine detection boundary and the crop detection boundary based on the pixel colors of the image(s) discussed above; however, some embodiments are not limited to this. For example, according to some embodiments, the perception processing unit 202 can detect the machine detection boundary and the crop detection boundary based on one or more disparity values ​​(or the disparity map) obtained using the stereo camera and / or based on the point cloud generated based on one or more disparity values ​​(or the disparity map).According to some embodiments, the perception processing unit 202 can detect the machine detection limit and the crop detection limit based on one or more distance values ​​(e.g., range values) obtained using the distance measuring device and / or based on the point cloud generated based on one or more distance values ​​(e.g., range values).

[0036] According to some embodiments, the perceptual processing unit 202 can determine the distance 306 between the support structure 302 and the array 112. For example, in scenarios where the perceptual structure is the support structure 302, the perceptual processing unit 202 can determine the distance 306 as a width between an edge of the support structure 302 and an edge of the array 112. In another example, the perceptual processing unit 202 can determine the distance 306 as a width of the space 114 between the support structure 302 and the array 112. According to some embodiments, the determined distance 306 between the support structure 302 and the array 112 can be represented as a pixel spacing, but some embodiments are not limited to this. According to some embodiments, the determined distance 306 can be represented as a distance between disparity values ​​(e.g.,The disparity map discussed above in connection with the stereo camera), a distance in points of the point cloud (as discussed above in connection with the stereo camera and / or the distance measuring device), etc., can be represented. In scenarios where the perceiving structure is a physical structure of the machine 100 that is not the support structure 302, the perceiving processing unit 202 can determine the distance 306 between the support structure 302 and the row 112, or as the width of the gap 114 between the support structure 302 and the row 112, based on a known distance or distances (in one or more dimensions) between the perceiving structure and the support structure 302.

[0037] This known distance(s) can be stored, for example, in memory 204. According to some embodiments, the perception processing unit 202 can determine the machine detection limit as the edge of the support structure 302 based on the known distance(s) between the perception structure and the support structure 302 and the perception data, and determine the distance 306 as the difference between the machine detection limit and the crop detection limit.

[0038] According to some embodiments, the distance 306 between the support structure 302 and the row 112 can be determined as the physical distance. For example, in implementations where the perceiving device 206 is the imaging device, the perception processing unit 202 can convert the pixel spacing into the physical distance based on the position and orientation (e.g., distance from the perceiving structure, row 112, and / or space 114) of the imaging device. According to some embodiments, the perception processing unit 202 can perform similar conversions of disparity values ​​(e.g., the disparity map discussed above in connection with the stereo camera), points of the point cloud (as discussed above in connection with the stereo camera and / or the distance measuring device), etc., into the physical distance.According to some embodiments, the position and orientation of the sensing device 206 can vary based on a type of machine 100, and the sensing processing unit 202 can convert the pixel spacing into the physical distance based on the type of machine 100.

[0039] According to some embodiments, the perception processing unit 202 can perform a calibration for the relative position of the row 112 to the perception structure. For example, the calibration can be initiated by a driver of the machine 100 in a training row 112 or performed automatically while the machine 100 travels along the row 112. According to some embodiments, the perception processing unit 202 can establish (e.g., configure) a standard or normal distance between the support structure 302 and the row 112 during the calibration. For example, the perception processing unit 202 can use the calibration to configure (e.g., set) one or more of the distance thresholds discussed below (e.g., the first threshold and / or the second threshold).According to some embodiments, the perception processing unit 202 can skip this calibration in scenarios where the perception structure is the support structure 302.

[0040] According to some embodiments, the perception processing unit 202 can determine a transverse error with respect to the position of the support structure 302 in the space 114 based on the distance 306 between the support structure 302 and the row 112 (also referred to herein as the "first distance 306"). For example, the perception processing unit 202 can compare the first distance 306 with a first threshold value, which represents a safe distance (either in pixel or physical terms) between the support structure 302 and the space 114. As long as the first distance 306 does not exceed this first threshold value, the machine 100 is assumed to be at a safe distance from the row 112, and the transverse error is determined to be zero. However, if the first distance 306 exceeds the first threshold value, the amount by which the first distance 306 exceeds the first threshold value is determined as the transverse error.

[0041] According to some embodiments, in scenarios where the rows 112 are evenly or nearly evenly spaced, the box 304 can contain the row 112 without including the sensing structure, and the sensing processing unit 202 can determine the transverse error based on the extent (e.g., width) of the row 112 detected in the sensing data. In such scenarios, the width of the row 112 can indicate the position of the support structure 302 relative to the row 112. For example, during calibration, the sensing processing unit 202 can set (e.g., configure) a standard or normal width for the row 112. In response to the detection that the width of the row 112 falls below a threshold distance from this standard width, the sensing processing unit 202 can determine that the support structure 302 has intruded into the row 112.The threshold distance can be determined by the perception processing unit 202 during calibration. According to some embodiments, the perception processing unit 202 can determine the machine detection limit and the crop detection limit based on the row width 112 and the standard row width 112. According to some embodiments, the perception processing unit 202 can also determine which side of the machine 100 the support structure 302 has penetrated into the row, based, for example, on the extent of the gap 114 detected on one side of the box 304. For example, the perception processing unit 202 can determine that the support structure 302 has penetrated into the row 112 on the right side of the machine 100 in response to the determination that the gap 114 is detected on the left side of the box 304.The perception processing unit 202 can use this determination that the supporting structure 302 has penetrated the row 112 to control the machine 100, as discussed in various examples herein.

[0042] According to some embodiments, the perception processing unit 202 can generate control signals to control the control system 210 and / or the UI 212 in response to the detection of the lateral error. For example, in response to the detection of the lateral error, the perception processing unit 202 can generate a control signal to control the steering mechanism to steer away from the row 112 and / or to control the speed control mechanism to reduce the speed of the machine 100. According to some embodiments, the perception processing unit 202 can control the steering mechanism to adjust a steering angle away from the row 112 to a greater extent when the lateral error is higher, and / or to control the speed control mechanism to reduce the speed of the machine 100 to a greater extent when the lateral error is higher.For example, the Perceptual Processing Unit 202 can compare the lateral error to one or more second thresholds (each progressively higher than the first threshold), each corresponding to a steering angle (e.g., a progressively larger steering angle) and / or a reduction in speed (e.g., a progressively lower speed), and control the steering mechanism and / or the speed control mechanism accordingly, depending on which of the one or more second thresholds corresponds to the specific lateral error. In another example, the Perceptual Processing Unit 202 can input the lateral error into one or more functions to obtain values ​​for the steering angle and / or the reduction in speed, and control the steering mechanism and / or the speed control mechanism according to the obtained steering angle and / or reduction in speed.As discussed herein, the Perception Processing Unit 202 can generate the control signals to control the Control System 210 and / or the UI 212 based on the transverse error; however, some embodiments are not limited to this. According to some embodiments, the Perception Processing Unit 202 can generate the control signals to control the Control System 210 and / or the UI 212 based on a directional error. For example, the Perception Processing Unit 202 can generate a vector representing a directional error based on the transverse error determined over time (e.g., based on several iteratively determined transverse error values). According to some embodiments, the directional error can be used to perform control loop activities.

[0043] According to some embodiments, the perception processing unit 202 can, in addition to or as an alternative to the control of the control system described above, generate a control signal based on the lateral error to control one or more of the display screen, the audio speaker, the signal lights, etc., to issue a notification. The notification can include information corresponding to the adjustments of the steering angle and speed of the machine 100 described above in connection with the control system 210.

[0044] According to some embodiments, the perception processing unit 202 can generate control signals to control the control system 210 and / or the UI 212 in response to the determination that the lateral error has fallen to zero. For example, the perception processing unit 202 can generate the control signals to cause the steering angle of the machine 100 to straighten and / or the speed of the machine 100 to increase.

[0045] Fig. Figure 3B shows another example scenario of a machine 100 according to some embodiments, which travels along rows 112 of a field 110.

[0046] With reference to Fig. In example scenario 3B, the support structure 302 is at least partially positioned in row 112. For instance, as discussed above, regardless of the control by the perception processing unit 202, the support structure 302 of machine 100 may still deviate into row 112. In such circumstances, the perception processing unit 202 can detect that the support structure 302 has entered row 112, determine the extent of the crop damage caused by the support structure 302, and / or generate a crop damage map.

[0047] According to some embodiments, the perception processing unit 202 can detect pixel boundaries of the image received from the imaging device. For example, the perception processing unit 202 can detect pixel boundaries of the perception structure, the row 112 and / or the space 114, as described above in conjunction with Fig. 3A discussed. The perceptual processing unit 202 can also determine the first distance 306, as above in connection with Fig. 3A discussed. In response to the determination that the first distance 306 is zero (or below a third threshold distance), the perception processing unit 202 can determine that the support structure 302 of machine 100 has penetrated the row 112. In response to the penetration of the support structure 302 into the row 112, the perception processing unit 202 can generate control signals to the control system 210 and / or the UI 212 similarly to the above in conjunction with Fig. 3A discusses controlling in order to cause machine 100 to leave row 112.

[0048] According to some embodiments, in addition to generating the control signals in response to the determination that the support structure 302 has penetrated the row 112, the perception processing unit 202 can determine a second distance 308 (transverse distance) by which the support structure 302 has penetrated the row 112. For example, in implementations where the perception device 206 is the imaging device and the perception structure is the support structure 302, the perception processing unit 202 can determine the second distance 308 as a distance by which the pixel boundary of the support structure 302 (e.g., the "machine detection boundary") extends beyond the pixel boundary of the row 112 (e.g., the "crop detection boundary"). However, some embodiments are not limited to this. According to some embodiments, the perception processing unit 202 can determine the second distance 308 as a distance between disparity values ​​(e.g.,the disparity map discussed above in connection with the stereo camera), a distance in points of the point cloud (as discussed above in connection with the stereo camera and / or the distance measuring device), etc. In scenarios where the sensing structure is a physical structure of the machine 100 that is not the support structure 302, the sensing processing unit 202 can determine the second distance 308 by which the support structure 302 projects into the row 11, based on the known distance(s) (in one or more dimensions) between the sensing structure and the support structure 302. According to some embodiments, the sensing processing unit 202 can determine the machine sensing boundary as the edge of the support structure 302 based on the known distance(s) (in one or more dimensions) between the sensing structure and the support structure 302.The known distances between the perception structure and the support structure 302 and the perception data are determined, and distance 306 is determined as the distance by which the machine detection limit extends beyond the crop detection limit. The perception processing unit 202 can store the second distance 308 in memory 204, assigning it to the time at which the image was captured.

[0049] In addition to or as an alternative to the above, the perceptual processing unit 202 can input the perceptual data received from the perceptual device 206 into a trained machine learning model to obtain the second distance 308. For example, the machine learning model can be trained using training perceptual data to output the distance by which the machine detection boundary extends beyond the crop detection boundary as the second distance 308. The training perceptual data can include data (e.g., images containing distances, etc.), perceptual structures, rows 112, and / or gaps 114, and each can be assigned to a corresponding second distance 308.The machine learning model is trained by iteratively inputting a subset of the training perception data into the machine learning model, obtaining an output from the machine learning model based on the input training perception data, determining a difference between the output and the second distance 308 corresponding to the input image, and adjusting at least one parameter of the machine learning model in response to this difference. This iterative training can be performed until the machine learning model outputs a second distance 308 that is sufficiently close to the second distance 308 associated with the input perception data, with a reliability threshold. According to some embodiments, the machine learning model can be trained by the perception processing unit 202 or a processing circuit arrangement outside the system 200.According to some embodiments, the machine learning model can be stored in memory 204 or outside the system 200.

[0050] According to some embodiments, the machine learning model can be implemented by a processing circuit arrangement (e.g., the Perception Processing Unit 202 or another processing circuit arrangement). For example, the machine learning model can be an artificial neural network trained on a set of training data, for instance, by a supervised, unsupervised, and / or reinforcement learning model, wherein the processing circuit arrangement implementing the machine learning model can process a feature vector to provide an output based on the training. Such artificial neural networks can use a variety of organizational and processing models for artificial neural networks, such as…Convolutional neural networks (CNNs), recurrent neural networks (RNNs), which may include LSTM (Long Short Memory) units and / or GRU (Gated Recurrent Units), stacking deep neural networks (S-DNNs), state-space dynamic neural networks (S-SDNNs), unfolding networks, deep belief networks (DBNs), and / or constrained Boltzmann machines (RBMs). Alternatively or additionally, the processing circuitry may include other forms of artificial intelligence and / or machine learning, such as linear and / or logistic regression, statistical clustering, Bayesian classification, decision trees, dimensionality reduction such as principal component analysis and expert systems, and / or combinations thereof, including assemblies such as random forests.

[0051] The machine learning model can have any structure that is trainable, e.g., with training data. For example, the machine learning model can include an artificial neural network, a decision tree, a support vector machine, a Bayesian network, a genetic algorithm, and / or the like.The machine learning model can be implemented as an artificial neural network, such as a convolutional neural network (CNN), a region-based convolutional neural network (R-CNN), a region-suggesting neural network (RPN), a recurrent neural network (RNN), a stack-based deep neural network (S-DNN), a state-space dynamic neural network (S-SDNN), an unfolding network, a deep persuasion network (DBN), a constrained Boltzmann machine (RBM), a fully convolutional network, a long-term short-term memory network (LSTM), a classification network and / or the like, however, some embodiments of the invention are not limited thereto.

[0052] Regardless of whether the second distances 308 are obtained according to procedures similar to those obtained in connection with Fig. As discussed above in conjunction with 3A (in relation to implementations of imaging devices and / or distance measuring devices) or as an output of a machine learning model, the Perception Processing Unit 202 can use the second distances 308 to determine an extent of crop damage caused by the supporting structure. For example, the Perception Processing Unit 202 stores, as above in conjunction with Fig. 2 mentioned, also positions of machine 100 in memory 204 in conjunction with the respective times at which machine 100 was actually positioned in those positions. According to some embodiments, machine 100 can use the time-correlated second distances 308 and the time-correlated positions of machine 100 to determine an extent of crop damage caused by the support structure 302. For example, the perception processing unit 202 can correlate the second distances 308 with positions of machine 100 based on the time information and calculate an area of ​​crop damage based on the second distances 308 and the distances between the positions of machine 100 when the images (on which the second distances 308 are based) were taken.In an example, if the second intervals 308 were 0.1 meters for each of ten consecutive images, and if the machine positions 100 indicated that the machine 100 traveled 10 meters during the time the ten consecutive images were taken, the perceptual processing unit 202 can calculate the extent of crop damage as over 1 square meter.

[0053] According to some embodiments, the perception processing unit 202 can generate a map of the crop damage detected as described above. For example, the perception processing unit 202 can correlate the second distances 308 with the positions of the machine 100 based on the time information, as discussed above. The perception processing unit 202 can also generate a map of these correlations across the field 110. For example, the perception processing unit 202 can combine the correlations with other geospatial information of the field 110 (e.g., geographic dimensions, terrain, features, curvatures, rows 112, etc.) received from the positioning device 208 to geospatially represent the crop damage in relation to the field 110.The perception processing unit 202 can store the generated map in memory 204, output the generated map on the display screen and / or transmit the generated map to an external device using a transmitter of the system 200 (e.g. a transmitter on the machine 100).

[0054] According to some embodiments, the generated map can be used to differentiate between crop yield reductions resulting (1) from crop run over by the machine 100 and (2) from other reasons (e.g., the seed variety of the crop). For example, the generated map can be fed into a predictive crop yield system. This information can be used for planning future crop plantings to avoid or reduce crop damage caused by crop run over by the machine 100. For example, future crops can be planted in rows with less curvature to reduce the difficulty of keeping the support structure 302 of the machine 100 within the space 114 between the rows 112.

[0055] Fig. Section 4 presents a method for controlling a machine according to some exemplary embodiments. According to some embodiments, the method can be carried out by the perception processing unit 202.

[0056] With reference to Fig.4. The procedure in operation 402 can include detecting a machine detection limit and a crop detection limit. According to some embodiments, the machine detection limit and the crop detection limit can be detected based on pixel values ​​in an image acquired by the imaging device; however, some embodiments are not limited to this. According to some embodiments, the machine detection limit and the crop detection limit can be detected based on distances to them measured by the distance measuring device. The machine detection limit can refer to the support structure 302 of the machine 100, and the crop detection limit can refer to a row 112 of the crop.

[0057] In process 404, the procedure may include determining a transverse error, which is represented by the first distance 306 between the machine detection limit and the crop detection limit. According to exemplary embodiments, process 404 may include comparing the first distance 306 with a threshold value to determine the transverse error, as discussed above.

[0058] In operation 406, the procedure may involve controlling a control system or user interface (UI) based on the lateral error. For example, the control may result in an adjustment of the steering angle and / or speed of the machine 100, preventing or reducing crop damage resulting from the crop being run over by the support structure 302. Operations 402, 404, and 406 may be performed iteratively to control the machine 100.

[0059] According to some embodiments, the system 200 can overcome the limitations of conventional guidance systems to detect whether the support structures 302 of the machine 100 have deviated into a row 112. Accordingly, the system 200 is able to control the machine 100 based on the position of the support structures 302 relative to the row 112 in order to avoid or reduce crop damage. Furthermore, the system 200 is able to generate crop damage maps that can be used to differentiate crop yield reductions due to such damage from other causes and to plan the dimensions and orientations of future crop rows 112.

[0060] The various operations of the methods discussed above can be performed by any suitable device capable of carrying out the operations, such as the processing circuit arrangement discussed above. For example, as discussed above, the operations of the methods discussed above can be performed by various hardware and / or software implemented in any form of hardware (e.g., processor, ASIC, etc.).

[0061] The software may contain an ordered list of executable instructions for implementing logical functions and may be implemented in any “processor-readable medium” for use by or in conjunction with an instruction execution system, apparatus or device, such as a single- or multi-core processor or a processor-containing system.

[0062] The blocks or operations of a method or algorithm and the functions discussed in connection with some exemplary embodiments disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of both. If the functions are implemented in software, they may be stored or transmitted as one or more instructions or code on a tangible, non-volatile, computer-readable medium (e.g., memory 204).

[0063] According to some exemplary embodiments, the memory 204 can be a tangible, non-volatile, computer-readable medium, such as random access memory (RAM), flash memory, read-only memory (ROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, a hard disk, a removable disk, a compact disk (CD) ROM, any combination thereof, or any other form of storage medium known in the art.

[0064] Some exemplary embodiments can be discussed with reference to actions and symbolic representations of processes (e.g., in the form of flowcharts, data flow diagrams, structure diagrams, block diagrams, etc.) that can be implemented in connection with the units and / or devices discussed in more detail below. Although discussed in a particular way, a function or process specified in a particular block may be executed differently than in a flowchart, process diagram, etc. For example, functions or processes shown as being executed sequentially in two successive blocks may in reality be executed concurrently, simultaneously, at the same time, or in some cases, in reverse order.

[0065] When an element is described as "connected" or "coupled" to another element, it is understood that it can be directly connected or coupled to the other element, or that there can be intervening elements. The term "and / or" used here includes all combinations of one or more of the points listed.

[0066] Although terms like "first" or "second" may be used to describe different components (or parameters, values, etc.), the components (or parameters, values, etc.) are not limited to these terms. These terms should only be used to distinguish one component from another. For example, a "first" component may be called a "second" component, or similarly, and the "second" component may be called a "first" component. Expressions like "at least one of" preceding a list of elements modify the entire list of elements, not the individual elements within it. For example, the expression "at least one of a, b, and c" is to be understood as including only a, only b, only c, both a and b, both a and c, both b and c, all of a, b, and c, or all variations of the foregoing examples.

Claims

[1] System (200), comprising: a control system (210) or a user interface (UI) (212); and a processing circuit arrangement (202) designed to cause the system (200) to to detect a machine detection limit and a crop detection limit, wherein the machine detection limit belongs to a support structure (302) of a machine (100) and the crop detection limit belongs to a row (112) of plant material, to determine a transverse error represented by a first distance (306) between the machine detection limit and the crop detection limit, and to control the control system (210) or the UI (212) based on the lateral error. [2] System (200) according to claim 1, wherein: the system (200) comprises the control system (210), wherein the control system (210) includes: a steering mechanism designed to control a steering angle of the machine (100), or a speed control mechanism designed to control a speed of the machine (100); and the processing circuit arrangement (202) is designed to generate a control signal to control the steering angle or speed based on the lateral error. [3] System (200) according to claim 2, wherein: the control system (210) includes the speed control mechanism; and the processing circuit arrangement (202) is designed to generate a control signal to control the speed based on the transverse error. [4] System (200) according to claim 1, further comprising: a perception sensor (206) designed to generate perception data, wherein the perception sensor (206) is attached to the machine (100) and a field of view (304) of the perception sensor (206) includes the row (112) of plant material, wherein the processing circuit arrangement (202) is designed to detect the machine detection limit and the crop detection limit based on values ​​of the perception data. [5] System (200) according to claim 4, wherein the field of view (304) of the perception sensor (206) is directed towards a rear part of the machine (100). [6] Procedure, comprehensive: Determining a machine detection limit and a crop detection limit, wherein the machine detection limit belongs to a support structure (302) of a machine (100) and the crop detection limit belongs to a row (112) of plant material; Determining a transverse error represented by a first distance (306) between the machine detection limit and the crop detection limit; and Controlling a control system (210) or a user interface (UI) (212) based on the lateral error. [7] Method according to claim 6, wherein the control system (210) includes: a steering mechanism designed to control a steering angle of the machine (100), or a speed control mechanism designed to control a speed of the machine (100); and The control system (210) controls the steering system based on the lateral error, including generating a control signal to control the steering angle or speed. [8] Method according to claim 7, wherein the control system (210) includes the speed control mechanism; and The control system (210) controls the control system based on the lateral error, including generating a control signal to control the speed. [9] The method of claim 6, further comprising: Receiving perception data from a perception sensor (206) attached to an underside of the machine (100), wherein a field of view (304) of the perception sensor (206) includes the row (112) of plant material, where the detection includes the detection of the machine detection limit and the crop detection limit based on values ​​of the perception data. [10] Method according to claim 9, wherein the field of view (304) of the perception sensor (206) is directed towards a rear part of the machine. [11] Non-volatile computer-readable medium (204) on which instructions are stored which, when executed by at least one processor (202), cause the at least one processor (202) to perform a procedure comprising: Determining a machine detection limit and a crop detection limit, wherein the machine detection limit belongs to a support structure (302) of a machine (100) and the crop detection limit belongs to a row (112) of plant material; Determining a transverse error represented by a first distance (306) between the machine detection limit and the crop detection limit; and Controlling a control system (210) or a user interface (UI) (212) based on the lateral error. [12] Non-volatile computer-readable medium (204) according to claim 11, wherein the control system (210) includes: a steering mechanism designed to control a steering angle of the machine (100), or a speed control mechanism designed to control a speed of the machine (100); and The control system (210) controls the steering system based on the lateral error, including generating a control signal to control the steering angle or speed. [13] Non-volatile computer-readable medium (204) according to claim 12, wherein the control system (210) includes the speed control mechanism; and The control system (210) controls the control system based on the lateral error, including generating a control signal to control the speed. [14] Non-volatile computer-readable medium (204) according to claim 11, wherein the method further comprises: Receiving perception data from a perception sensor (206) attached to the underside of the machine (100), wherein a field of view (304) of the perception sensor (206) is directed towards a rear part of the machine (100), the field of view (304) includes the row (112) of plant material, and the detection includes the detection of the machine detection limit and the crop detection limit based on values ​​of the perception data. [15] Non-volatile computer-readable medium (204) according to claim 11, wherein the method further comprises: Receiving machine position information (100) from a positioning system (208); Determining a second distance (308) by which the supporting structure (302) has penetrated the row (112) of plant material, based on the machine detection limit and the crop detection limit; and Generating a geospatial map based on the position information and the second distance (308).

Images