A combine harvester automatic unloading method and system based on binocular vision

Abstract

This invention provides an automatic grain unloading method and system for combine harvesters based on binocular vision. The invention acquires binocular visual flow information of the unloading vehicle using a binocular camera installed slightly below the grain outlet of the unloading hopper. An instance segmentation neural network outputs a mask of the unloading area. The unloading sub-regions are dynamically divided based on the actual length of the minimum bounding rectangle and the recommended width of the single grain pile, with the center of each sub-region serving as the optimal unloading point. A unloading time-grain pile height mapping model is constructed to calculate the optimal unloading time for each point. A dual-axis decoupled mapping model is established between the actual unloading point and the lifting angle of the unloading hopper. A visual servo closed loop is constructed using pixel Euclidean distance to drive the unloading hopper to accurately track and fall into the circular deviation area, cyclically unloading according to the planned duration until all unloading is completed. This invention can robustly achieve fully automatic unloading online, effectively improving harvesting efficiency, reducing manual labor requirements, and promoting the intelligent development of combine harvesters.

Description

Technical Field

[0001] This invention belongs to the field of intelligent agricultural machinery technology, and in particular relates to an automatic grain unloading method and system for a combine harvester based on binocular vision. Background Technology

[0002] Automatic unloading of combine harvesters refers to the machine's ability to automatically unload harvested crops onto designated transport vehicles or storage facilities through its built-in system. This is an important research topic in the field of agricultural harvesting, involving improvements in modern agricultural production efficiency, reductions in labor costs, and the automation and intelligentization of agricultural harvesting machinery. In current agricultural production, unloading still largely requires manual operation. Operators are responsible not only for controlling the raising, lowering, and rotating of the unloading hopper but also for continuously monitoring the filling status of the grain silo to ensure accurate unloading and prevent overflow. This traditional unloading method not only reduces operational efficiency but can also lead to grain leakage or spillage due to operational errors, resulting in crop losses. Therefore, the development of automatic unloading methods and systems is of great significance.

[0003] Currently, the mainstream sensing methods for automated grain unloading technology mainly rely on traditional image processing or a single depth threshold setting to extract the target unloading area. However, these methods have serious limitations in actual field environments: on the one hand, their robustness is extremely limited, and they are prone to misjudgment when faced with rugged terrain, cluttered backgrounds, and interference from existing grain piles inside the truck; on the other hand, the high-intensity grain spraying, dust, and natural wind disturbances accompanying grain unloading operations severely degrade the quality of image and depth data acquired by the sensors, causing the sensing system to fail directly. In addition, existing research is mostly limited to large trucks and a specific stage of grain unloading, making it difficult to adapt to the more common small and medium-sized transfer equipment (such as agricultural tricycles) in actual agricultural production, and failing to form a complete closed loop. Particularly noteworthy is that, looking at the currently published related patents, the vast majority of technical solutions only focus on the design improvement of the unloading mechanical structure and the underlying motion control strategy, while systematic research on high-precision visual information processing and target perception in complex field environments remains relatively scarce. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides an automatic grain unloading method and system for combine harvesters based on binocular vision. This system enables robust online automatic grain unloading of combine harvesters in a non-contact manner, effectively improving agricultural harvesting efficiency.

[0005] This invention employs a binocular camera installed at the lower end of the grain outlet of a combine harvester to acquire binocular visual flow information. An instance segmentation neural network is used to obtain the corresponding mask for the unloading area containing the unloading vehicle's frame in the image. The grain stacking angle of repose characteristics and actual production experience values ​​are used to divide the unloading sub-regions and extract the corresponding optimal unloading points. The constructed unloading time-grain stack height model is used to calculate the optimal unloading time for each optimal unloading point. The actual unloading point is calculated using a dual-axis decoupled mapping model of the actual unloading point's two-dimensional image coordinates and the unloading hopper's lifting angle. Finally, a visual servo closed loop is constructed based on the pixel deviation of the landing point, driving the actual unloading point to accurately enter the target tolerance area and execute the planned unloading time. Fully automated unloading operations are achieved through iterative loops.

[0006] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0007] An automatic grain unloading method for a combine harvester based on binocular vision includes the following steps:

[0008] Step S1: Obtain the unloading area mask: Use a binocular camera fixedly installed at the lower end of the grain outlet of the unloading can of the combine harvester to obtain the binocular visual flow information of the unloading area of ​​the unloading vehicle. Input the left view image into the pre-trained instance segmentation neural network and output the corresponding mask of the unloading area containing the frame of the unloading vehicle.

[0009] Step S2: Determine the optimal unloading point Construct the minimum bounding rectangle based on the mask described in step S1, and calculate the length of the minimum bounding rectangle using binocular depth data. actual length It also incorporates a recommended stacking width for a single grain pile based on the grain stacking repose angle characteristics. The unloading area is divided into several unloading sub-areas, and the center point of each unloading sub-area is taken as the optimal unloading point.

[0010] Step S3: Calculate the optimal unloading time: based on the pre-calibrated grain pile height. With unloading time mapping model According to the depth value of each optimal unloading point in step S2 Grain unloading truck edge depth and overflow safety margin Calculate the corresponding optimal unloading time ;

[0011] Step S4: Calculate the actual unloading point Constructing the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading hopper. The biaxial decoupled mapping model, where the horizontal axis First-order linear fitting, ordinate A second-order polynomial fitting method was used, and the lifting angle of the unloading hopper was measured in real time. Calculate the current actual grain unloading point;

[0012] Step S5, Visual Servo Closed-Loop Grain Unloading: Using the optimal grain unloading point from step S2... Centered on a circle with a preset threshold Construct a circular deviation region for the radius and calculate the actual unloading point in step S4 in real time. with the best unloading point The pixel Euclidean distance between the unloading hoppers is used as the system deviation. The horizontal rotation and pitch of the unloading hopper are adjusted based on this distance until the actual unloading point falls within the circular deviation area. Then, the corresponding optimal unloading time is used. Perform unloading and cycle through to the next optimal unloading point until all points have completed unloading.

[0013] In the above scheme, the actual length in step S2 The calculation formula is:

[0014]

[0015] in, and The pixel length of the minimum bounding rectangle The three-dimensional coordinates of the two endpoints in the binocular camera coordinate system;

[0016] Furthermore, the number of sub-regions for unloading grain in step S2... Determine using the following formula:

[0017]

[0018] in, Recommended width for single-grain piles, pre-defined based on the grain stacking repose angle characteristics.

[0019] In the above scheme, the mapping model in step S3 Obtained through the following methods:

[0020] Under a fixed unloading flow rate of the harvester, multiple sets of unloading times were collected at fixed time intervals before the grain reached the critical point of repose and collapse. Corresponding grain pile height The data were processed using nonlinear least squares method for function fitting and averaging, where... The height of the grain pile The optimal unloading time is the time for unloading grain. The calculation formula is:

[0021]

[0022] Among them, the edge depth of the unloading vehicle It is obtained by extracting the average of the least least quantile of all effective depth values ​​within the mask.

[0023] In the above scheme, the dual-axis decoupling mapping model in step S4 is as follows:

[0024]

[0025]

[0026] in, The x-coordinate of the actual grain unloading point in the image. The vertical coordinate of the actual grain unloading point in the image. The lifting angle for unloading the grain hopper; Based on the horizontal offset constant, This is the linear compensation coefficient for lateral drift. Based on the height constant, and The first-order nonlinear compensation coefficient for the parabolic trajectory. is the second-order nonlinear compensation coefficient for the parabolic trajectory.

[0027] In the above scheme, in step S1, the binocular camera is fixedly installed at the lower end of the grain outlet of the unloading hopper of the combine harvester, and the optical axis of the binocular camera is oriented towards the unloading area of ​​the unloading vehicle, so that both the left and right perspectives can completely cover the unloading area.

[0028] An automatic grain unloading system for a combine harvester based on binocular vision is used to implement the automatic grain unloading method for the combine harvester based on binocular vision. The system includes an image acquisition module, an instance segmentation module, an optimal unloading point extraction module, an optimal unloading time calculation module, an actual unloading point calculation module, and a landing point correction and control and cyclic unloading module.

[0029] The image acquisition module is used to acquire binocular visual flow information of the unloading area of ​​the grain unloading truck using a binocular camera.

[0030] The instance segmentation module is used to input the left view image into a pre-trained instance segmentation neural network and output a corresponding mask containing the unloading area of ​​the unloading truck.

[0031] The optimal unloading point extraction module is used to construct a minimum bounding rectangle based on the mask, and calculate the length of the minimum bounding rectangle using binocular depth data. actual length It also incorporates a recommended stacking width for a single grain pile based on the grain stacking repose angle characteristics. The unloading area is divided into several unloading sub-areas, and the center point of each sub-area is extracted as the optimal unloading point. ;

[0032] The optimal unloading time calculation module is used based on a mapping model. Based on the depth value of each optimal unloading point Grain unloading truck edge depth and overflow safety margin Calculate the corresponding optimal unloading time ;

[0033] The actual unloading point calculation module is used to construct the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading cylinder. The biaxial decoupled mapping model, where the horizontal axis First-order linear fitting, ordinate A second-order polynomial fitting method was used, and the lifting angle of the unloading hopper was measured in real time. Calculate the current actual grain unloading point;

[0034] The landing point correction and cyclic unloading module is used to determine the optimal unloading point at the current time. Centered on a circle with a preset threshold Construct a circular deviation region for the radius and calculate the actual unloading point in real time. with the best unloading point The pixel Euclidean distance between the unloading hoppers is used to adjust the horizontal rotation and pitch of the unloading hopper until the actual unloading point falls within the circular deviation area. Then, the optimal unloading time is determined. Perform unloading and cycle through to the next optimal unloading point until all points have completed unloading.

[0035] The above solution also includes a binocular camera, an edge development board, an unloading electric control clutch motor, a lifting proportional valve for the unloading hopper, a left and right rotation servo motor for the unloading hopper, and a rotation angle sensor for the unloading hopper.

[0036] The binocular camera is installed at the lower end of the grain outlet of the combine harvester's unloading canister to acquire binocular visual flow information of the unloading area of ​​the unloading vehicle.

[0037] The edge development board is connected to the binocular camera, the unloading electric clutch motor controller, the unloading cylinder lifting proportional valve controller, the unloading cylinder left and right rotation servo motor, and the unloading cylinder rotation angle sensor, respectively. The edge development board is embedded with an instance segmentation neural network. The left view image captured by the binocular camera is input into the instance segmentation neural network, and the corresponding mask containing the unloading area of ​​the unloading vehicle frame is output.

[0038] The output shaft of the unloading electric clutch motor is mechanically connected to the unloading clutch mechanism; the input end of the unloading electric clutch motor controller is connected to the edge development board, and the output end of the unloading electric clutch motor controller is electrically connected to the unloading electric clutch motor.

[0039] The hydraulic interface of the unloading hopper lifting proportional valve is connected to the pitch drive cylinder of the unloading hopper; the input end of the unloading hopper lifting proportional valve controller is connected to the edge development board, and the output end of the unloading hopper lifting proportional valve controller is electrically connected to the unloading hopper lifting proportional valve.

[0040] The output shaft of the servo motor for left and right rotation of the unloading cylinder is connected to the rotary mechanism of the unloading cylinder; the control terminal of the servo motor for left and right rotation of the unloading cylinder is connected to the edge development board.

[0041] The unloading hopper rotation angle sensor is installed on the rotation axis and / or pitch axis of the unloading hopper. The signal output terminal of the unloading hopper rotation angle sensor is connected to the edge development board for real-time detection of the lifting angle of the unloading hopper. .

[0042] In the above scheme, the edge development board is also configured to implement the functions of the image acquisition module, instance segmentation module, optimal unloading point extraction module, optimal unloading time calculation module, actual unloading point calculation module, and landing point correction control and cyclic unloading module.

[0043] A combine harvester includes the aforementioned binocular vision-based automatic grain unloading system for combine harvesters.

[0044] Compared with the prior art, the beneficial effects of the present invention are:

[0045] This invention acquires binocular visual flow information using a binocular camera installed slightly below the grain outlet of the unloading hopper of a combine harvester.

[0046] This method employs an instance segmentation neural network to obtain masks corresponding to unloading areas in images containing the unloading truck's outline. It then uses the grain stacking repose angle characteristics and actual production experience to divide unloading sub-regions and extract corresponding optimal unloading points. A constructed unloading time-grain pile height model is used to calculate the optimal unloading time for each point. A dual-axis decoupled mapping model, using the two-dimensional image coordinates of the actual unloading point and the unloading hopper lifting angle, is used to calculate the actual unloading point. Finally, a visual servo closed loop is constructed based on the pixel deviation of the landing point, driving the actual unloading point to accurately enter the target tolerance area and execute the planned unloading time. Through iterative iteration, fully automated unloading operations are achieved. This method enables non-contact, robust online automatic unloading of combine harvesters, effectively improving agricultural harvesting efficiency, reducing labor requirements, and providing a strong impetus for the automation and intelligence of combine harvesting machinery. Attached Figure Description

[0047] Figure 1 This is a flowchart illustrating one embodiment of the present invention.

[0048] Figure 2 This is an overall architecture diagram of one embodiment of the present invention.

[0049] Figure 3 This is an embodiment of the present invention, showing how a camera is mounted on a combine harvester and a grain unloading vehicle.

[0050] Figure 4 This is an example segmentation result and minimum bounding rectangle extraction according to an embodiment of the present invention.

[0051] Figure 5 This is a theoretical diagram of sub-unloading area division and optimal unloading point extraction according to an embodiment of the present invention.

[0052] Figure 6 This is a sub-unloading area division and optimal unloading point extraction and detection diagram according to an embodiment of the present invention.

[0053] Figure 7 This is a method for calibrating the grain stacking height and unloading time according to one embodiment of the present invention.

[0054] Figure 8 This is a calculation of grain unloading time according to one embodiment of the present invention.

[0055] Figure 9 This is an embodiment of the automatic grain unloading process of the present invention, wherein... Figure 9 (a) Initial deviation correction; Figure 9 (b) Correct by turning to the right; Figure 9 (c) Adjusting the attitude downwards; Figure 9 (d) Fine-tune the upward posture; Figure 9 (e) Point A location is complete; Figure 9 (f) Grain unloading begins at point A; Figure 9 (g) Switch to point B for positioning; Figure 9 (h) Grain unloading begins at point B; Figure 9 (i) Grain unloading begins at point C. Detailed Implementation

[0056] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0057] Figure 1The image shows a preferred embodiment of the binocular vision-based automatic grain unloading method for combine harvesters, which includes the following steps:

[0058] Step S1: Obtain the unloading area mask: Use a binocular camera 1 fixedly installed at the lower end of the grain outlet of the unloading hopper 10 of the combine harvester to obtain the binocular visual flow information of the unloading area of ​​the unloading vehicle. Input the left view image into the pre-trained instance segmentation neural network and output the corresponding mask of the unloading area containing the frame of the unloading vehicle.

[0059] Step S2: Determine the optimal unloading point Construct the minimum bounding rectangle based on the mask described in step S1, and calculate the length of the minimum bounding rectangle using binocular depth data. actual length It also incorporates a recommended stacking width for a single grain pile based on the grain stacking repose angle characteristics. The unloading area is divided into several unloading sub-areas, and the center point of each unloading sub-area is taken as the optimal unloading point.

[0060] Step S3: Calculate the optimal unloading time: based on the pre-calibrated grain pile height. With unloading time mapping model Based on the depth value of each optimal unloading point in step S2 Grain unloading truck edge depth and overflow safety margin Calculate the corresponding optimal unloading time ;

[0061] Step S4: Calculate the actual unloading point Constructing the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading hopper. The biaxial decoupled mapping model, where the horizontal axis First-order linear fitting, ordinate A second-order polynomial fitting method was used, and the lifting angle of the unloading hopper was measured in real time. Calculate the current actual grain unloading point;

[0062] Step S5, Visual Servo Closed-Loop Grain Unloading: Using the optimal grain unloading point from step S2... Centered on a circle with a preset threshold Construct a circular deviation region for the radius and calculate the actual unloading point in step S4 in real time. with the best unloading point The pixel Euclidean distance between the unloading hoppers is used as the system deviation. Based on this distance, the horizontal rotation and pitch of the unloading hopper 10 are adjusted until the actual unloading point falls within the circular deviation area. Then, the corresponding optimal unloading time is used. Perform unloading and cycle through to the next optimal unloading point until all points have completed unloading.

[0063] like Figure 4 As shown, according to this embodiment, preferably, the acquired image is input into a pre-trained instance segmentation neural network, and the neural network directly outputs the corresponding mask of the unloading area in the image that contains the frame of the unloading truck.

[0064] According to this embodiment, preferably,

[0065] The actual length in step S2 The calculation formula is:

[0066]

[0067] in, and The pixel length of the minimum bounding rectangle The two endpoints are in the three-dimensional coordinates of the binocular camera coordinate system; the depth data of the three-dimensional coordinates can be directly obtained in the binocular visual flow information.

[0068] According to this embodiment, the preferred number of sub-regions for grain unloading is... Determine using the following formula:

[0069]

[0070] in, Recommended width for single-grain piles, pre-defined based on the grain stacking repose angle characteristics.

[0071] in, In this embodiment, the number of sub-regions is [number]. The value is 3.

[0072] According to this embodiment, preferably, the recommended width of the single-grain pile stack is... In this embodiment, the angle of repose of grain accumulation is pre-set based on actual production experience. 45 .

[0073] like Figure 5 and Figure 6 As shown, according to this embodiment, preferably, the number of sub-regions is determined. Divide the smallest bounding rectangle into equal parts, and take the center point of each equally divided rectangle as the optimal unloading point for that area.

[0074] like Figure 7As shown, according to this embodiment, preferably, under a fixed unloading flow rate of the harvester, multiple sets of 'time-height accumulation' data are collected at fixed time intervals before the grain reaches the critical angle of repose and collapse. Subsequently, the above-mentioned multiple sets of experimental data are subjected to function fitting and averaging using the nonlinear least squares method to establish the relevant mapping relationship.

[0075] Mapping Model Obtained through the following methods:

[0076] Under a fixed unloading flow rate of the harvester, multiple sets of unloading times were collected at fixed time intervals before the grain reached the critical point of repose and collapse. Corresponding grain pile height The data were processed using nonlinear least squares method for function fitting and averaging, where... The height of the grain pile This refers to the time for unloading grain.

[0077] like Figure 8 As shown, according to this embodiment, preferably, the optimal grain unloading time... The calculation formula is:

[0078]

[0079] in, This is the optimal time for unloading grain;

[0080] The optimal depth for unloading grain;

[0081] This refers to the depth of the grain unloading truck's frame.

[0082] To prevent overflow, a safety margin is set based on experience in this embodiment. 2 .

[0083] The depth of the grain unloading truck's side frame It is obtained by extracting the average of the least least quantile of all effective depth values ​​within the mask.

[0084] Furthermore, the depth of the unloading vehicle's frame is obtained by combining a mask of the unloading area containing the frame of the unloading vehicle with stereo depth data.

[0085] According to this embodiment, preferably, the edge depth of the grain unloading vehicle is... The statistical acquisition method is as follows: extract all effective depth values ​​greater than 0 within the entire mask area, and directly calculate the average of the lowest quantiles (in this embodiment, depth values ​​in the range of 1% to 5%) at the minimum end of the distribution, using this as the reference depth for the edge of the carriage. .

[0086] According to this embodiment, preferably, the actual grain unloading point in step S4... The calculation method is as follows: Construct a dual-axis decoupled mapping model between the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading drum. Perform second-order polynomial fitting on the vertical coordinate of the image, which is significantly affected by the parabolic trajectory of the grain, and perform first-order linear fitting on the horizontal coordinate of the image, which is only affected by minor mechanical yaw and perspective distortion.

[0087] The dual-axis decoupling mapping model is as follows:

[0088]

[0089]

[0090] in, The x-coordinate of the actual grain unloading point in the image. The vertical coordinate of the actual grain unloading point in the image. The lifting angle for unloading the grain hopper; Based on the horizontal offset constant, This is the linear compensation coefficient for lateral drift. Based on the height constant, and The first-order nonlinear compensation coefficient for the parabolic trajectory. is the second-order nonlinear compensation coefficient for the parabolic trajectory.

[0091] like Figure 9 As shown in this embodiment, preferably, the combine harvester achieves the entire process of automatic grain unloading on the unloading truck.

[0092] Phase 1: Initial calibration of the actual unloading point and the expected unloading point (optimal unloading point)

[0093] Figure 9 (a) Initial deviation correction - when the status is "Not FULL FRAME", the unloading drum rotates to the right. At this time, there are lateral and longitudinal deviations between the actual unloading point and the expected unloading point.

[0094] Phase 2: Location and unloading of the desired unloading point A

[0095] Figure 9 (b) Desired unloading point A positioning and frame correction - the status changes to "Full frame detected", the unloading hopper continues to rotate to the right, with the goal of aligning with the desired unloading point A.

[0096] Figure 9 (c) Attitude adjustment (downward) - The unloading can rotates downward to further reduce the deviation from point A.

[0097] Figure 9 (d) Attitude adjustment (upward) - The unloading can rotates upward to achieve precise alignment with the desired unloading point A.

[0098] Figure 9 (e) Point A positioning completed - the deviation is less than the preset threshold, point A positioning completed.

[0099] Figure 9 (f) Grain unloading begins at point A – Grain unloading is carried out according to the corresponding optimal unloading time.

[0100] Phase 3: Location and unloading of the desired unloading point B

[0101] Figure 9 (g) Grain unloading at point A is completed, then switch to point B - the status is still "complete unloading area". The actual unloading point and the expected unloading point B are marked in the image at the same time. The unloading drum rotates to the right and switches to point B after unloading at point A is completed.

[0102] Figure 9 (h) Point B positioning completed - Point B deviation is less than the threshold, positioning completed, start unloading grain.

[0103] Phase 4: Location and unloading of the desired unloading point C

[0104] Figure 9 (i) C-point positioning and unloading - The status is "complete unloading area". Mark the actual unloading point and the expected unloading point C. After C-point positioning is completed, unloading is performed.

[0105] Process Summary: The entire automatic grain unloading process follows a cyclical logic of "positioning correction → threshold determination → grain unloading → target point switching", and completes the precise unloading of grain at the three optimal unloading points A, B, and C in sequence.

[0106] Preferably, in step S1, the binocular camera 1 is fixedly installed at the lower end of the grain outlet of the unloading hopper 10 of the combine harvester, and the optical axis of the binocular camera 1 is oriented towards the unloading area of ​​the unloading vehicle, so that both the left and right perspectives can completely cover the unloading area.

[0107] An automatic grain unloading system for a combine harvester based on binocular vision is used to implement the automatic grain unloading method for the combine harvester based on binocular vision. The system includes an image acquisition module, an instance segmentation module, an optimal unloading point extraction module, an optimal unloading time calculation module, an actual unloading point calculation module, and a landing point correction and control and cyclic unloading module.

[0108] The image acquisition module is used to acquire binocular visual flow information of the unloading area of ​​the grain unloading truck using the binocular camera 1. Specifically, the binocular camera is stably installed and fixed at the lower end of the grain outlet of the unloading hopper of the combine harvester. The distance between the binocular camera and the unloading hopper wall is optimized according to the lifting angle of the unloading hopper during unloading to ensure that both the left and right perspectives can stably capture images of the unloading area of ​​the grain unloading truck, obtaining binocular visual flow information of acceptable quality.

[0109] The instance segmentation module is used to obtain the unloading area in the obtained image data; specifically, it is used to input the left view image into a pre-trained instance segmentation neural network, and the neural network directly outputs the corresponding mask of the unloading area containing the frame of the unloading truck.

[0110] The optimal unloading point extraction module extracts the calculated optimal unloading point within the unloading area. Specifically, it constructs a minimum bounding rectangle based on the geometric features of the unloading area mask obtained by the instance segmentation module, and locates the length and width of this rectangle. The length of the minimum bounding rectangle is then calculated using binocular depth data from the binocular visual flow information. actual length It also incorporates a recommended stacking width for a single grain pile based on the grain stacking repose angle characteristics. The unloading area is divided into several unloading sub-areas, and the center point of each sub-area is extracted as the optimal unloading point. ;

[0111] The optimal unloading time calculation module is used to calculate the optimal unloading time at each optimal unloading point; specifically based on a mapping model. Based on the depth value of each optimal unloading point Grain unloading truck edge depth and overflow safety margin Calculate the corresponding optimal unloading time Specifically, under a fixed unloading flow rate of the harvester, multiple sets of 'time-height accumulation' data were collected at fixed time intervals before the grain reached the critical point of repose and collapse. The nonlinear least squares method was used to perform function fitting and averaging on these multiple sets of experimental data to establish relevant mapping relationships. The optimal unloading time was calculated by incorporating the reference depth at the edge of the truck bed and the depth at each optimal unloading point.

[0112] The actual unloading point calculation module is used to calculate the position of the actual grain unloading point in the image; specifically, it constructs the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading drum. The biaxial decoupled mapping model, where the horizontal axis First-order linear fitting, ordinate A second-order polynomial fitting method was used, and the lifting angle of the unloading hopper was measured in real time. Calculate the current actual unloading point; perform second-order polynomial fitting on the vertical coordinate of the image, which is significantly affected by the parabolic trajectory of the grain, and perform first-order linear fitting on the horizontal coordinate of the image, which is only affected by minor mechanical yaw and perspective distortion, thereby calculating the position of the actual unloading point in the image.

[0113] The landing point correction and control and cyclic unloading module is used to realize cyclic automatic unloading; specifically, it uses the current optimal unloading point. Centered on a circle with a preset threshold Construct a circular deviation region for the radius and calculate the actual unloading point in real time. with the best unloading point The pixel Euclidean distance between the unloading hopper (10) is used to adjust the horizontal rotation and pitch of the unloading hopper (10) until the actual unloading point falls within the circular deviation area. Then, the corresponding optimal unloading time is used. Perform unloading and cycle through to the next optimal unloading point until all points have completed unloading.

[0114] like Figure 2 As shown, according to this embodiment, preferably, the system includes a binocular camera 1, an edge development board 2, an unloading electric clutch motor 3, an unloading electric clutch motor controller 4, an unloading cylinder lifting proportional valve 5, an unloading cylinder lifting proportional valve controller 6, an unloading cylinder left and right rotation servo motor 8, an unloading cylinder rotation angle sensor 7, and a CAN bus (9).

[0115] The binocular camera 1 is installed at the lower end of the grain outlet of the unloading canister 10 of the combine harvester, and is used to acquire binocular visual flow information of the unloading area of ​​the unloading vehicle.

[0116] like Figure 3 As shown, according to this embodiment, preferably, the binocular camera 1 is stably installed and fixed at the lower end of the grain outlet of the combine harvester's unloading canister using a bracket, with a distance of 400mm from the top of the grain outlet; the binocular camera 1 faces the unloading truck and ensures that both the left and right perspectives can clean and stably photograph the unloading area of ​​the unloading truck to obtain binocular visual flow information that meets the quality standards.

[0117] The edge development board 2 is connected to the binocular camera 1, the unloading electric clutch motor controller 4, the unloading cylinder lifting proportional valve controller 6, the unloading cylinder left and right rotation servo motor 8, and the unloading cylinder rotation angle sensor 7 respectively; the edge development board 2 is embedded with an instance segmentation neural network, which inputs the left view image collected by the binocular camera 1 into the instance segmentation neural network and outputs the corresponding mask of the unloading area containing the unloading vehicle frame.

[0118] The output shaft of the unloading electric clutch motor 3 is mechanically connected to the unloading clutch mechanism; the input end of the unloading electric clutch motor controller 4 is connected to the edge development board 2, and the output end of the unloading electric clutch motor controller 4 is electrically connected to the unloading electric clutch motor 3.

[0119] The hydraulic interface of the unloading drum lifting proportional valve 5 is connected to the pitch drive cylinder of the unloading drum 10; the input end of the unloading drum lifting proportional valve controller 6 is connected to the edge development board 2, and the output end of the unloading drum lifting proportional valve controller 6 is electrically connected to the unloading drum lifting proportional valve 5.

[0120] The output shaft of the left and right rotation servo motor 8 of the unloading cylinder is connected to the rotation mechanism of the unloading cylinder 10; the control end of the left and right rotation servo motor 8 of the unloading cylinder is connected to the edge development board 2.

[0121] The unloading cylinder rotation angle sensor 7 is installed on the rotation axis and / or pitch axis of the unloading cylinder 10. The signal output terminal of the unloading cylinder rotation angle sensor 7 is connected to the edge development board 2 for real-time detection of the lifting angle of the unloading cylinder 10. .

[0122] The edge development board 2 is also configured to implement the functions of the image acquisition module, instance segmentation module, optimal unloading point extraction module, optimal unloading time calculation module, actual unloading point calculation module, and landing point correction and control and cyclic unloading module.

[0123] A combine harvester includes the aforementioned binocular vision-based automatic grain unloading system for combine harvesters.

[0124] This invention utilizes a binocular camera to acquire binocular visual flow information of the unloading area of ​​a grain unloading truck, and uses an instance segmentation neural network to obtain the corresponding mask of the unloading area containing the frame of the grain unloading truck in the image; it then divides the unloading sub-regions and extracts the corresponding optimal unloading points. Construct a grain unloading time-grain pile height model and calculate the optimal unloading time for each optimal unloading point; construct a biaxial decoupled mapping model of the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading hopper, and calculate the actual unloading point. Finally, a visual servo closed loop is constructed based on the pixel deviation of the unloading point, driving the actual unloading point to accurately enter the target tolerance area and execute the planned unloading time. Fully automated unloading is achieved through iterative iteration. This invention enables non-contact, robust online unloading of combine harvesters, effectively improving agricultural harvesting efficiency, reducing labor requirements, and providing a strong impetus for the automation and intelligence of combine harvesting machinery.

[0125] It should be understood that although this specification is described according to various embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other implementation methods that can be understood by those skilled in the art.

[0126] The detailed descriptions listed above are merely specific illustrations of feasible embodiments of the present invention and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for automatic grain unloading in a combine harvester based on binocular vision, characterized in that, Includes the following steps: Step S1: Obtain the unloading area mask: Use a binocular camera (1) fixedly installed at the lower end of the grain outlet of the unloading hopper (10) of the combine harvester to obtain the binocular visual flow information of the unloading area of ​​the unloading vehicle. Input the left view image into the pre-trained instance segmentation neural network and output the corresponding mask of the unloading area containing the frame of the unloading vehicle. Step S2: Determine the optimal unloading point Construct the minimum bounding rectangle based on the mask described in step S1, and calculate the length of the minimum bounding rectangle using binocular depth data. actual length It also incorporates a recommended stacking width for a single grain pile based on the grain stacking repose angle characteristics. The unloading area is divided into several unloading sub-areas, and the center point of each unloading sub-area is taken as the optimal unloading point. Step S3: Calculate the optimal unloading time: based on the pre-calibrated grain pile height. With unloading time mapping model According to the depth value of each optimal unloading point in step S2 Grain unloading truck edge depth and overflow safety margin Calculate the corresponding optimal unloading time ; Step S4: Calculate the actual unloading point Constructing the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading hopper. The biaxial decoupled mapping model, where the horizontal axis First-order linear fitting, ordinate A second-order polynomial fitting method was used, and the lifting angle of the unloading hopper was measured in real time. Calculate the current actual grain unloading point; Step S5, Visual Servo Closed-Loop Grain Unloading: Using the optimal grain unloading point from step S2... Centered on a circle with a preset threshold Construct a circular deviation region for the radius and calculate the actual unloading point in step S4 in real time. with the best unloading point The pixel Euclidean distance between the two is used as the system deviation. The horizontal rotation and pitch of the unloading hopper (10) are adjusted according to this distance until the actual unloading point falls into the circular deviation area. Then, the corresponding optimal unloading time is used. Perform unloading and cycle through to the next optimal unloading point until all points have completed unloading.

2. The automatic grain unloading method for a combine harvester based on binocular vision according to claim 1, characterized in that, The actual length in step S2 The calculation formula is: in, and The pixel length of the minimum bounding rectangle The three-dimensional coordinates of the two endpoints in the binocular camera coordinate system.

3. The automatic grain unloading method for a combine harvester based on binocular vision according to claim 2, characterized in that, The number of sub-regions for unloading grain in step S2 Determine by the following formula: in, Recommended width for single-grain piles, pre-defined based on the grain stacking repose angle characteristics.

4. The automatic grain unloading method for a combine harvester based on binocular vision according to claim 1, characterized in that, The mapping model in step S3 Obtained through the following methods: Under a fixed unloading flow rate of the harvester, multiple sets of unloading times were collected at fixed time intervals before the grain reached the critical point of repose and collapse. Corresponding grain pile height The data were processed using nonlinear least squares method for function fitting and averaging, where... The height of the grain pile For unloading time; The optimal unloading time The calculation formula is: Among them, the edge depth of the unloading vehicle It is obtained by extracting the average of the least least quantile of all effective depth values ​​within the mask.

5. The automatic grain unloading method for a combine harvester based on binocular vision according to claim 1, characterized in that, The dual-axis decoupling mapping model in step S4 is as follows: in, The x-coordinate of the actual grain unloading point in the image. The vertical coordinate of the actual grain unloading point in the image. The lifting angle for unloading the grain hopper; Based on the horizontal offset constant, This is the linear compensation coefficient for lateral drift. Based on the height constant, and The first-order nonlinear compensation coefficient for the parabolic trajectory. is the second-order nonlinear compensation coefficient for the parabolic trajectory.

6. The automatic grain unloading method for a combine harvester based on binocular vision according to claim 1, characterized in that, In step S1, the binocular camera (1) is fixedly installed at the lower end of the grain outlet of the unloading hopper (10) of the combine harvester. The optical axis of the binocular camera (1) is oriented towards the unloading area of ​​the unloading vehicle, so that both the left and right perspectives can completely cover the unloading area.

7. An automatic grain unloading system for a combine harvester based on binocular vision, used to implement the automatic grain unloading method for a combine harvester based on binocular vision as described in any one of claims 1 to 6, characterized in that, It includes an image acquisition module, an instance segmentation module, an optimal unloading point extraction module, an optimal unloading time calculation module, an actual unloading point calculation module, and a landing point correction and control and cyclic unloading module; The image acquisition module is used to acquire binocular visual flow information of the unloading area of ​​the grain unloading truck using a binocular camera (1); The instance segmentation module is used to input the left view image into a pre-trained instance segmentation neural network and output a corresponding mask containing the unloading area of ​​the unloading truck. The optimal unloading point extraction module is used to construct a minimum bounding rectangle based on the mask, and calculate the length of the minimum bounding rectangle using binocular depth data. actual length It also incorporates a recommended stacking width for a single grain pile based on the grain stacking repose angle characteristics. The unloading area is divided into several unloading sub-areas, and the center point of each sub-area is extracted as the optimal unloading point. ; The optimal unloading time calculation module is used based on a mapping model. Based on the depth value of each optimal unloading point Grain unloading truck edge depth and overflow safety margin Calculate the corresponding optimal unloading time ; The actual unloading point calculation module is used to construct the two-dimensional image coordinates of the actual unloading point and the lifting angle of the unloading cylinder. The biaxial decoupled mapping model, where the horizontal axis First-order linear fitting, ordinate A second-order polynomial fitting method was used, and the lifting angle of the unloading hopper was measured in real time. Calculate the current actual grain unloading point; The landing point correction and cyclic unloading module is used to determine the optimal unloading point at the current time. Centered on a circle with a preset threshold Construct a circular deviation region for the radius and calculate the actual unloading point in real time. with the best unloading point The pixel Euclidean distance between the unloading hopper (10) is used to adjust the horizontal rotation and pitch of the unloading hopper (10) until the actual unloading point falls within the circular deviation area. Then, the corresponding optimal unloading time is used. Perform unloading and cycle through to the next optimal unloading point until all points have completed unloading.

8. The automatic grain unloading system for a combine harvester based on binocular vision according to claim 7, characterized in that, It also includes a binocular camera (1), an edge development board (2), an unloading electric control clutch motor (3), a lifting proportional valve for the unloading cylinder (5), a left and right rotation servo motor for the unloading cylinder (8), and a rotation angle sensor for the unloading cylinder (7). The binocular camera (1) is installed at the lower end of the grain outlet of the unloading canister (10) of the combine harvester to obtain binocular visual flow information of the unloading area of ​​the unloading vehicle; The edge development board (2) is connected to the binocular camera (1), the unloading electric clutch motor controller (4), the unloading cylinder lifting proportional valve controller (6), the unloading cylinder left and right rotation servo motor (8), and the unloading cylinder rotation angle sensor (7), respectively. The edge development board (2) is embedded with an instance segmentation neural network. The left view image collected by the binocular camera (1) is input into the instance segmentation neural network, and the corresponding mask containing the unloading area of ​​the unloading vehicle frame is output. The output shaft of the unloading electric clutch motor (3) is mechanically connected to the unloading clutch mechanism; the input end of the unloading electric clutch motor controller (4) is connected to the edge development board (2), and the output end of the unloading electric clutch motor controller (4) is electrically connected to the unloading electric clutch motor (3). The hydraulic interface of the unloading cylinder lifting proportional valve (5) is connected to the pitch drive cylinder of the unloading cylinder (10); the input end of the unloading cylinder lifting proportional valve controller (6) is connected to the edge development board (2), and the output end of the unloading cylinder lifting proportional valve controller (6) is electrically connected to the unloading cylinder lifting proportional valve (5). The output shaft of the left and right rotation servo motor (8) of the unloading cylinder is connected to the rotation mechanism of the unloading cylinder (10); the control end of the left and right rotation servo motor (8) of the unloading cylinder is connected to the edge development board (2). The unloading cylinder rotation angle sensor (7) is installed on the rotation axis and / or pitch axis of the unloading cylinder (10). The signal output terminal of the unloading cylinder rotation angle sensor (7) is connected to the edge development board (2) for real-time detection of the lifting angle of the unloading cylinder (10). .

9. The automatic grain unloading system for a combine harvester based on binocular vision according to claim 8, characterized in that, The edge development board (2) is also configured to implement the functions of the image acquisition module, instance segmentation module, optimal unloading point extraction module, optimal unloading time calculation module, actual unloading point calculation module, and landing point correction control and cyclic unloading module.

10. A combine harvester, characterized in that, The automatic grain unloading system for combine harvesters based on binocular vision, as described in any one of claims 7-9, is included.

Images