Multi-modal fusion corn harvester header height self-adaptive adjusting method and multi-modal fusion corn harvester header height self-adaptive adjusting system

Abstract

The invention relates to a multi-modal fusion corn harvester header height adaptive adjustment method and system, and belongs to the technical field of automation control, the method comprises the following steps: using a binocular camera to collect crop images, and preprocessing the crop images to obtain a depth map; the image is input into a YOLOV8n model, and coordinates of a center point at the bottom of a bounding box of the corncob are obtained through model operation; performing plane fitting on the depth map to obtain a ground height; calculating a corn ear height initial value based on the ground height and the center point coordinate; vibration data of working of the harvester is collected to perform vibration compensation on the initial height value to obtain a height correction value; and dynamically adjusting the header height of the harvester by adopting a dynamic fuzzy PID algorithm based on the correction value. According to the method, the target detection model and the dynamic fuzzy PID algorithm are combined, so that the problems of low visual detection precision and control response lag of existing agricultural machinery in a complex field scene are solved, and crop loss in harvesting operation is effectively reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of agricultural machinery automation control, and in particular relates to a multi-modal fusion corn harvester header height adaptive adjustment method and system. Background Art

[0002] The genetic material of seed corn kernels is extremely sensitive and has a very low tolerance for mechanical damage. The damage rate must be strictly controlled within 2% to ensure the germination rate of the seeds and the reliability of subsequent planting.

[0003] Traditional corn harvesters control the harvesting height of the header by preset heights or manual intervention. There is a certain degree of lag and inaccuracy, and it is difficult to adapt to the complex and changeable terrain of the field and the differences in plant heights in real time, which can easily lead to corn ears being hit or missed, and the damage rate can rise to 4% - 6%, which seriously exceeds the harvesting standards for breeding corn, has a serious impact on seed quality, and brings huge losses to the breeding industry.

[0004] To solve this problem, the existing technology usually adopts a monocular vision system combined with a fixed parameter PID algorithm to control the harvesting height of the harvesting table. Although this method reduces the harvesting damage rate to a certain extent, the monocular vision system is easily affected by dense planting and canopy reflection interference, the ear positioning error is large, and the false detection rate surges in rainy weather, which cannot meet the seed-level harvesting accuracy requirements. In addition, the fixed parameter PID algorithm has a large response delay at complex operating speeds, and the vibration of the harvesting table can easily cause secondary damage. It is still difficult to meet the damage rate requirements for harvesting breeding corn. Summary of the invention

[0005] The purpose of the present invention is to solve one of the above-mentioned technical problems and to provide a method and system for adaptively adjusting the height of a corn harvester header using multi-modal fusion.

[0006] To achieve the above object, the technical solution adopted by the present invention is: A multi-modal fusion corn harvester header height adaptive adjustment method comprises the following steps: A binocular camera is used to collect raw images of unharvested crops in front of the corn harvester in real time, and the raw images are preprocessed to obtain a depth map; Input the original image and depth map into the pre-trained YOLOV8n model, output the bounding box of the ear of corn in the image through model operation, and calculate the coordinates of the bottom center point of the bounding box; The ground height is obtained by plane fitting based on the depth map and the plane equation; Convert the coordinates of the center point at the bottom of the bounding box to the camera coordinate system based on the ground height and depth values in the depth map, calculate the height of the corn ear root based on its coordinates in the camera coordinate system, and then calculate the initial value of the corn ear height based on the height of the corn ear root; Collect vibration data during the operation of the corn harvester through sensors; fuse the vibration data with the initial value of the corn ear height through a vibration compensation algorithm to calculate and obtain the corrected value of the corn ear height; Dynamically adjust the height of the cutting table of the corn harvester based on the corrected value of the corn ear height using a dynamic fuzzy PID control algorithm. In some embodiments of the present invention, the original image includes a left-eye image and a right-eye image, and the method for preprocessing the original image includes the following steps: Obtain the internal parameter matrix of the binocular camera, and make the left-eye image and the right-eye image planes collected by the binocular camera coplanar and the epipolar lines parallel through geometric transformation based on the internal parameter matrix; Use the semi-global block matching algorithm to find corresponding pixel points in the left-eye image and the right-eye image to calculate the disparity value and obtain the disparity map: Calculate the depth value based on the disparity value, and then convert the disparity map to a depth map. During the conversion process, generate the depth map by assisting with structured light projection in the low-texture area of the image based on the active light source characteristics of the binocular camera.

[0007] In some embodiments of the present invention, the following steps are further included: Before pre-training, improve the structure of the YOLOV8n model. The specific improvement methods include: Add a depth feature preprocessing module to the input layer of the YOLOV8n model to perform normalization processing on the input original image and depth map; Add a dual-branch feature extraction module to the BackBone part of the YOLOV8n model to process the original image and the disparity map obtained based on the original image respectively; Add a double-layer routing attention mechanism to the BackBone part of the YOLOV8n model; Introduce an EfficientRepGFPN feature fusion network into the Neck part of the YOLOV8n model; Replace the original loss function of the YOLOV8n model with an Inner-IoU loss function.

[0008] In some embodiments of the present invention, the coordinates of the center point at the bottom of the bounding box are , and the corresponding coordinates of the center point at the bottom of the bounding box in the camera coordinate system are . Convert the coordinates of the center point at the bottom of the bounding box to the camera coordinate system . The calculation formula is: ; ; ; Among them, and are the focal lengths of the binocular camera, and are the principal point coordinates of the binocular camera, is the depth value of the depth map; the calculation formula for the depth value is: ; Among them, is the disparity value, is the focal length of the binocular camera, is the baseline distance of the binocular camera, is the camera pitch angle, and the camera pitch angle is obtained in real time by a sensor installed on the corn harvester; The calculation formula for the height of the corn ear root is: ; Among them, is the height of the corn ear root, is the ground height.

[0009] In some embodiments of the present invention, the expression of the plane equation is: ; The ground height The calculation formula for is: .

[0010] In some embodiments of the present invention, the sensor includes but is not limited to a triaxial acceleration sensor and a gyroscope. The method for fusing through a vibration compensation algorithm includes the following steps: Calculate the offset of the corn ear height caused by vibration using the vibration data collected by the sensor : ; Among them, a(t) is is the vertical acceleration measured by the triaxial acceleration sensor, ω(t) is the vertical angular velocity measured by the gyroscope, α is the tilt compensation coefficient; Fuse and calculate the offset of the corn ear height and the initial value of the corn ear height to obtain the corrected value of the corn ear height : ; Among them, is the weight function, The expression is: ; where is the variance of the offset , is the variance of binocular vision measurement, is the attenuation coefficient.

[0011] In some embodiments of the present invention, the method for dynamically adjusting the cutting table height of a corn harvester based on the corrected value of the corn ear height using a dynamic fuzzy PID control algorithm specifically includes the following steps: Using a dynamic fuzzy PID control algorithm, by real-time controlling the switch of the hydraulic proportional valve in the hydraulic actuator of the corn harvester to control the stroke of the hydraulic cylinder, thereby changing the inclination angle of the cutting table of the corn harvester to adjust the cutting height of the cutting table.

[0012] In some embodiments of the present invention, the method for dynamically adjusting the cutting table height of a corn harvester based on the corrected value of the corn ear height using a dynamic fuzzy PID control algorithm specifically includes the following steps: By installing an angle sensor on the cutting table to collect the angle change data of the cutting table in real time, and calculating the pitch angle of the cutting table as the cutting table height through a predetermined angle-height conversion algorithm. When the calculated cutting table height reaches a predetermined threshold, the cutting table height is no longer adjusted to achieve closed-loop control of the cutting table height.

[0013] In some embodiments of the present invention, the method for dynamically adjusting the cutting table height of a corn harvester using a dynamic fuzzy PID control algorithm further includes: Based on the vibration-load coupling model and multi-sensor embedded Kalman filter, compensating for the non-linear characteristics of the hydraulic actuator of the corn harvester.

[0014] Some embodiments of the present invention further provide a multi-modal fusion adaptive adjustment system for the cutting table height of a corn harvester, which is used to implement the above-mentioned multi-modal fusion adaptive adjustment method for the cutting table height of a corn harvester, including a vision detection module, a multi-sensor fusion module, a control terminal, and a hydraulic actuator: The vision detection module includes a binocular camera, which is used to collect the original images of the unharvested crops in front of the corn harvester in real time, and preprocess the collected original images to obtain a depth map; The multi-sensor fusion module includes multiple sensors, which are used to collect the vibration data during the working process of the corn harvester The control terminal is respectively communicatively connected to the vision detection module, the multi-sensor fusion module, and the hydraulic actuator; The control terminal is used to run the pre-trained YOLOV8n model to perform model operations on the images collected and preprocessed by the vision detection module to obtain the bounding boxes of the corn ears in the images, and calculate the coordinates of the center point at the bottom of the bounding boxes; perform plane fitting based on the depth map and the ground plane equation to obtain the ground height; calculate the initial value of the corn ear height based on the ground height, the coordinates of the center point at the bottom of the bounding box, and the depth value of the depth map; and use the multi-sensor fusion module to collect vibration data to perform vibration compensation on the initial value of the corn ear height, and calculate and obtain the corrected value of the corn ear height; and then output a control command for the hydraulic actuator based on the corrected value of the corn ear height. The hydraulic actuator includes a hydraulic proportional valve. The hydraulic actuator is used to execute the control command sent by the control terminal, adjust the stroke of the hydraulic cylinder by controlling the opening and closing of the hydraulic proportional valve, and then change the inclination angle of the cutting table of the corn harvester to adjust the harvesting height of the cutting table.

[0015] The beneficial effects of the present invention are as follows: 1. By combining the YOLOv8n target detection model with the dynamic fuzzy PID algorithm, the present invention solves the problems of low visual detection accuracy and lag in control response of existing agricultural machinery in complex field scenarios, and effectively reduces crop losses during harvesting operations. 2. By improving the structure of the YOLOv8n model, the present invention realizes the multi-modal input of RGB images and depth maps. Through the synergistic effect of the multi-modal input and the adaptive attention mechanism of the YOLOv8n model, the accuracy and accuracy of model target detection are improved, and the dynamic feature fusion of RGB images and depth information is realized, effectively overcoming the interference of complex field scenarios such as light changes and leaf occlusion on the positioning of corn ears. 3. The present invention fuses visual deep learning data with vibration data generated during the operation of the harvester, improves the accuracy of corn ear height calculation, and is beneficial to further reducing crop losses during harvesting operations. 4. The present invention adopts the dynamic fuzzy PID control algorithm to output the control strategy for the corn harvester, dynamically adjusts the PID parameters through the fuzzy rule base, effectively improves the anti-bumping ability of the corn harvester, and ensures the continuity and stability of corn harvesting.

[0016] Other features and advantages of the present invention will be described in the following specification, and, in part, will be obvious from the specification, or will be understood by implementing the present invention. The objectives and other advantages of the present invention can be realized and obtained by the structures pointed out in the specification, the claims, and the drawings. Description of the Drawings

[0017] To more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will describe the specific embodiments of the present invention in detail with reference to the accompanying drawings. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0018] Figure 1 It is a flowchart of an adaptive adjustment method for the cutting table height of a multi-modal fusion corn harvester; Figure 2 It is an architecture diagram of an adaptive adjustment system for the cutting table height of a multi-modal fusion corn harvester; Figure 3 It is a working flowchart of an adaptive adjustment system for the cutting table height of a multi-modal fusion corn harvester; Figure 4 It is a schematic diagram of the module installation positions in a corn harvester. Detailed implementation manners

[0019] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the following will describe and explain the present application in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0020] It should be noted that the terms used herein are only for describing the specific implementation manners and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless otherwise clearly specified in the context, the singular form is also intended to include the plural form. In addition, it should also be understood that the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device that includes a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products or devices.

[0021] Without conflict, the embodiments in the present invention and the features in the embodiments can be combined with each other.

[0022] The following will describe the technical solutions of the present invention in detail in combination with specific embodiments and the accompanying drawings of the specification.

[0023] As shown in the attached Figure 1 figures, in an exemplary embodiment of an adaptive adjustment method for the cutting table height of a multi-modal fusion corn harvester according to the present invention, the adjustment method includes the following steps.

[0024] S1: When the corn harvester is performing the harvesting work, a binocular camera is used to collect the original images of the unharvested crops in front of the corn harvester in real time, and the original images are preprocessed to obtain a depth map. Among them, the binocular camera is specifically a structured light binocular IR camera, and the original images it collects include binocular RGB images.

[0025] In some embodiments of the present invention, the original images include a left-eye image and a right-eye image, and the method for preprocessing the original images includes the following steps: Use the Zhang-Zhengyou calibration method to obtain the internal parameter matrix of the binocular camera: ; Among them, and are the focal lengths of the binocular camera, and are the principal point coordinates of the binocular camera.

[0026] Based on the internal parameter matrix, use the Bouguet algorithm to make the left-eye image and the right-eye image planes collected by the binocular camera coplanar and the epipolar lines parallel through geometric transformation, so as to achieve stereo rectification.

[0027] Use the semi-global block matching algorithm (SGBM algorithm) to find corresponding pixel points in the left-eye image and the right-eye image, calculate the disparity value, and obtain a disparity map.

[0028] Calculate the depth value based on the disparity value, and then convert the disparity map into a depth map. During the conversion process, based on the active light source characteristics of the structured light binocular IR camera, a depth map is generated by auxiliary structure light projection in the low-texture area of the image.

[0029] In this process, the calculation formula for the depth value is: .

[0030] Among them, is the disparity value, is the focal length of the binocular camera, is the baseline distance of the binocular camera, is the camera pitch angle, and the camera pitch angle is obtained in real time by a sensor installed on the corn harvester.

[0031] S2: Input the left-eye RGB image in the original image and the depth map obtained by preprocessing into the pre-trained YOLOV8n model, and output the bounding box and confidence C (C≥0.8) of the corn ears in the image through model operation. The output results of the model include the vertex coordinates of the bounding box, which are respectively x min , y min , x max , ymax , calculate the coordinates of the center point at the bottom of the bounding box based on the vertex coordinates of the bounding box , take the center point at the bottom of the bounding box as the projection position of the root of the corn ear, and the coordinates of the center point at the bottom of the bounding box The calculation formula is: .

[0032] In some embodiments of the present invention, before pre-training, the structure of the YOLOV8n model is improved, and the improvement method specifically includes the following steps.

[0033] Add a depth feature preprocessing module to the input layer of the YOLOV8n model to normalize the input left-eye RGB image and depth map.

[0034] Add a dual-branch feature extraction module to the BackBone part of the YOLOV8n model to process the left-eye RGB image and disparity map respectively, and adaptively fuse the feature weights through the channel attention mechanism (CBAM) to solve the misdetection problem caused by occlusion.

[0035] Add a bi-level routing attention mechanism to the BackBone part of the YOLOV8n model to enhance the small target detection ability of the model.

[0036] Introduce the EfficientRepGFPN feature fusion network into the Neck part of the YOLOV8n model to optimize the multi-scale feature fusion efficiency.

[0037] Replace the original loss function of the YOLOV8n model with the Inner-IoU loss function, and introduce the calculation of the auxiliary bounding box (Inner Box) to solve the missed detection problem when the corn ears are densely arranged.

[0038] S3: Dynamic fitting of ground height. Based on the depth map and the ground plane equation Perform plane fitting to obtain the ground height.

[0039] In some embodiments of the present invention, the specific implementation method of step S3 includes: Perform height screening on the preprocessed depth map, and retain the point cloud data within 0.1m - 0.5m.

[0040] Use the RANSAC algorithm to fit the ground plane equation , calculate and obtain the ground height , where the ground height The calculation formula is: .

[0041] Update the ground model in the form of a real-time sliding window according to the moving direction of the cutting table.

[0042] S4: Calculate the height of the corn ear. Based on the ground height and the depth value of the depth map, perform three-dimensional coordinate transformation on the pixel coordinates of the center point at the bottom of the bounding box, convert the pixel coordinates of the center point at the bottom of the bounding box to the camera coordinate system, and calculate the height of the corn ear root based on its coordinates in the camera coordinate system. Furthermore, calculate the initial value of the corn ear height based on the height of the corn ear root.

[0043] In some embodiments of the present invention, the coordinates of the center point at the bottom of the bounding box are , and the corresponding coordinates of the center point at the bottom of the bounding box in the camera coordinate system are . Convert the coordinates of the center point at the bottom of the bounding box to the camera coordinate system . The calculation formula is: ; ; ; Among them, and are the focal lengths of the binocular camera, and are the principal point coordinates of the binocular camera, is the depth value of the depth map.

[0044] In some embodiments of the present invention, the calculation formula for the height of the corn ear root is: .

[0045] S5: Collect the vibration data during the operation of the corn harvester through a variety of sensors installed on the corn harvester.

[0046] Fuse the vibration data with the initial value of the corn ear height through a vibration compensation algorithm to calculate and obtain the corrected value of the corn ear height.

[0047] In some embodiments of the present invention, the sensors installed on the corn harvester include but are not limited to triaxial acceleration sensors and gyroscopes. In step S5, the method of fusion through the vibration compensation algorithm includes the following steps.

[0048] S51: Calculate the offset of the corn ear height caused by vibration using the vibration data collected by the sensor :

[0049] Among them, a(t) isThe vertical acceleration measured by the triaxial acceleration sensor, with the unit of m / s². It should be noted that in this embodiment, a(t) the gravity component g = 9.8 m / s² needs to be deducted; ω(t) is the vertical angular velocity measured by the gyroscope, with the unit of rad / s, which is used to compensate for the measurement error caused by tilt; α is the tilt compensation coefficient, which is determined through a calibration experiment. In this embodiment, the value range of α is 0.1 - 0.3.

[0050] S52: Perform a fusion calculation on the offset and the initial value of the corn ear height to obtain the corrected value of the corn ear height : .

[0051] Among them, is the weight function, and the expression of is:

[0052] Among them, is the variance of the offset , which is used to reflect the reliability of the vibration data, is the variance of the binocular vision measurement, which is obtained through offline calibration, is the attenuation coefficient, which is used to control the sensitivity of vibration compensation. In this embodiment, the value range of the attenuation coefficient is 0.5 - 1.0.

[0053] In some embodiments of the present invention, the method of fusion through the vibration compensation algorithm further includes the following steps.

[0054] S53: When there is a conflict between vibration and visual data, use the Kalman filter to dynamically adjust the fusion weight , .

[0055] Among them, both A and B are state coefficients. Among them, A = 1, indicating that the height change is a random walk, and B ==, indicating no external control input.

[0056] is the process noise covariance, which is determined by the vibration energy: ; Among them, .

[0057] is the observation noise covariance: ; Among them, 。

[0058] S53: Output the adjusted correction value and use the adjusted correction value as the correction value for the corn ear height to perform subsequent calculations. The calculation formula for the adjusted correction value is as follows: 。

[0059] S6: Based on the correction value of the corn ear height, adopt the dynamic fuzzy PID control algorithm to dynamically adjust the cutting table height of the corn harvester.

[0060] In some embodiments of the present invention, the method for dynamically adjusting the cutting table height of the corn harvester based on the correction value of the corn ear height by using the dynamic fuzzy PID algorithm specifically includes the following steps.

[0061] Adopt the dynamic fuzzy PID control algorithm, and by controlling the switch of the hydraulic proportional valve in the hydraulic actuator of the corn harvester in real time, further control the stroke of the hydraulic cylinder, and then change the inclination angle of the cutting table of the corn harvester to realize the adjustment of the cutting height of the cutting table.

[0062] In some embodiments of the present invention, the method for dynamically adjusting the cutting table height of the corn harvester based on the correction value of the corn ear height by using the dynamic fuzzy PID control algorithm further includes the following steps.

[0063] Real-time collect the angle change data of the cutting table through the angle sensor installed on the cutting table, calculate the pitch angle of the cutting table as the cutting table height through the predetermined angle and height conversion algorithm, and when the calculated cutting table height reaches the predetermined threshold, the cutting table height is no longer adjusted to realize the closed-loop control of the cutting table height.

[0064] In some embodiments of the present invention, the method for dynamically adjusting the cutting table height of the corn harvester by using the dynamic fuzzy PID control algorithm further includes the following steps.

[0065] Based on the vibration-load coupling model and multi-sensor embedded Kalman filter, compensate for the nonlinear characteristics of the hydraulic actuator of the corn harvester.

[0066] Some embodiments of the present invention further provide a multi-modal fusion adaptive adjustment system for the cutting table height of a corn harvester, as shown in Attached Figure 2 -Attached Figure 4 shown, for implementing the above-mentioned multi-modal fusion adaptive adjustment method for the cutting table height of the corn harvester. The system includes a vision detection module, a multi-sensor fusion module, a control terminal, and a hydraulic actuator.

[0067] Among them, as shown in AttachedFigure 2 As shown in Figure 2 , the visual detection module includes a binocular camera and an improved YOLOV8n model. The binocular camera is used to collect the original images of the unharvested crops in front of the corn harvester in real time, and preprocess the collected original images to obtain depth maps. After collecting and preprocessing the image data, the visual detection module transmits it to the control terminal in real time.

[0068] The multi-sensor fusion module is an IMU module, which includes multiple sensors and is used to collect vibration data during the operation of the corn harvester for IMU vibration compensation. After collecting the vibration data, the multi-sensor fusion module transmits it to the control terminal in real time.

[0069] The control terminal is specifically an embedded control platform, which is communicatively connected to the visual detection module, the multi-sensor fusion module, and the hydraulic actuator respectively.

[0070] The control terminal is used to run the pre-trained YOLOV8n model accelerated by TensorTR to perform object detection on the images collected and preprocessed by the visual detection module, obtain the bounding boxes of the corn ears in the images, and calculate the coordinates of the center points at the bottoms of the bounding boxes; and perform plane fitting based on the depth map and the ground plane equation to obtain the ground height; furthermore, calculate the initial value of the corn ear height based on the ground height, the coordinates of the center points at the bottoms of the bounding boxes, and the depth values of the depth map; and use the vibration data collected by the multi-sensor fusion module to perform vibration compensation on the initial value of the corn ear height to calculate the corrected value of the corn ear height; furthermore, output control instructions for the hydraulic actuator based on the corrected value of the corn ear height.

[0071] The hydraulic actuator includes a hydraulic proportional valve. The hydraulic actuator is used to execute the control instructions sent by the control terminal, adjust the stroke of the hydraulic cylinder by controlling the opening and closing of the hydraulic proportional valve, and then change the inclination angle of the cutting table of the corn harvester to adjust the harvesting height of the cutting table.

[0072] In some embodiments of the present invention, the control terminal adopts a heterogeneous computing architecture combining a Jetson Nano controller and an STM32 controller.

[0073] The Jetson Nano controller is used to run the pre-trained YOLOV8n model to perform model operations on the images collected and preprocessed by the visual detection module and complete the calculation of the initial value of the corn ear height. At the same time, the Jetson Nano controller is also used to perform vibration compensation on the initial value of the corn ear height based on the vibration data collected by the multi-sensor fusion module to obtain the corrected value of the corn ear height.

[0074] The STM32 controller is communicatively connected to the Jetson Nano controller, and based on the corrected value of the corn ear height calculated by the Jetson Nano controller, a dynamic fuzzy PID control algorithm is executed to output a control command for the hydraulic actuator.

[0075] In some embodiments of the present invention, the hydraulic actuator further includes an angle sensor for real-time acquisition of the pitch angle of the cutter bar, so as to calculate the harvesting height of the cutter bar, and further perform feedback adjustment on the harvesting height of the cutter bar.

[0076] As shown in the appendix Figure 4 In the multi-modal fusion corn harvester cutter bar height adaptive adjustment system provided by the present invention, the installation positions of each module in the corn harvester are as follows: A binocular camera and a fill light for night lighting and fill light are installed at the front end of the top of the cutter bar; a blower is installed on the side of the binocular camera for cleaning the camera lens through the air flow; an angle sensor is installed at the hinge point of the cutter bar lifting arm; a multi-sensor fusion module, that is, an IMU module, is installed in the center of the cutter bar main body frame; an STM32 controller is installed in the middle of the left side of the cutter bar, and a hydraulic cylinder is installed in the hydraulic system area on the right side of the cutter bar.

[0077] The transmission process of the control data in the system is as follows: The real-time image / depth data collected by the binocular camera is transmitted to the Jetson Nano controller in the control terminal. The target height command obtained by the operation of the Jetson Nano controller is transmitted to the STM32 controller. The STM32 controller simultaneously receives the height data of the Jetson Nano controller, the lifting angle data of the angle sensor, and the attitude vibration data of the multi-sensor fusion module, and generates a PWM control signal through the algorithm operation in the STM32 controller and transmits it to the proportional valve to control the hydraulic cylinder.

[0078] As shown in the appendix Figure 3 In the multi-modal fusion corn harvester cutter bar height adaptive adjustment system provided by the present invention, the working process is as follows: When the corn harvester is performing the harvesting work, the binocular camera installed on the cutter bar of the harvester is used to collect real-time photos of the unharvested field corn in front of the harvester, and the photo data collected by the binocular camera is transmitted to the Jetson Nano controller in the control terminal in real time. The pre-trained yolov8n model accelerated by TensorTR is run on the Jetson Nano controller to perform target detection on the corn ear, and the initial value of the corn ear height with a horizontal distance of 5 cm - 10 cm from the front edge of the cutter bar is calculated by using the installation height of the binocular camera and the baseline length of the binocular camera.

[0079] After the initial value of the corn ear height is calculated, the Jetson Nano controller transmits the initial value of the corn ear height to the STM32 controller via the CAN bus. In the STM32 controller, a dynamic fuzzy PID control algorithm is executed to control the hydraulic proportional valve in real time, causing the stroke of the hydraulic cylinder to change, thereby changing the inclination angle of the cutting table. During this process, the angle sensor installed on the fixed shaft of the cutting table also changes its angle in real time and collects the angle change data in real time. Through the angle-height conversion algorithm preset in the STM32, the pitch angle of the cutting table is converted into the harvesting height of the cutting table, and the calculated height value is fed back to the STM32 controller in real time for feedback adjustment, so that when the harvesting height of the cutting table reaches the predetermined threshold, the harvesting height of the cutting table will no longer be adjusted, realizing closed-loop control.

[0080] Finally, it should be noted that the various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. For the same or similar parts among the various embodiments, reference may be made to each other.

[0081] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit them; although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that: it is still possible to modify the specific implementation manners of the present invention or perform equivalent replacements for some technical features; without departing from the spirit of the technical solutions of the present invention, they should all be covered within the scope of the technical solutions claimed by the present invention.

Claims

1. A method for adaptively adjusting the height of the cutting table of a multi-modal fusion corn harvester, characterized in that, Including the following steps: Using a binocular camera to collect the original images of the unharvested crops in front of the corn harvester in real time, and preprocessing the original images to obtain depth maps; Inputting the original images and the depth maps into a pre-trained YOLOV8n model, and through model operations, outputting the bounding boxes of the corn ears in the images and calculating the coordinates of the center points at the bottoms of the bounding boxes; Performing plane fitting based on the depth maps and plane equations to obtain the ground height; Converting the coordinates of the center points at the bottoms of the bounding boxes to the camera coordinate system based on the ground height and the depth values of the depth maps, calculating the root height of the corn ears based on their coordinates in the camera coordinate system, and further calculating the initial value of the corn ear height based on the root height of the corn ears; Collecting vibration data during the operation of the corn harvester through sensors; fusing the vibration data and the initial value of the corn ear height through a vibration compensation algorithm to calculate and obtain the corrected value of the corn ear height; Based on the corrected value of the corn ear height, using a dynamic fuzzy PID control algorithm to dynamically adjust the cutting table height of the corn harvester.

2. The method for adaptively adjusting the height of the header of the multi-modal fusion corn harvester according to claim 1, characterized in that, The original images include left-eye images and right-eye images, and the method for preprocessing the original images includes the following steps: Obtaining the internal parameter matrix of the binocular camera, and based on the internal parameter matrix, making the left-eye image and the right-eye image planes collected by the binocular camera coplanar and the epipolar lines parallel through geometric transformation; Using the semi-global block matching algorithm to find corresponding pixel points in the left-eye image and the right-eye image, calculating the disparity values, and obtaining a disparity map; Calculating the depth values based on the disparity values, and further converting the disparity map into a depth map. During the conversion process, based on the active light source characteristics of the binocular camera, auxiliary generation of the depth map is carried out by structure light projection in the low-texture area of the image.

3. The method for adaptively adjusting the height of the header of a multi-modal fusion corn harvester according to claim 1, characterized in that, Before pre-training, improving the structure of the YOLOV8n model. The improvement method specifically includes the following steps: Adding a depth feature preprocessing module to the input layer of the YOLOV8n model to perform normalization processing on the input original images and depth maps; Adding a double-branch feature extraction module to the BackBone part of the YOLOV8n model to respectively process the original images and the disparity maps obtained based on the original images; Adding a double-layer routing attention mechanism to the BackBone part of the YOLOV8n model; Introducing an EfficientRepGFPN feature fusion network into the Neck part of the YOLOV8n model; Replacing the original loss function of the YOLOV8n model with an Inner-IoU loss function.

4. The method for adaptively adjusting the height of the header of a multi-modal fusion corn harvester according to any one of claims 1-3, characterized in that, The coordinates of the center point at the bottom of the bounding box are , and the corresponding coordinates of the center point at the bottom of the bounding box in the camera coordinate system are . Convert the coordinates of the center point at the bottom of the bounding box to the camera coordinate system . The calculation formula is: ； ； ； Among them, and are the focal lengths of the binocular camera, and are the principal point coordinates of the binocular camera, is the depth value of the depth map; The calculation formula for the root height of the corn ears is: ； Among them, is the height of the root of the corn ear, is the ground height.

5. The method for adaptively adjusting the cutting table height of the multi-modal fusion corn harvester according to any one of claims 1-3, characterized in that, The expression of the plane equation is: ； The ground height The calculation formula is as follows: 。 6. The method for adaptively adjusting the height of the corn harvester header with multimodal fusion according to any one of claims 1-3, characterized in that, The sensors installed on the corn harvester include but are not limited to triaxial acceleration sensors and gyroscopes. The method for fusion through a vibration compensation algorithm includes the following steps: Calculating the offset of the corn ear height caused by vibration using the vibration data collected by the sensor : ； Among them, For the vertical acceleration measured by the three-axis acceleration sensor, is the vertical angular velocity measured by the gyroscope, is the tilt compensation coefficient; Perform fusion calculation on the height offset of the corn ear and the initial value of the corn ear height to obtain the corrected value of the corn ear height : ； Among them, is a weight function, The expression of is as follows: ； Among them, is the variance of the offset , is the variance measured by the binocular camera, is the attenuation coefficient.

7. The method for adaptively adjusting the height of the header of the multi-modal fusion corn harvester according to claim 1, characterized in that, The method for dynamically adjusting the cutting table height of the corn harvester based on the corrected value of the corn ear height using a dynamic fuzzy PID control algorithm specifically includes the following steps: Adopt a dynamic fuzzy PID control algorithm to control the stroke of the hydraulic cylinder by controlling the switch of the hydraulic proportional valve in the hydraulic actuator of the corn harvester in real time, so as to change the inclination angle of the cutting table of the corn harvester and realize the adjustment of the harvesting height of the cutting table.

8. The method for adaptively adjusting the height of the corn harvester header with multi-modal fusion according to claim 7, characterized in that, The method for dynamically adjusting the cutting table height of a corn harvester based on the corn ear height correction value using a dynamic fuzzy PID control algorithm specifically includes the following steps: Real-time collect the angle change data of the cutting table through the angle sensor installed on the cutting table, calculate the pitch angle of the cutting table as the cutting table height through a predetermined angle-height conversion algorithm, and when the calculated cutting table height reaches a predetermined threshold, the cutting table height is no longer adjusted to achieve closed-loop control of the cutting table height.

9. The method for adaptively adjusting the height of the corn harvester header with multimodal fusion according to claim 7 or 8, characterized in that, The method for dynamically adjusting the cutting table height of a corn harvester using a dynamic fuzzy PID control algorithm further includes: Based on the vibration-load coupling model and multi-sensor embedded Kalman filter, compensate for the non-linear characteristics of the hydraulic actuator of the corn harvester.

10. A multi-modal fusion corn harvester header height adaptive adjustment system for implementing the multi-modal fusion corn harvester header height adaptive adjustment method according to any one of claims 1-9, characterized in that, It includes a vision detection module, a multi-sensor fusion module, a control terminal and a hydraulic actuator: The vision detection module includes a binocular camera, which is used to collect the original images of the unharvested crops in front of the corn harvester in real time and preprocess the collected original images to obtain a depth map; The multi-sensor fusion module includes multiple sensors, which are used to collect vibration data during the operation of the corn harvester The control terminal is respectively communicatively connected to the vision detection module, the multi-sensor fusion module and the hydraulic actuator; The control terminal is used to run the pre-trained YOLOV8n model to perform model operations on the images collected and preprocessed by the vision detection module to obtain the bounding box of the corn ear in the image, and calculate the coordinates of the center point at the bottom of the bounding box; perform plane fitting based on the depth map and the plane equation to obtain the ground height; furthermore, calculate the initial value of the corn ear height based on the ground height, the coordinates of the center point at the bottom of the bounding box and the depth value of the depth map; and use the vibration data collected by the multi-sensor fusion module to perform vibration compensation on the initial value of the corn ear height to calculate and obtain the corn ear height correction value; furthermore, output a control command for the hydraulic actuator based on the corn ear height correction value; The hydraulic actuator includes a hydraulic proportional valve, and the hydraulic actuator is used to execute the control command sent by the control terminal, adjust the stroke of the hydraulic cylinder by controlling the switch of the hydraulic proportional valve, and then change the inclination angle of the cutting table of the corn harvester to achieve the adjustment of the harvesting height of the cutting table.

Images