A multi-modal fusion corn harvester header height adaptive adjustment method and system

Abstract

The present invention relates to a multimodal fusion method and system for adaptively adjusting the height of a corn harvester's header, belonging to the field of automated control technology. The method comprises the following steps: using a binocular camera to capture crop images and preprocessing them to obtain a depth map; inputting the images into a YOLO V8n model, and obtaining the coordinates of the bottom center point of the corn ear's bounding box through model calculations; performing plane fitting on the depth map to obtain the ground height; and calculating an initial value for the corn ear's height based on the ground height and the center point coordinates; collecting vibration data from the harvester's operation, performing vibration compensation on the initial value to obtain a height correction value; and dynamically adjusting the harvester's header height using a dynamic fuzzy PID algorithm based on the correction value. The present invention combines a target detection model with a dynamic fuzzy PID algorithm to address the problems of low visual detection accuracy and delayed control response in existing agricultural machinery in complex field scenarios, effectively reducing crop losses during harvesting operations.

Description

Technical Field

[0001] The present invention belongs to the technical field of agricultural machinery automation control, and in particular relates to a multi-modal fusion corn harvester header height adaptive adjustment method and system. Background Art

[0002] The genetic material of seed corn kernels is extremely sensitive and has a very low tolerance for mechanical damage. The damage rate must be strictly controlled within 2% to ensure the seed germination rate and the reliability of subsequent planting.

[0003] Traditional corn harvesters control the harvesting height of the header through preset heights or manual intervention, which results in a certain degree of lag and inaccuracy. These devices are unable to adapt to the complex and varied terrain of the field and the differences in plant heights in real time, which can easily lead to corn ears being hit or missed, causing the damage rate to climb to 4%-6%, seriously exceeding the harvest standard for seed corn, severely affecting seed quality, and causing huge losses to the seed industry.

[0004] To solve this problem, existing technologies usually use a monocular vision system combined with a fixed-parameter PID algorithm to control the harvesting height of the header. Although this method reduces the harvesting damage rate to a certain extent, the monocular vision system is easily affected by dense planting and canopy reflection interference, resulting in large ear positioning errors and a sharp increase in false detection rates in rainy weather, which cannot meet the seed-level harvesting accuracy requirements. In addition, the fixed-parameter PID algorithm has a large response delay at complex operating speeds, and the vibration of the header can easily cause secondary damage. It is still difficult to meet the damage rate requirements for harvesting high-seed corn. Summary of the Invention

[0005] The purpose of the present invention is to solve one of the above technical problems and provide a multi-modal fusion corn harvester header height adaptive adjustment method and system.

[0006] To achieve the above object, the technical solution adopted by the present invention is:

[0007] A multi-modal fusion method for adaptively adjusting the height of a corn harvester header comprises the following steps:

[0008] A binocular camera is used to capture raw images of unharvested crops in front of a corn harvester in real time, and the raw images are preprocessed to obtain a depth map.

[0009] Input the original image and depth map into the pre-trained YOLOV8n model, and use the model to calculate the bounding box of the ear of corn in the output image and the coordinates of the bottom center point of the bounding box;

[0010] The ground height is obtained by plane fitting based on the depth map and plane equation;

[0011] Based on the ground height and the depth value of the depth map, the coordinates of the bottom center point of the bounding box are converted to the camera coordinate system, and the height of the root of the corn ear is calculated based on its coordinates in the camera coordinate system. Then, the initial value of the height of the corn ear is calculated based on the height of the root of the corn ear;

[0012] The vibration data of the corn harvester during operation is collected through sensors; the vibration data is integrated with the initial value of the corn ear height through a vibration compensation algorithm to calculate the corrected value of the corn ear height;

[0013] Based on the corn ear height correction value, the dynamic fuzzy PID control algorithm is used to dynamically adjust the cutting platform height of the corn harvester.

[0014] In some embodiments of the present invention, the original image includes a left image and a right image, and the method for preprocessing the original image includes the following steps:

[0015] Obtain the intrinsic parameter matrix of the binocular camera, and perform geometric transformation based on the intrinsic parameter matrix so that the left and right image planes captured by the binocular camera are coplanar and their epipolar lines are parallel;

[0016] Use the semi-global block matching algorithm to find the corresponding pixel points in the left and right images to calculate the disparity value and obtain the disparity map:

[0017] The depth value is calculated based on the disparity value, and then the disparity map is converted into a depth map. During the conversion process, the active light source characteristics of the binocular camera are used to assist in generating a depth map by projecting structured light in the low-texture area of the image.

[0018] In some embodiments of the present invention, the following steps are further included:

[0019] Before pre-training, the structure of the YOLOV8n model is improved. The improvement methods include:

[0020] Add a depth feature preprocessing module to the input layer of the YOLOV8n model to normalize the input original image and depth map;

[0021] Add a dual-branch feature extraction module to the BackBone part of the YOLOV8n model to process the original image and the disparity map obtained based on the original image respectively;

[0022] Add a two-layer routing attention mechanism to the BackBone part of the YOLOV8n model;

[0023] Introduce the EfficientRepGFPN feature fusion network into the Neck part of the YOLOV8n model;

[0024] Replace the original loss function of the YOLOV8n model with the Inner-IoU loss function.

[0025] In some embodiments of the present invention, the coordinates of the bottom center point of the bounding box are , the coordinates of the bottom center point of the bounding box in the camera coordinate system are , the coordinates of the bottom center point of the bounding box Convert to camera coordinate system The calculation formula is:

[0026] ;

[0027] ;

[0028] ;

[0029] in, and is the focal length of the binocular camera, and is the principal point coordinate of the binocular camera, is the depth value of the depth map; the calculation formula of the depth value is:

[0030] ;

[0031] in, is the disparity value, is the focal length of the binocular camera, is the baseline distance of the binocular camera, is the camera pitch angle, which is obtained in real time by a sensor installed on the corn harvester;

[0032] The formula for calculating the root height of the corn ear is:

[0033] ;

[0034] in, is the height of the corn ear root, is the ground height.

[0035] In some embodiments of the present invention, the plane equation is expressed as:

[0036] ;

[0037] Ground height The calculation formula is:

[0038] .

[0039] In some embodiments of the present invention, the sensors include but are not limited to a three-axis acceleration sensor and a gyroscope, and the method for fusing the sensors using a vibration compensation algorithm includes the following steps:

[0040] Calculate the height deviation of the ear of corn caused by vibration using vibration data collected by the sensor :

[0041] ;

[0042] in, a(t) is The vertical acceleration measured by the triaxial accelerometer, ω(t) is the vertical angular velocity measured by the gyroscope, α is the tilt compensation coefficient;

[0043] Height offset for ears of corn and the initial value of corn ear height Perform fusion calculation to obtain the corrected value of corn ear height :

[0044] ;

[0045] in, is the weight function, The expression is:

[0046] ;

[0047] in, is the offset The variance of is the variance of binocular vision measurement, is the attenuation coefficient.

[0048] In some embodiments of the present invention, a method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm based on a corn ear height correction value specifically includes the following steps:

[0049] The dynamic fuzzy PID control algorithm is adopted to control the stroke of the hydraulic cylinder by real-time controlling the switch of the hydraulic proportional valve in the hydraulic actuator of the corn harvester, thereby changing the inclination angle of the corn harvester header to achieve the adjustment of the harvesting height of the header.

[0050] In some embodiments of the present invention, a method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm based on a corn ear height correction value specifically includes the following steps:

[0051] The angle sensor installed on the cutting platform collects the angle change data of the cutting platform in real time, and the pitch angle of the cutting platform is calculated as the cutting platform height through a predetermined angle and height conversion algorithm. When the calculated cutting platform height reaches the predetermined threshold, the cutting platform height is no longer adjusted to achieve closed-loop control of the cutting platform height.

[0052] In some embodiments of the present invention, the method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm further includes:

[0053] Based on the vibration-load coupling model and multi-sensor embedded Kalman filtering, the nonlinear characteristics of the hydraulic actuator of the corn harvester are compensated.

[0054] Some embodiments of the present invention further provide a multimodal fusion corn harvester header height adaptive adjustment system, which is used to implement the above-mentioned multimodal fusion corn harvester header height adaptive adjustment method, including a visual detection module, a multi-sensor fusion module, a control terminal and a hydraulic actuator:

[0055] The visual inspection module includes a binocular camera, which is used to collect raw images of unharvested crops in front of the corn harvester in real time and pre-process the collected raw images to obtain a depth map;

[0056] The multi-sensor fusion module includes multiple sensors to collect vibration data during the operation of the corn harvester.

[0057] The control terminal is respectively connected to the visual detection module, the multi-sensor fusion module and the hydraulic actuator in communication;

[0058] The control terminal is used to run the pre-trained YOLOV8n model to perform model operations on the images collected and pre-processed by the visual detection module to obtain the bounding box of the corn ear in the image and calculate the coordinates of the bottom center point of the bounding box; perform plane fitting based on the depth map and the ground plane equation to obtain the ground height; calculate the initial value of the corn ear height based on the ground height, the coordinates of the bottom center point of the bounding box, and the depth value of the depth map; and use the multi-sensor fusion module to collect vibration data, perform vibration compensation on the initial value of the corn ear height, and calculate the corrected value of the corn ear height; and then output a control instruction to the hydraulic actuator based on the corrected value of the corn ear height.

[0059] The hydraulic actuator includes a hydraulic proportional valve, which is used to execute the control instructions sent by the control terminal. By controlling the switch of the hydraulic proportional valve, the stroke of the hydraulic cylinder is adjusted, thereby changing the inclination angle of the corn harvester's header to adjust the harvesting height of the header.

[0060] The beneficial effects of the present invention are:

[0061] 1. By combining the YOLOv8n target detection model with the dynamic fuzzy PID algorithm, this paper solves the problems of low visual detection accuracy and delayed control response of existing agricultural machinery in complex field scenes, effectively reducing crop losses during harvesting operations;

[0062] 2. This paper improves the structure of the YOLOv8n model to achieve multimodal input of RGB images and depth maps. Through the synergy of multimodal input and the adaptive attention mechanism of the YOLOv8n model, the precision and accuracy of the model's target detection are improved, and the dynamic feature fusion of RGB images and depth information is achieved, effectively overcoming the interference of complex field scenes such as lighting changes and leaf occlusion on corn ear positioning.

[0063] 3. This invention integrates visual deep learning data with vibration data generated during harvester operation, improving the accuracy of corn ear height calculation and helping to further reduce crop losses during harvesting operations.

[0064] 4. The present invention adopts dynamic fuzzy PID control algorithm to output the control strategy for the corn harvester, and dynamically adjusts the PID parameters through the fuzzy rule library, which effectively improves the corn harvester's anti-bumping ability and ensures the continuity and stability of corn harvesting.

[0065] Other features and advantages of the present invention will be described in the following description, and in part will become apparent from the description, or will be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0066] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0067] Figure 1 This is a flowchart of a multi-modal fusion method for adaptive adjustment of the height of a corn harvester header.

[0068] Figure 2 This is an architecture diagram of a multi-modal fusion corn harvester header height adaptive adjustment system;

[0069] Figure 3 This is a workflow diagram of a multi-modal fusion corn harvester header height adaptive adjustment system;

[0070] Figure 4 This is a schematic diagram of the module installation position in the corn harvester. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is described and illustrated below in conjunction with the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely used to explain this application and are not intended to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making any creative efforts are within the scope of protection of this application.

[0072] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0073] In the absence of conflict, the embodiments of the present invention and the features thereof may be combined with each other.

[0074] The technical solution of the present invention is described in detail below in conjunction with specific embodiments and the accompanying drawings.

[0075] As attached Figure 1 As shown, in an illustrative embodiment of a multi-modal fusion corn harvester header height adaptive adjustment method of the present invention, the adjustment method includes the following steps.

[0076] S1: While a corn harvester is harvesting, it uses a binocular camera to capture raw images of the unharvested crops in front of the corn harvester in real time and pre-processes the raw images to obtain a depth map. Specifically, the binocular camera is a structured light binocular IR camera, and the raw images it captures include binocular RGB images.

[0077] In some embodiments of the present invention, the original image includes a left image and a right image, and the method for preprocessing the original image includes the following steps:

[0078] Use Zhang Zhengyou calibration method to obtain the intrinsic parameter matrix of the binocular camera:

[0079] ;

[0080] in, and is the focal length of the binocular camera, and is the principal point coordinate of the binocular camera.

[0081] Based on the intrinsic parameter matrix, the Bouguet algorithm is used to achieve stereo correction by geometric transformation so that the left and right image planes captured by the binocular camera are coplanar and their epipolar lines are parallel.

[0082] The semi-global block matching algorithm (SGBM algorithm) is used to find the corresponding pixel points in the left and right images to calculate the disparity values and obtain the disparity map.

[0083] The depth value is calculated based on the disparity value, and then the disparity map is converted into a depth map. During the conversion process, the active light source characteristics of the structured light binocular IR camera are used to assist in generating a depth map by projecting structured light in the low-texture area of the image.

[0084] In this process, the depth value is calculated as follows:

[0085] .

[0086] in, is the disparity value, is the focal length of the binocular camera, is the baseline distance of the binocular camera, is the camera pitch angle, which is obtained in real time by a sensor installed on the corn harvester.

[0087] S2: Input the left RGB image and the depth map obtained by preprocessing into the pre-trained YOLOV8n model. The model outputs the bounding box and confidence C (C ≥ 0.8) of the corn ear in the image. The output of the model includes the vertex coordinates of the bounding box, which are x min 、 y min 、 x max 、 y max , calculate the coordinates of the bottom center point of the bounding box based on the vertex coordinates of the bounding box , the bottom center point of the bounding box is used as the projection position of the corn ear root, and the coordinates of the bottom center point of the bounding box are The calculation formula is:

[0088] .

[0089] In some embodiments of the present invention, the structure of the YOLOV8n model is improved before pre-training, and the improvement method specifically includes the following steps.

[0090] A depth feature preprocessing module is added to the input layer of the YOLOV8n model to normalize the input left-eye RGB image and depth map.

[0091] A dual-branch feature extraction module is added to the BackBone part of the YOLOV8n model to process the left-eye RGB image and disparity map separately, and adaptively fuse feature weights through the channel-wise attention mechanism (CBAM) to solve the problem of false detection caused by occlusion.

[0092] A bi-level routing attention mechanism is added to the BackBone part of the YOLOV8n model to enhance the model's small target detection capability.

[0093] The EfficientRepGFPN feature fusion network is introduced into the Neck part of the YOLOV8n model to optimize the efficiency of multi-scale feature fusion.

[0094] The original loss function of the YOLOV8n model is replaced with the Inner-IoU loss function, and the auxiliary bounding box (Inner Box) calculation is introduced to solve the problem of missed detection when corn ears are densely arranged.

[0095] S3: Dynamic fitting of ground height. Based on depth map and ground plane equation Perform plane fitting to obtain the ground height.

[0096] In some embodiments of the present invention, the specific implementation method of step S3 includes:

[0097] The depth map obtained by preprocessing is highly screened to retain the point cloud data between 0.1m and 0.5m.

[0098] Use RANSAC algorithm to fit the ground plane equation , calculate the ground height , where ground height The calculation formula is:

[0099] .

[0100] The ground model is updated in the form of a real-time sliding window according to the direction of header movement.

[0101] S4: Calculate the height of the ear of corn. Perform a three-dimensional coordinate transformation on the pixel coordinates of the bottom center point of the bounding box based on the ground height and the depth value of the depth map. The pixel coordinates of the bottom center point of the bounding box are converted to the camera coordinate system. The height of the base of the ear of corn is calculated based on its coordinates in the camera coordinate system. The initial value of the height of the ear of corn is then calculated based on the height of the base of the ear of corn.

[0102] In some embodiments of the present invention, the coordinates of the bottom center point of the bounding box are , the coordinates of the bottom center point of the bounding box in the camera coordinate system are , the coordinates of the bottom center point of the bounding box Convert to camera coordinate system The calculation formula is:

[0103] ;

[0104] ;

[0105] ;

[0106] in, and is the focal length of the binocular camera, and is the principal point coordinate of the binocular camera, is the depth value of the depth map.

[0107] In some embodiments of the present invention, the root height of the corn ear is The calculation formula is:

[0108] .

[0109] S5: Vibration data of the corn harvester during operation is collected through various sensors installed on the corn harvester.

[0110] The vibration data and the initial value of corn ear height are fused through the vibration compensation algorithm to calculate the corrected value of corn ear height.

[0111] In some embodiments of the present invention, the sensors installed on the corn harvester include but are not limited to a three-axis acceleration sensor and a gyroscope. In step S5, the method of fusing by using a vibration compensation algorithm includes the following steps.

[0112] S51: Calculate the height offset of the ear of corn caused by vibration using vibration data collected by the sensor :

[0113] .

[0114] in, a(t) is The vertical acceleration measured by the triaxial acceleration sensor is in units of m / s². It should be noted that in this embodiment, a(t) The gravity component g=9.8m / s2 needs to be deducted; ω(t) The vertical angular velocity measured by the gyroscope is in rad / s and is used to compensate for the measurement error caused by the tilt. α is the tilt compensation coefficient, which is determined through calibration experiments. In this embodiment, the value range of α is 0.1~0.3.

[0115] S52: Offset and the initial value of corn ear height Perform fusion calculation to obtain the corrected value of corn ear height :

[0116] .

[0117] in, is the weight function, The expression is:

[0118] .

[0119] in, is the offset The variance is used to reflect the reliability of vibration data. is the variance of binocular vision measurement, obtained through offline calibration, is an attenuation coefficient used to control the sensitivity of vibration compensation. In this embodiment, the attenuation coefficient ranges from 0.5 to 1.0.

[0120] In some embodiments of the present invention, the method for performing fusion using a vibration compensation algorithm further includes the following steps.

[0121] S53: When vibration and visual data conflict, Kalman filtering is used to dynamically adjust the fusion weights ,

[0122] .

[0123] Among them, A and B are state coefficients, where A=1 means that the height change is a random walk, and B== means there is no external control input.

[0124] is the process noise covariance, which is determined by the vibration energy:

[0125] ;

[0126] in, .

[0127] is the observation noise covariance:

[0128] ;

[0129] in, .

[0130] S53: Output the adjusted correction value and use the adjusted correction value As a correction for the height of the ear of corn Perform subsequent calculations and adjust the correction value The calculation formula is:

[0131] .

[0132] S6: Based on the corn ear height correction value, the dynamic fuzzy PID control algorithm is used to dynamically adjust the header height of the corn harvester.

[0133] In some embodiments of the present invention, a method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID algorithm control algorithm based on a corn ear height correction value specifically includes the following steps.

[0134] The dynamic fuzzy PID control algorithm is used to control the switch of the hydraulic proportional valve in the hydraulic actuator of the corn harvester in real time, thereby controlling the stroke of the hydraulic cylinder, thereby changing the inclination angle of the corn harvester header to achieve the adjustment of the harvesting height of the header.

[0135] In some embodiments of the present invention, the method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm based on a corn ear height correction value further includes the following steps.

[0136] The angle sensor installed on the cutting platform collects the angle change data of the cutting platform in real time, and the pitch angle of the cutting platform is calculated as the cutting platform height through a predetermined angle and height conversion algorithm. When the calculated cutting platform height reaches the predetermined threshold, the cutting platform height is no longer adjusted to achieve closed-loop control of the cutting platform height.

[0137] In some embodiments of the present invention, the method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm further includes the following steps.

[0138] Based on the vibration-load coupling model and multi-sensor embedded Kalman filtering, the nonlinear characteristics of the hydraulic actuator of the corn harvester are compensated.

[0139] Some embodiments of the present invention further provide a multi-modal fusion corn harvester header height adaptive adjustment system, as shown in the attached Figure 2 -Attached Figure 4 As shown, the method for realizing the above-mentioned multi-modal fusion corn harvester header height adaptive adjustment method includes a visual detection module, a multi-sensor fusion module, a control terminal and a hydraulic actuator.

[0140] Among them, as attached Figure 2As shown in the figure, the visual detection module includes a binocular camera and an improved YOLOV8n model. The binocular camera is used to collect raw images of unharvested crops in front of the corn harvester in real time and preprocess the collected raw images to obtain a depth map. The visual detection module collects and preprocesses the image data and then transmits it to the control terminal in real time.

[0141] The multi-sensor fusion module is an IMU module, which includes multiple sensors and is used to collect vibration data during the operation of the corn harvester for IMU vibration compensation. The multi-sensor fusion module collects vibration data and transmits it to the control terminal in real time.

[0142] The control terminal is specifically an embedded control platform, which is communicated with the visual detection module, the multi-sensor fusion module and the hydraulic actuator respectively.

[0143] The control terminal is used to run the pre-trained YOLOV8n model accelerated by TensorTR to perform target detection on the images collected and preprocessed by the visual detection module, obtain the bounding box of the corn ear in the image, and calculate the coordinates of the bottom center point of the bounding box; and perform plane fitting based on the depth map and the ground plane equation to obtain the ground height; then calculate the initial value of the corn ear height based on the ground height, the coordinates of the bottom center point of the bounding box and the depth value of the depth map; and use the multi-sensor fusion module to collect vibration data to perform vibration compensation on the initial value of the corn ear height, and calculate the corn ear height correction value; then output the control instruction to the hydraulic actuator based on the corn ear height correction value.

[0144] The hydraulic actuator includes a hydraulic proportional valve, which is used to execute the control instructions sent by the control terminal. By controlling the switch of the hydraulic proportional valve, the stroke of the hydraulic cylinder is adjusted, thereby changing the inclination angle of the corn harvester's header to adjust the harvesting height of the header.

[0145] In some embodiments of the present invention, the control terminal adopts a heterogeneous computing architecture combining a Jetson Nano controller and an STM32 controller.

[0146] The Jetson Nano controller is used to run the pre-trained YOLOV8n model to perform model operations on the images collected and pre-processed by the visual inspection module and complete the calculation of the initial value of the corn ear height. At the same time, the Jetson Nano controller is also used to collect vibration data based on the multi-sensor fusion module to perform vibration compensation on the initial value of the corn ear height to obtain the corrected value of the corn ear height.

[0147] The STM32 controller is connected to the Jetson Nano controller and executes a dynamic fuzzy PID control algorithm based on the corn ear height correction value calculated by the Jetson Nano controller to output control instructions for the hydraulic actuator.

[0148] In some embodiments of the present invention, the hydraulic actuator further includes an angle sensor for acquiring the pitch angle of the header in real time so as to calculate the harvesting height of the header and then perform feedback adjustment on the harvesting height of the header.

[0149] As attached Figure 4 As shown, the installation positions of the modules in the corn harvester header height adaptive adjustment system of the multimodal fusion provided by the present invention are as follows:

[0150] A binocular camera and a fill light for night lighting and fill light are installed at the front end of the top of the cutting platform; a fan is installed on the side of the binocular camera to clean the camera lens through the wind; an angle sensor is installed at the hinge point of the cutting platform lifting arm; a multi-sensor fusion module, namely the IMU module, is installed in the center of the main frame of the cutting platform; an STM32 controller is installed in the middle of the left side of the cutting platform, and a hydraulic cylinder is installed in the hydraulic system area on the right side of the cutting platform.

[0151] The transmission process of control data in the system is as follows:

[0152] The real-time image and depth data collected by the binocular camera are transmitted to the Jetson Nano controller. The target height command obtained by the Jetson Nano controller is transmitted to the STM32 controller. The STM32 controller simultaneously receives the height data from the Jetson Nano controller, the lift angle data from the angle sensor, and the posture and vibration data from the multi-sensor fusion module. The algorithm in the STM32 controller generates a PWM control signal and transmits it to the proportional valve to control the hydraulic cylinder.

[0153] As attached Figure 3 As shown, the working process of the multi-modal fusion corn harvester header height adaptive adjustment system provided by the present invention is as follows:

[0154] When a corn harvester is harvesting, a binocular camera installed on the harvester's header collects real-time photos of the unharvested corn in the field in front of the harvester. The photo data collected by the binocular camera is transmitted in real time to the Jetson Nano controller in the control terminal. The pre-trained yolov8n model accelerated by TensorTR is run on the Jetson Nano controller to detect corn ears. The initial value of the corn ear height is calculated based on the installation height and baseline length of the binocular camera, which is 5 cm to 10 cm horizontally from the front edge of the header.

[0155] After completing the initial calculation of the corn ear height, the Jetson Nano controller transmits the initial corn ear height value to the STM32 controller via the CAN bus. The dynamic fuzzy PID control algorithm is executed in the STM32 controller to control the hydraulic proportional valve in real time, changing the hydraulic cylinder stroke and thus the inclination angle of the header. During this process, the angle sensor installed on the fixed axis of the header also changes its angle in real time and collects angle change data in real time. The pitch angle of the header is converted into the harvesting height of the header through the angle and height conversion algorithm preset in the STM32. The calculated height value is fed back to the STM32 controller in real time for feedback adjustment. When the harvesting height of the header reaches the predetermined threshold, the harvesting height of the header is no longer adjusted, thus achieving closed-loop control.

[0156] Finally, it should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0157] The above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the same. Although the present invention has been described in detail with reference to preferred embodiments, persons skilled in the art should understand that the specific implementation methods of the present invention may still be modified or some technical features may be replaced by equivalents without departing from the spirit of the technical solutions of the present invention, and all of these should fall within the scope of the technical solutions claimed for protection by the present invention.

Claims

1. A multi-modal fusion corn harvester header height adaptive adjustment method, characterized in that: The following steps are involved: Using a binocular camera to collect raw images of unharvested crops in front of a corn harvester in real time, and preprocessing the raw images to obtain a depth map; Input the original image and the depth map into a pre-trained YOLOv8n model, output the bounding box of the ear of corn in the image through model operation, and calculate the coordinates of the center point of the bottom of the bounding box; Performing plane fitting based on the depth map and the plane equation to obtain the ground height; Converting the coordinates of the bottom center point of the bounding box to a camera coordinate system based on the ground height and the depth value of the depth map, calculating the height of the base of the ear of corn based on the coordinates of the bottom center point in the camera coordinate system, and further calculating an initial value of the height of the ear of corn based on the height of the base of the ear of corn; The vibration data of the corn harvester during operation is collected by a sensor; the vibration data is integrated with the initial value of the corn ear height through a vibration compensation algorithm to calculate a corrected value of the corn ear height; Based on the corn ear height correction value, a dynamic fuzzy PID control algorithm is adopted to dynamically adjust the cutting platform height of the corn harvester.

2. The multimodal fusion corn harvester header height adaptive adjustment method according to claim 1 is characterized in that: The original image includes a left image and a right image, and the method for preprocessing the original image includes the following steps: Obtaining an intrinsic parameter matrix of the binocular camera, and performing geometric transformation based on the intrinsic parameter matrix so that the left and right image planes captured by the binocular camera are coplanar and their epipolar lines are parallel; Use the semi-global block matching algorithm to find the corresponding pixel points in the left and right images to calculate the disparity value and obtain the disparity map: A depth value is calculated based on the disparity value, and then the disparity map is converted into a depth map. During the conversion process, the depth map is assisted by structured light projection in the low-texture area of the image based on the active light source characteristics of the binocular camera.

3. The multimodal fusion corn harvester header height adaptive adjustment method according to claim 1, characterized in that: Before pre-training, the structure of the YOLOV8n model is improved. The improvement method specifically includes the following steps: Add a depth feature preprocessing module to the input layer of the YOLOV8n model to normalize the input original image and depth map; Adding a dual-branch feature extraction module to the BackBone part of the YOLOV8n model to process the original image and the disparity map obtained based on the original image respectively; Add a two-layer routing attention mechanism to the BackBone part of the YOLOV8n model; Introducing the EfficientRepGFPN feature fusion network into the Neck part of the YOLOV8n model; The original loss function of the YOLOV8n model is replaced with the Inner-IoU loss function.

4. The method for adaptively adjusting the height of a corn harvester header using multimodal fusion according to any one of claims 1 to 3, characterized in that: The coordinates of the bottom center point of the bounding box are , the coordinates of the bottom center point of the bounding box in the camera coordinate system are , the coordinates of the bottom center point of the bounding box Convert to camera coordinate system The calculation formula is: ； ； ； in, and is the focal length of the binocular camera, and is the principal point coordinate of the binocular camera, is the depth value of the depth map; The calculation formula for the root height of the corn ear is: ； in, is the height of the root of the corn ear, is the ground height.

5. The method for adaptively adjusting the height of a corn harvester header using multimodal fusion according to any one of claims 1 to 3, characterized in that: The expression of the plane equation is: ； The ground height The calculation formula is: 。 6. The method for adaptively adjusting the height of a corn harvester header using multimodal fusion according to any one of claims 1 to 3, characterized in that: The sensors installed on the corn harvester include a three-axis accelerometer and a gyroscope. The method of fusing them through a vibration compensation algorithm includes the following steps: Calculate the height deviation of the ear of corn caused by vibration using vibration data collected by the sensor : ； in, for The vertical acceleration measured by the triaxial accelerometer, is the vertical angular velocity measured by the gyroscope, is the tilt compensation coefficient; Height offset for ears of corn and the initial value of corn ear height Perform fusion calculation to obtain the corrected value of corn ear height : ； in, is the weight function, The expression is: ； in, is the offset The variance of is the variance of the binocular camera measurement, is the attenuation coefficient.

7. The multimodal fusion corn harvester header height adaptive adjustment method according to claim 1, characterized in that: The method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm based on the corn ear height correction value specifically comprises the following steps: The dynamic fuzzy PID control algorithm is used to control the stroke of the hydraulic cylinder by real-time controlling the switch of the hydraulic proportional valve in the hydraulic actuator of the corn harvester, so that the inclination angle of the corn harvester header is changed to achieve the adjustment of the harvesting height of the header.

8. The multi-modal fusion corn harvester header height adaptive adjustment method according to claim 7, characterized in that: The method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm based on the corn ear height correction value specifically comprises the following steps: The angle sensor installed on the cutting platform collects the angle change data of the cutting platform in real time, and the pitch angle of the cutting platform is calculated as the cutting platform height through a predetermined angle and height conversion algorithm. When the calculated cutting platform height reaches the predetermined threshold, the cutting platform height is no longer adjusted to achieve closed-loop control of the cutting platform height.

9. The method for adaptively adjusting the height of a corn harvester header based on multimodal fusion according to claim 7 or 8, characterized in that: The method for dynamically adjusting the header height of a corn harvester using a dynamic fuzzy PID control algorithm further includes: Based on the vibration-load coupling model and multi-sensor embedded Kalman filtering, the nonlinear characteristics of the hydraulic actuator of the corn harvester are compensated.

10. A multi-modal fusion corn harvester header height adaptive adjustment system, used to implement the multi-modal fusion corn harvester header height adaptive adjustment method according to any one of claims 1 to 9, characterized in that: Including visual detection module, multi-sensor fusion module, control terminal and hydraulic actuator: The visual detection module includes a binocular camera for collecting raw images of unharvested crops in front of the corn harvester in real time and preprocessing the collected raw images to obtain a depth map; The multi-sensor fusion module includes multiple sensors for collecting vibration data during the operation of the corn harvester. The control terminal is respectively in communication with the visual detection module, the multi-sensor fusion module and the hydraulic actuator; The control terminal is used to run a pre-trained YOLOV8n model to perform model calculations on the image collected and pre-processed by the visual detection module to obtain a bounding box of the corn ear in the image, and calculate the coordinates of the bottom center point of the bounding box; perform plane fitting based on the depth map and the plane equation to obtain the ground height; then calculate the initial value of the corn ear height based on the ground height, the coordinates of the bottom center point of the bounding box, and the depth value of the depth map; and use the multi-sensor fusion module to collect vibration data to perform vibration compensation on the initial value of the corn ear height to calculate a corrected value of the corn ear height; and then output a control instruction for the hydraulic actuator based on the corrected value of the corn ear height; The hydraulic actuator includes a hydraulic proportional valve, which is used to execute the control instructions sent by the control terminal. By controlling the switch of the hydraulic proportional valve, the stroke of the hydraulic cylinder is adjusted, thereby changing the inclination angle of the corn harvester's cutting table to achieve adjustment of the harvesting height of the cutting table.

Images