A multi-source information fusion combined harvester auxiliary driving night-time early harvesting control system and method

Abstract

The application provides a multi-source information fusion combined harvester auxiliary driving night-time rush harvesting control system and method, which comprises a multi-modal perception module, an airborne computing terminal and an adaptive decision control module; the multi-modal perception module collects three-dimensional geometric point clouds, visible light texture images, infrared thermal radiation images and six-axis vibration data; the airborne computing terminal is used for executing a space-time joint calibration and data fusion algorithm to generate an enhanced semantic point cloud map containing spatial coordinates, RGB color information and temperature information; the adaptive decision control module identifies crop lodging areas, harvesting boundaries and headlands, and outputs corresponding control instructions to realize speed reduction, cutting width correction and collision-free turning. Through multi-modal information fusion and dynamic anti-vibration compensation, the application breaks through the night-time perception bottleneck, realizes the identification and adaptive control of lodging crops, harvesting boundaries and headlands, reduces the harvesting loss rate and the turning collision risk, and improves the intelligence and safety of night-time harvesting operations.

Description

Technical Field

[0001] This invention belongs to the field of intelligent control technology for agricultural machinery, specifically relating to a multi-source information fusion-based assistive driving control system and method for nighttime harvesting of combine harvesters. Background Technology

[0002] Nighttime operation of combine harvesters is an important means to improve agricultural production efficiency and seize the farming season. However, the nighttime environment brings great challenges to safe and efficient harvesting operations. Poor lighting conditions during nighttime harvesting mean drivers can only rely on the vehicle's headlights for observation, severely limiting their field of vision and clarity, leading to visual fatigue, and making it difficult to accurately judge complex terrain and obstacles, operating conditions, and work boundaries.

[0003] Traditional harvester systems largely rely on single-vision navigation or conventional satellite positioning. Existing single-vision or single-radar systems struggle to distinguish between normal, low-stalk crops and lodged crops at night. Without timely deceleration and lowering of the header, lodged crops can easily entangle and clog the header, leading to significant missed harvests. Limited visibility at night makes it difficult to perfectly align with the harvesting boundary of the previous row using only path planning. Excessive cutting width can cause a surge in feed, potentially leading to drum blockage and engine shutdown; insufficient cutting width reduces operational efficiency. Existing turning strategies are mostly based on planar path planning, neglecting the large reach and wide size of the harvester's header. At night, when the height of field ridges is unclear, the header can easily sweep across ridges or irrigation facilities during rotation, causing equipment damage. Summary of the Invention

[0004] To address the aforementioned technical problems, this invention provides a multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system and method. This invention utilizes multi-source information fusion of lidar, vision, and thermal imaging to achieve accurate perception of lodged crops, harvesting boundaries, and field ridges in nighttime environments, and automatically adjusts the harvester's operating parameters and driving path.

[0005] This invention overcomes the bottleneck of nighttime perception by integrating multimodal information and dynamic seismic compensation, enabling the identification and adaptive control of lodged crops, harvesting boundaries, and field ridges, thereby reducing harvesting loss rates, minimizing the risk of collisions when turning around, and improving the intelligence and safety of nighttime harvesting operations.

[0006] The present invention achieves the above-mentioned technical objectives through the following technical means.

[0007] A multi-source information fusion-based assisted driving control system for combine harvesters for nighttime emergency harvesting includes a multimodal perception module, an onboard computing terminal, and an adaptive decision control module.

[0008] The multimodal sensing module is installed at a high position at the front of the combine harvester to collect data. The multimodal sensing module includes a lidar, a binocular stereo camera, an infrared thermal imager, and a high-frequency inertial measurement unit. The lidar is used to collect three-dimensional geometric point clouds of the working environment. The binocular stereo camera is used to collect visible light texture images of the working environment. The infrared thermal imager is used to collect infrared thermal radiation images of the working environment. The high-frequency inertial measurement unit is used to collect six-axis vibration acceleration and attitude angular velocity data during the harvester's operation.

[0009] The airborne computing terminal is communicatively connected to the multimodal perception module and is configured to execute a spatiotemporal joint calibration and data fusion algorithm to generate an enhanced semantic point cloud map containing spatial coordinates, RGB color information and temperature information, as an enhanced three-dimensional environment model.

[0010] The adaptive decision control module calculates the current working conditions in real time based on the enhanced semantic point cloud map and sends control commands to the underlying actuators of the combine harvester.

[0011] The control logic of the adaptive decision control module includes: when a lodged crop area is detected, a first control command is output to control the chassis to reduce its driving speed and simultaneously control the hydraulic system of the header to lower the header height; when the boundary between harvested and unharvested areas is detected, a second control command is output to adjust the steering angle according to the relative position of the header and the boundary to maintain the preset cutting width; when a field ridge is detected, a third control command is output to plan a collision-free turning path based on the three-dimensional spatial envelope of the header.

[0012] In the above scheme, the multimodal sensing module is configured to perform the following steps to collect data:

[0013] Step S11: Perform intrinsic parameter calibration on the binocular stereo camera and the infrared thermal imager respectively, and obtain focal length, principal point coordinates, distortion coefficients and baseline parameters.

[0014] Step S12: A multimodal universal calibration target is set up in the area where the lidar and vision co-exist in front of the harvester. With the vehicle body completely stationary, multiple sets of synchronous feature data are collected by changing the distance and deflection attitude of the calibration target in space. The initial static extrinsic parameter matrix of the lidar to the coordinate system of the binocular left camera and the initial static extrinsic parameter matrix of the lidar to the coordinate system of the infrared thermal imager are calculated. The multimodal universal calibration target includes a substrate layer, an active heating layer and a surface pattern layer. The surface pattern layer adopts a black and white checkerboard design. The black squares and white squares are made of two materials with different thermal emissivity, so that the calibration target presents a hot and cold checkerboard pattern in the infrared thermal imager that is consistent with the visible light image.

[0015] Step S13: Based on the ROS environment, the data collected by the lidar, binocular stereo camera and infrared thermal imager are time-stamped and synchronized. The closest set of radar frames, visual frames and infrared frames within the time window are packaged into a synchronization data frame.

[0016] Step S14: Downsample and filter out outliers from the synchronized point cloud data, and perform distortion correction on the binocular visible light image and infrared thermal image.

[0017] Furthermore, the airborne computing terminal is also used to perform dynamic online calibration of seismic compensation, including:

[0018] Using the high-frequency vibration data acquired by the high-frequency inertial measurement unit, a dynamic displacement deviation model of the sensor support is established, and instantaneous attitude compensation is performed on the raw data of the lidar and the binocular stereo camera.

[0019] During the harvester's operation, ground plane features in the point cloud and crop row edge features in the visual image are extracted in real time. By minimizing the geometric residual function, the real-time extrinsic matrix of the lidar to the binocular stereo camera is dynamically corrected.

[0020] Extended Kalman filtering is used to smooth the corrected displacement matrix, and the real-time corrected dynamic extrinsic parameter matrix is ​​output to eliminate point cloud ghosting or layering caused by the severe shaking of the harvester.

[0021] Furthermore, the airborne computing terminal is configured to perform the following steps to generate an enhanced semantic point cloud map:

[0022] Step S21: Traverse the original point cloud data collected by the lidar to obtain each 3D lidar point. ,

[0023] Obtain three-dimensional geometric information;

[0024] Step S22: Using the real-time corrected dynamic extrinsic parameter matrix and the calibrated camera intrinsic parameter matrix, the laser point is projected...

[0025] The image is projected onto the visible light image plane, and the RGB color components of the corresponding pixel positions are extracted to obtain RGB color information;

[0026] Step S23: Using the real-time corrected dynamic extrinsic parameter matrix and the calibrated thermal imager intrinsic parameter matrix, the same...

[0027] A laser point is projected onto an infrared image plane to obtain the grayscale value of the corresponding pixel position, and then converted into a physical temperature value according to the thermoradiative model to obtain temperature information.

[0028] Step S24: Write the 3D geometric information, RGB color information, and temperature information into the same target point data structure to generate an enhanced semantic point cloud map. The data format is as follows: As an enhanced 3D environment model.

[0029] In the above scheme, the adaptive decision control module is configured to perform the following steps to identify the collapsed area:

[0030] In the enhanced semantic point cloud map, point cloud clusters with heights lower than a preset threshold for normal crop height are selected;

[0031] By combining the infrared temperature characteristics and visible light texture characteristics corresponding to the point cloud clusters, and eliminating interference from bare soil and puddles, the remaining low-lying, high-density point cloud areas are identified as collapsed areas.

[0032] In the above scheme, the generation logic of the second control command is specifically as follows:

[0033] Extract crop elevation features from the enhanced semantic point cloud map and fit the edge curves of unharvested crops;

[0034] Calculate the lateral deviation between the outermost tangent point of the harvester header and the edge curve;

[0035] When the lateral deviation value indicates that the current cutting width is less than half of the preset cutting width and it is not the final stage, a correction signal is generated to turn towards the unharvested side until the current cutting width reaches between two-thirds and three-quarters of the preset cutting width.

[0036] In the above scheme, the generation logic of the third control command is specifically as follows:

[0037] A three-dimensional terrain model of field ridges was reconstructed using high-density point clouds from lidar.

[0038] Construct a three-dimensional kinematic model of the harvester, including the length, width, and height of the header.

[0039] Under the constraint of satisfying the minimum turning radius of the harvester, a turning trajectory is planned that ensures that the three-dimensional kinematic model of the harvester maintains a safe distance from the three-dimensional terrain model of the field ridge during the movement.

[0040] The above solution also includes a remote mobile monitoring terminal;

[0041] The remote mobile monitoring terminal is connected to the airborne computing terminal via a wireless network. It is used to display the visualization rendering results of the enhanced semantic point cloud map in real time, and has human-computer interaction functions such as remotely issuing work tasks, modifying work parameter thresholds, and performing emergency shutdown.

[0042] A control method for a combine harvester assisted driving nighttime harvesting control system based on multi-source information fusion includes the following steps:

[0043] Step S1: Simultaneously collect lidar point cloud data, binocular vision image data, and infrared thermal imaging data of the environment in front of the harvester through the multimodal perception module.

[0044] Step S2: The airborne computing terminal performs spatiotemporal registration and fusion of multi-source data, and constructs an enhanced 3D environment model with temperature and texture information based on the enhanced semantic point cloud map.

[0045] Step S3: Based on the enhanced 3D environment model, the adaptive decision control module performs fall detection, boundary tracking, and obstacle recognition in parallel.

[0046] Step S4: If crop lodging is detected, the adaptive decision control module automatically calculates the target operating speed and target header height based on the lodging area and degree, and drives the harvester to decelerate and lower the header.

[0047] Step S5: If the harvesting is in progress, the crop boundary line is detected in real time by the adaptive decision control module, and the header is kept at the optimal operating width by lateral correction control.

[0048] Step S6: If a field ridge is detected ahead, the adaptive decision control module automatically generates and executes a collision avoidance turning maneuver strategy based on the geometry of the field ridge and the spatial position of the cutting platform.

[0049] In the above scheme, step S3 uses infrared thermal imaging data to assist in nighttime identification. By using the difference in thermal inertia between crops and the abiotic environment, upright crops, lodged crops, and field obstacles can be distinguished under low light conditions.

[0050] Compared with the prior art, the beneficial effects of the present invention are:

[0051] 1. This invention overcomes the bottleneck of nighttime perception, enabling highly reliable all-weather operation. It employs a multimodal perception architecture combining lidar, a binocular stereo camera, and an infrared thermal imager. Through spatiotemporal joint calibration and dynamic online seismic compensation, it generates an enhanced semantic point cloud map that integrates 3D geometry, RGB color, and temperature information. Even under low-light conditions at night, it can accurately distinguish between upright crops, lodged crops, field ridges, and obstacles, solving the problem of single-vision or single-radar perception failure at night and facilitating highly reliable all-weather operation.

[0052] 2. This invention features intelligent adaptive control, significantly reducing harvesting loss rates. It accurately identifies lodged crop areas and harvested / unharvested boundaries using enhanced semantic point cloud maps. When lodged crops are detected, the adaptive decision control module automatically reduces the travel speed and header height to prevent header blockage and missed harvesting. When a deviation in the cutting width is detected, the system automatically adjusts the steering angle towards the unharvested side, maintaining the cutting width between two-thirds and three-quarters of the preset width, avoiding overloading of the feed or missed operations, thus significantly reducing harvesting loss rates.

[0053] 3. This invention eliminates the risk of collisions when turning around by using a spatial path planning based on a three-dimensional model. The invention utilizes a high-density point cloud from a lidar system to reconstruct a three-dimensional terrain model of the field ridges, and combines it with a three-dimensional kinematic model of the harvester that includes the length, width, and height of the cutter head. Under the constraint of satisfying the minimum turning radius, a collision-free turning trajectory that always maintains a safe distance from the field ridges is planned, effectively avoiding damage to the cutter head caused by collisions at night due to the inability to clearly see the height of the field ridges.

[0054] 4. This invention constructs a "human-machine collaborative" remote supervision mode to improve operational safety. This invention wirelessly connects a remote mobile monitoring terminal with an onboard computing terminal to display the visualization rendering results of an enhanced semantic point cloud map in real time. It also supports remotely issuing work tasks, modifying work parameter thresholds, and executing emergency shutdowns, realizing real-time monitoring and intervention of harvesting operations by humans. This constructs a "human-machine collaborative" safety supervision mode, which significantly improves the safety of nighttime operations. Attached Figure Description

[0055] Figure 1 This is a schematic diagram of the overall hardware architecture of the system of the present invention.

[0056] Figure 2 A flowchart for multimodal data fusion and enhanced semantic point cloud map generation.

[0057] Figure 3 A schematic diagram illustrating the logic for identifying lodged areas and adjusting adaptive operation parameters.

[0058] Figure 4 This is a schematic diagram of the automatic control principle for the cutting width based on boundary recognition.

[0059] Figure 5 This is a schematic diagram of barrier-free U-turn path planning based on the spatial envelope of the cutting platform.

[0060] In the diagram, 1 is a combine harvester; 101 is a multimodal sensing module; 102 is an onboard computing terminal; and 103 is an adaptive decision control module. Detailed Implementation

[0061] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0062] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "front," "rear," "left," "right," "upper," "lower," "axial," "radial," "vertical," "horizontal," "inner," and "outer," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined with "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0063] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0064] This invention aims to address the problems of insufficient environmental perception, low safety, poor work quality, and inability to adaptively control complex working conditions such as lodging in existing combine harvesters during nighttime operations. It provides a control system and method that can deeply integrate three-dimensional geometry, visible light, and infrared thermal imaging information to achieve comprehensive and accurate monitoring of the working environment and status, thereby improving the intelligence level and safety of combine harvester harvesting operations.

[0065] like Figure 1 As shown, a multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system includes a multimodal perception module 101, an airborne computing terminal 102, and an adaptive decision control module 103.

[0066] The multimodal sensing module 101 is installed at a high position at the front of the combine harvester 1 to collect data. The multimodal sensing module 101 includes a time-synchronized LiDAR, a binocular stereo camera, an infrared thermal imager, and a high-frequency inertial measurement unit. The LiDAR is used to collect three-dimensional geometric point clouds of the working environment. The binocular stereo camera is used to collect visible light texture images of the working environment. The infrared thermal imager is used to collect infrared thermal radiation images of the working environment. The high-frequency inertial measurement unit is used to collect six-axis vibration acceleration and attitude angular velocity data during the operation of the harvester.

[0067] The airborne computing terminal 102 is communicatively connected to the multimodal perception module 101 and is configured to execute a spatiotemporal joint calibration and data fusion algorithm to generate an enhanced semantic point cloud map containing spatial coordinates, RGB color information and temperature information as an enhanced three-dimensional environment model.

[0068] The adaptive decision control module 103 calculates the current working condition in real time based on the enhanced semantic point cloud map and sends control commands to the underlying actuators of the combine harvester.

[0069] The control logic of the adaptive decision control module 103 includes: when a lodged crop area is detected, a first control command is output to control the chassis to reduce its driving speed and simultaneously control the hydraulic system of the header to reduce the height of the header; when the boundary between harvested and unharvested areas is detected, a second control command is output to adjust the steering angle according to the relative position of the header and the boundary to maintain the preset cutting width; when a field ridge is detected, a third control command is output to plan a collision-free turning path based on the three-dimensional spatial envelope of the header.

[0070] The multimodal sensing module 101 is configured to perform the following steps to acquire data:

[0071] Step S11: Perform intrinsic parameter calibration on the binocular stereo camera and the infrared thermal imager respectively, and obtain focal length, principal point coordinates, distortion coefficients and baseline parameters.

[0072] Specifically, using a standard black and white checkerboard calibration board, multiple sets of images were captured at different angles and distances. OpenCV was used to calculate the focal length, principal point coordinates, and distortion coefficients (radial and tangential distortion) of the left and right cameras. Baseline calibration and epipolar correction of the binocular cameras were completed. A thermal source calibration board was fabricated, infrared images were acquired from different orientations, and the intrinsic parameter matrix and distortion parameters of the thermal imager were calculated.

[0073] Step S12: A multimodal universal calibration target is set up in the area where the lidar and vision co-exist in front of the harvester. With the vehicle body completely stationary, multiple sets of synchronous feature data are collected by changing the distance and deflection attitude of the calibration target in space. The initial static extrinsic parameter matrix of the lidar to the coordinate system of the binocular left camera and the initial static extrinsic parameter matrix of the lidar to the coordinate system of the infrared thermal imager are calculated. The multimodal universal calibration target includes a substrate layer, an active heating layer and a surface pattern layer. The surface pattern layer adopts a black and white checkerboard design. The black squares and white squares are made of two materials with different thermal emissivity, so that the calibration target presents a hot and cold checkerboard pattern in the infrared thermal imager that is consistent with the visible light image.

[0074] Specifically, a special composite calibration target is used to fabricate a multimodal universal calibration target. This target material possesses visual features, thermal infrared features, and geometric structural features simultaneously. While keeping the calibration target stationary, laser point clouds and binocular images are acquired simultaneously. Using a joint calibration tool, the planar or edge features of the calibration board are extracted from the point cloud data, and the corresponding corner pixels are extracted from the image data. The rotation and translation matrix from the lidar coordinate system to the binocular left camera coordinate system is calculated using the PnP algorithm or a nonlinear optimization method. Again, while keeping the calibration target stationary, laser point clouds and infrared thermal imaging images are acquired. Calibration board features are extracted from the point cloud, and heat source corner features are extracted from the infrared image. By establishing the correspondence between 3D point cloud feature points and 2D infrared pixel feature points, the rotation and translation matrix from the lidar coordinate system to the infrared thermal imager coordinate system is calculated. The multimodal universal calibration target is specifically a multi-layer composite target plate. The target plate includes: a substrate layer made of a rigid material with good flatness, providing a flat point cloud reflective surface and clear geometric edge features for the lidar; an active heating layer embedded inside the substrate or attached to the back, used to provide a stable and uniform heat source; and a surface pattern layer with a black and white checkerboard design, where the black and white squares are made of two materials with significantly different thermal emissivity. The black areas use a special light-absorbing black paint with high emissivity, while the white areas use smooth metal foil with low emissivity. When the active heating layer is turned on, due to the difference in surface emissivity, the calibration target will present a high-contrast black and white checkerboard pattern in the infrared thermal imager that is completely consistent with that of a visible light binocular camera, thereby achieving simultaneous calibration by radar, vision, and infrared.

[0075] Step S13: Based on the ROS environment, the data collected by the lidar, binocular stereo camera and infrared thermal imager are time-stamped and synchronized. The closest set of radar frames, visual frames and infrared frames within the time window are packaged into a synchronization data frame.

[0076] Specifically, in one embodiment of the present invention, when the harvester operating system starts, each sensor driver node is initialized based on the ROS environment. The lidar driver node publishes the raw point cloud topic / lidar_points at a frequency of 10Hz; the binocular camera driver node publishes the image topics / left / image_raw and / right / image_raw at a frequency of 15Hz; and the infrared thermal imager driver node publishes the thermal infrared image topic / thermal / image_raw at a frequency of 30Hz. A time synchronization node is started, which subscribes to the topics of the above three sensors. Using an approximate timestamp matching algorithm, within the allowable time error window, a set of "radar frame-visual frame-infrared frame" with the closest timestamps is found, and they are packaged into a set of synchronized data frames for output, ensuring the consistency of data in the time dimension during fusion.

[0077] Step S14: Downsample and filter out outliers from the synchronized point cloud data, and perform distortion correction on the binocular visible light image and infrared thermal imaging image.

[0078] Specifically, the point cloud data after subscription and synchronization is downsampled using a voxel grid filter to remove redundant dense points and reduce computational load; at the same time, a statistical outlier removal filter is used to filter out discrete noise points caused by dust or flying insects.

[0079] Using the intrinsic parameters and distortion coefficients obtained in S11, distortion correction is performed on the binocular visible light image and infrared thermal imaging image respectively to eliminate the influence of lens distortion on geometric projection accuracy and obtain an ideal linear perspective image.

[0080] The airborne computing terminal 102 is also used to perform dynamically online calibrated seismic compensation, including:

[0081] Using the high-frequency vibration data obtained by the high-frequency inertial measurement unit, a dynamic displacement deviation model of the sensor support is established. That is, by integrating the three-axis acceleration and angular velocity under the synchronous timestamp, the instantaneous translation vector and rotation quaternion compensation of the sensing array relative to the initial static external parameters are calculated, and then the instantaneous attitude compensation of the original data of the lidar and the binocular stereo camera is performed.

[0082] During the harvester's operation, ground planar features from the point cloud and crop row edge features from the visual image are extracted in real time. By minimizing the geometric residual function, the real-time extrinsic parameter matrix of the lidar-camera stereo camera is dynamically corrected: radar-camera extrinsic parameter matrix. ;

[0083] Extended Kalman filtering is used to smooth the corrected displacement matrix, and the real-time corrected dynamic extrinsic parameter matrix is ​​output to eliminate point cloud ghosting or layering caused by the severe shaking of the harvester.

[0084] In one specific embodiment of the present invention, the lidar is a 32-line lidar.

[0085] The airborne computing terminal 102 is configured to perform the following steps to generate an enhanced semantic point cloud map:

[0086] Step S21: Initialize an enhanced point cloud container with a custom structure, and iterate through the original points cloud data collected by the 32-line LiDAR.

[0087] Point cloud data, to obtain each 3D laser point This yields three-dimensional geometric information;

[0088] Step S22: Using the real-time corrected dynamic extrinsic parameter matrix and the calibrated camera intrinsic parameter matrix, the laser point is projected...

[0089] Projecting the image onto the visible light image plane, extracting the RGB color components at the corresponding pixel positions, and obtaining the RGB color information:

[0090] Specifically, using the calibrated radar-camera extrinsic parameter matrix , laser point Transform from the radar coordinate system to the left camera coordinate system of the stereo camera to obtain the camera coordinate points. Using the camera's intrinsic parameter matrix (Including focal length) , He Guangxin , ) and distortion coefficients, to three-dimensional points Projected onto the pixel plane of a two-dimensional image, the corresponding pixel coordinates are calculated. ,examine Does it fall within the resolution range of the visible light image? If it does, then index the visible light image in... Extract the red, green, and blue color components from the pixel values ​​at that location. ;

[0091] Step S23: Using the real-time corrected dynamic extrinsic parameter matrix and the calibrated thermal imager intrinsic parameter matrix, ...

[0092] The same laser point is projected onto the infrared image plane, the grayscale value of the corresponding pixel position is obtained, and then converted into a physical temperature value according to the thermoradiative model to obtain the temperature information:

[0093] Specifically, using the calibrated radar-thermal imager extrinsic parameter matrix The same laser point The coordinates of the thermal imager are obtained by transforming from the radar coordinate system to the coordinate system of the infrared thermal imager. Using the intrinsic parameter matrix of an infrared thermal imager , three-dimensional points Projected onto an infrared two-dimensional image plane, the corresponding thermal image pixel coordinates are calculated. ,examine Whether it falls within the resolution range of the infrared image; if it is within the range, then acquire the infrared image. The original gray value at the location And based on the thermal imager's thermometric radiation model, the grayscale values... Convert to actual physical temperature value ;

[0094] Step S24: Create a target point data structure containing multi-dimensional attributes, including three-dimensional geometric information. RGB color information With temperature information Write data to the same target point data structure to generate an enhanced semantic point cloud map. The data format is as follows: As an enhanced 3D environment model;

[0095] Step S25: Store the encapsulated multi-attribute points into an enhanced point cloud container. Repeat the above steps for all laser points to finally generate an enhanced semantic point cloud map with timestamp alignment, which includes three-dimensional geometry, texture color, and thermal field distribution information, for subsequent analysis by the decision-making module.

[0096] The adaptive decision control module 103 is configured to perform the following steps to identify the collapsed area:

[0097] In the enhanced semantic point cloud map, point cloud clusters with heights lower than a preset normal crop height threshold are filtered out; preferably, the preset normal crop height threshold is obtained by automatically calculating the crop height distribution during the initial scanning stage after the harvester enters the plot, or is preset by the user according to the crop type.

[0098] By combining the infrared temperature characteristics and visible light texture characteristics corresponding to the point cloud clusters, and eliminating interference from bare soil and puddles, the remaining low-lying, high-density point cloud areas are identified as collapsed areas.

[0099] The generation logic for the second control instruction is as follows:

[0100] Extract crop elevation features from the enhanced semantic point cloud map and fit the edge curves of unharvested crops;

[0101] Calculate the lateral deviation between the outermost tangent point of the harvester header and the edge curve;

[0102] When the lateral deviation value indicates that the current cutting width is less than half of the preset cutting width, and the current operation stage is determined to be a non-final stage based on point cloud boundary recognition, a correction signal is generated to turn to the unharvested side until the current cutting width reaches between two-thirds and three-quarters of the preset cutting width, to prevent missed harvesting or empty runs.

[0103] The generation logic of the third control command is as follows:

[0104] A three-dimensional terrain model of field ridges was reconstructed using high-density point clouds from lidar.

[0105] Construct a three-dimensional kinematic model of the harvester, including the length, width, and height of the header.

[0106] Under the constraint of satisfying the minimum turning radius of the harvester, a turning trajectory is planned that ensures that the three-dimensional kinematic model of the harvester maintains a safe distance from the three-dimensional terrain model of the field ridge during the movement.

[0107] The multi-source information fusion combine harvester assisted driving night harvesting control system also includes a remote mobile monitoring terminal;

[0108] The remote mobile monitoring terminal is connected to the airborne computing terminal 102 via a wireless network. It is used to display the visualization rendering results of the enhanced semantic point cloud map in real time, and has human-computer interaction functions such as remotely issuing work tasks, modifying work parameter thresholds, and performing emergency shutdown.

[0109] A control method for a combine harvester assisted driving nighttime harvesting control system based on multi-source information fusion includes the following steps:

[0110] Step S1: The multimodal sensing module 101 synchronously collects lidar point cloud data, binocular vision image data and infrared thermal imaging data of the environment in front of the harvester.

[0111] Step S2: The airborne computing terminal 102 performs spatiotemporal registration and fusion of multi-source data, and constructs an enhanced three-dimensional environment model with temperature and texture information based on the enhanced semantic point cloud map.

[0112] Step S3: The adaptive decision control module 103 performs fall detection, boundary tracking and obstacle recognition in parallel based on the enhanced 3D environment model.

[0113] Step S4: If crop lodging is detected, the adaptive decision control module 103 automatically calculates the target operating speed and target header height based on the lodging area and degree, and drives the harvester to decelerate and lower the header.

[0114] Step S5: If the harvesting is in progress, the crop boundary line is detected in real time by the adaptive decision control module 103, and the cutting platform is kept at the optimal operating width by lateral correction control.

[0115] Step S6: If a field ridge is detected ahead, the adaptive decision control module 103 automatically generates and executes a collision avoidance turning maneuver strategy based on the geometric shape of the field ridge and the spatial position of the cutting platform.

[0116] In step S3, infrared thermal imaging data is used to assist in nighttime identification. By using the difference in thermal inertia between crops and the abiotic environment, upright crops, lodged crops, and field obstacles can be distinguished under low light conditions.

[0117] In one specific embodiment of the present invention, such as Figure 1 As shown, the multi-source information fusion combine harvester assisted driving nighttime harvesting control system is installed on a tracked combine harvester. The multimodal perception module 101 includes a 32-line mechanical rotating lidar mounted at the center of the top of the cab, with a binocular stereo camera and a long-wave infrared thermal imager arranged on either side. The industrial control computer of the onboard computing terminal 102 collects radar point clouds, visual images, and thermal imaging data via an Ethernet switch.

[0118] In one specific embodiment of the present invention, such as Figure 2 As shown in the flowchart, the overall data fusion process, from multi-source data acquisition, spatiotemporal joint calibration, dynamic seismic compensation to the generation of enhanced semantic point cloud maps, is illustrated. The data fusion process is as follows:

[0119] A multimodal universal calibration target is pre-set within the radar and vision shared field of view directly in front of the harvester. Multiple sets of synchronous feature data are collected while the vehicle is completely stationary, accurately calculating the initial static rotation and translation matrices of the camera and thermal imager relative to the lidar. During actual harvester operation, the system reads high-frequency vibration data from the IMU in real time and performs dynamic online anti-vibration compensation by combining point cloud and image features, outputting a corrected dynamic extrinsic parameter matrix in real time. During data fusion, each frame of point cloud from the lidar is projected onto the infrared and visible light image planes based on this real-time corrected dynamic extrinsic parameter matrix, assigning color and temperature information to each lidar point. For points in the point cloud... Find its corresponding pixel on the thermal image. Assign the temperature value of the pixel to the point The final generated data format is This solves the problems of a single radar being unable to distinguish camouflage, single visual system failing at night, and multimodal data projection misalignment caused by severe vibrations.

[0120] During nighttime operations, the lidar detected a sudden drop in crop height 5 meters ahead from the average height of upright rice plants. However, due to the lodged rice plants not being fully in contact with the ground and having an angle, the point cloud information was uneven, and the fused thermal imaging data showed that the temperature in the area was between that of the surrounding upright crops and the exposed soil. The system confirmed that the crops had "lodged".

[0121] The decision module immediately issues the following instructions: (1) Change the operating speed from 4 km / h to stationary operation until the lodged rice in the work area is harvested, and then the harvester continues to move forward; (2) Control the hydraulic cylinder of the header lifting mechanism to adjust the header height from about 20 cm above the ground to flush with the ground; (3) Adjust the speed and extension position of the reel. After passing through the lodged area, the parameters automatically return to normal. This process requires no manual intervention and effectively prevents machine malfunctions caused by poor visibility of lodged rice at night.

[0122] The process for identifying lodged areas and adjusting adaptive operating parameters is as follows: Figure 3 As shown. During the harvesting process, the system identifies the work boundary between harvested and unharvested areas by the difference in point cloud density. When it detects that the current cutting width is less than half of the preset cutting width, the adaptive decision control module 103 outputs a second control command, automatically adjusting the steering angle towards the unharvested side, so that the cutting width is maintained between two-thirds and three-quarters of the preset cutting width, achieving efficient operation.

[0123] The principle of automatic slitting control based on boundary recognition is as follows: Figure 4 As shown, the principle of automatic control of the cutting width based on boundary recognition is as follows: Figure 4 As shown. The adaptive decision control module 103 executes the following automatic cutting width control logic: First, it extracts the crop elevation features from the enhanced semantic point cloud map and fits the edge curve of the unharvested crop; second, it calculates the lateral deviation value between the outermost tangent point of the harvester's header and the edge curve; finally, when the lateral deviation value indicates that the current cutting width is less than half of the preset cutting width, and the current operation stage is determined to be a non-final stage based on GPS positioning or point cloud boundary recognition, it generates a correction signal to turn towards the unharvested side until the current cutting width reaches between two-thirds and three-quarters of the preset cutting width.

[0124] Accessible U-turn path planning based on the spatial envelope of the cutting platform, such as... Figure 5As shown, when the vehicle reaches the edge of the field, and the multi-line lidar detects an irregular ridge higher than the ground ahead, the adaptive decision control module 103 constructs a three-dimensional kinematic model of the harvester, including the length, width, and height of the cutting platform, and reconstructs a three-dimensional terrain model of the ridge using the high-density point cloud of the lidar. Under the constraint of satisfying the harvester's minimum turning radius, a turning trajectory is planned that ensures the harvester's three-dimensional kinematic model maintains a preset safe distance from the ridge's three-dimensional terrain model during movement, ensuring the cutting platform maintains a safe distance from the ridge surface at all times, achieving fully automatic and lossless lane changing.

[0125] In one specific embodiment of the invention, when the vehicle reaches the edge of a field, the 32-line radar detects an irregular field ridge 20cm above the ground. Traditional control might directly turn in place, causing the extended cutting platform to hit the ridge. This system calculates the three-dimensional bounding box of the cutting platform and plans a composite U-turn path of "reversing - small-angle entry - forward movement," ensuring that the cutting platform maintains a safe distance of at least 20cm from the ridge surface at all times, achieving fully automatic and lossless lane changing. A schematic diagram of the barrier-free U-turn path planning is shown below. Figure 5 The collision avoidance U-turn path planning results based on the three-dimensional spatial envelope of the cutting platform and the field ridge terrain model are shown in the form of a path planning diagram.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Those skilled in the art should understand that modifications can be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-source information fusion-based auxiliary driving control system for combine harvesters for nighttime emergency harvesting, characterized in that, It includes a multimodal perception module (101), an airborne computing terminal (102), and an adaptive decision control module (103). The multimodal sensing module (101) is installed at a high position at the front of the combine harvester (1) to collect data. The multimodal sensing module (101) includes a lidar, a binocular stereo camera, an infrared thermal imager, and a high-frequency inertial measurement unit. The lidar is used to collect three-dimensional geometric point clouds of the working environment. The binocular stereo camera is used to collect visible light texture images of the working environment. The infrared thermal imager is used to collect infrared thermal radiation images of the working environment. The high-frequency inertial measurement unit is used to collect six-axis vibration acceleration and attitude angular velocity data during the operation of the harvester. The airborne computing terminal (102) is communicatively connected to the multimodal perception module (101) and is configured to execute a spatiotemporal joint calibration and data fusion algorithm to generate an enhanced semantic point cloud map containing spatial coordinates, RGB color information and temperature information as an enhanced three-dimensional environment model. The adaptive decision control module (103) calculates the current working condition in real time based on the enhanced semantic point cloud map and sends control commands to the underlying actuator of the combine harvester. The control logic of the adaptive decision control module (103) includes: when a lodged crop area is detected, a first control command is output to control the chassis to reduce the driving speed and at the same time control the hydraulic system of the header to reduce the height of the header; when the boundary between the harvested and unharvested areas is detected, a second control command is output to adjust the steering angle according to the relative position of the header and the boundary to maintain the preset cutting width; when the field ridge is detected, a third control command is output to plan a collision-free turning path based on the three-dimensional spatial envelope of the header.

2. The multi-source information fusion-based combine harvester assisted driving and nighttime harvesting control system according to claim 1, characterized in that, The multimodal sensing module (101) is configured to perform the following steps to acquire data: Step S11: Perform intrinsic parameter calibration on the binocular stereo camera and the infrared thermal imager respectively, and obtain focal length, principal point coordinates, distortion coefficients and baseline parameters. Step S12: A multimodal universal calibration target is set up in the area where the lidar and vision co-exist in front of the harvester. With the vehicle body completely stationary, multiple sets of synchronous feature data are collected by changing the distance and deflection attitude of the calibration target in space. The initial static extrinsic parameter matrix of the lidar to the coordinate system of the binocular left camera and the initial static extrinsic parameter matrix of the lidar to the coordinate system of the infrared thermal imager are calculated. The multimodal universal calibration target includes a substrate layer, an active heating layer and a surface pattern layer. The surface pattern layer adopts a black and white checkerboard design. The black squares and white squares are made of two materials with different thermal emissivity, so that the calibration target presents a hot and cold checkerboard pattern in the infrared thermal imager that is consistent with the visible light image. Step S13: Based on the ROS environment, the data collected by the lidar, binocular stereo camera and infrared thermal imager are time-stamped and synchronized. The closest set of radar frames, visual frames and infrared frames within the time window are packaged into a synchronization data frame. Step S14: Downsample and filter out outliers from the synchronized point cloud data, and perform distortion correction on the binocular visible light image and infrared thermal image.

3. The multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system according to claim 2, characterized in that, The airborne computing terminal (102) is also used to perform dynamic online calibration of seismic compensation, including: Using the high-frequency vibration data acquired by the high-frequency inertial measurement unit, a dynamic displacement deviation model of the sensor support is established, and instantaneous attitude compensation is performed on the raw data of the lidar and the binocular stereo camera. During the harvester's operation, ground plane features in the point cloud and crop row edge features in the visual image are extracted in real time. By minimizing the geometric residual function, the real-time extrinsic matrix of the lidar to the binocular stereo camera is dynamically corrected. Extended Kalman filtering is used to smooth the corrected displacement matrix, and the real-time corrected dynamic extrinsic parameter matrix is ​​output to eliminate point cloud ghosting or layering caused by the severe shaking of the harvester.

4. The multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system according to claim 3, characterized in that, The airborne computing terminal (102) is configured to perform the following steps to generate an enhanced semantic point cloud map: Step S21: Traverse the original point cloud data collected by the lidar to obtain each 3D lidar point. , Obtain three-dimensional geometric information; Step S22: Using the real-time corrected dynamic extrinsic parameter matrix and the calibrated camera intrinsic parameter matrix, the laser point is projected... The image is projected onto the visible light image plane, and the RGB color components of the corresponding pixel positions are extracted to obtain RGB color information; Step S23: Using the real-time corrected dynamic extrinsic parameter matrix and the calibrated thermal imager intrinsic parameter matrix, the same... A laser point is projected onto an infrared image plane to obtain the grayscale value of the corresponding pixel position, and then converted into a physical temperature value according to the thermoradiative model to obtain temperature information. Step S24: Write the 3D geometric information, RGB color information, and temperature information into the same target point data structure to generate an enhanced semantic point cloud map. The data format is as follows: As an enhanced 3D environment model.

5. The multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system according to claim 1, characterized in that, The adaptive decision control module (103) is configured to perform the following steps to identify the collapsed area: In the enhanced semantic point cloud map, point cloud clusters with heights lower than a preset threshold for normal crop height are selected; By combining the infrared temperature characteristics and visible light texture characteristics corresponding to the point cloud clusters, and eliminating interference from bare soil and puddles, the remaining low-lying, high-density point cloud areas are identified as collapsed areas.

6. The multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system according to claim 1, characterized in that, The generation logic for the second control instruction is as follows: Extract crop elevation features from the enhanced semantic point cloud map and fit the edge curves of unharvested crops; Calculate the lateral deviation between the outermost tangent point of the harvester header and the edge curve; When the lateral deviation value indicates that the current cutting width is less than half of the preset cutting width and it is not the final stage, a correction signal is generated to turn towards the unharvested side until the current cutting width reaches between two-thirds and three-quarters of the preset cutting width.

7. The multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system according to claim 1, characterized in that, The generation logic of the third control command is as follows: A three-dimensional terrain model of field ridges was reconstructed using high-density point clouds from lidar. Construct a three-dimensional kinematic model of the harvester, including the length, width, and height of the header. Under the constraint of satisfying the minimum turning radius of the harvester, a turning trajectory is planned that ensures that the three-dimensional kinematic model of the harvester maintains a safe distance from the three-dimensional terrain model of the field ridge during the movement.

8. The multi-source information fusion-based combine harvester assisted driving nighttime harvesting control system according to claim 1, characterized in that, It also includes remote mobile monitoring terminals; The remote mobile monitoring terminal is connected to the airborne computing terminal (102) via a wireless network to display the visualization rendering results of the enhanced semantic point cloud map in real time, and has human-computer interaction functions such as remotely issuing work tasks, modifying work parameter thresholds, and performing emergency shutdown.

9. A control method for a combine harvester assisted driving nighttime harvesting control system based on multi-source information fusion according to any one of claims 1 to 8, characterized in that, Includes the following steps: Step S1: The multimodal sensing module (101) synchronously collects lidar point cloud data, binocular vision image data and infrared thermal imaging data of the environment in front of the harvester. Step S2: The airborne computing terminal (102) performs spatiotemporal registration and fusion of multi-source data, and constructs an enhanced three-dimensional environment model with temperature and texture information based on the enhanced semantic point cloud map; Step S3: The adaptive decision control module (103) performs fall detection, boundary tracking and obstacle recognition in parallel based on the enhanced three-dimensional environment model. Step S4: If crop lodging is detected, the adaptive decision control module (103) automatically calculates the target operating speed and target header height based on the lodging area and degree, and drives the harvester to decelerate and lower the header. Step S5: If the harvesting is in progress, the crop boundary line is detected in real time by the adaptive decision control module (103), and the cutting platform is kept at the optimal operating width by lateral correction control. Step S6: If a field ridge is detected ahead, the adaptive decision control module (103) automatically generates and executes an anti-collision turning maneuver strategy based on the geometric shape of the field ridge and the spatial position of the cutting platform.

10. The control method of the multi-source information fusion combine harvester assisted driving night harvesting control system according to claim 9, characterized in that, In step S3, infrared thermal imaging data is used to assist in nighttime identification. By using the difference in thermal inertia between crops and the abiotic environment, upright crops, lodged crops, and field obstacles can be distinguished under low light conditions.

Images