Intelligent navigation method, device, equipment and medium for agricultural machinery equipment

Abstract

This application relates to an intelligent navigation method, device, equipment, and medium for agricultural machinery. The method includes: acquiring lidar point cloud data, camera image data, inertial measurement unit (IMU) data, and global navigation satellite system (GNSS) positioning data of the agricultural machinery; obtaining a first pose increment based on the lidar point cloud data and IMU data; obtaining a second pose increment based on the camera image data and IMU data; fusing the first and second pose increments to obtain a local pose; fusing the local pose and GNSS positioning data to obtain a global pose; determining the environmental semantic information of the agricultural machinery; and generating a navigation path for the agricultural machinery based on the global pose and the environmental semantic information. The navigation path is used to instruct the agricultural machinery to navigate, thereby improving the accuracy of the navigation path generation.

Description

Technical Field

[0001] This application relates to the field of intelligent navigation technology for agricultural machinery, and in particular to an intelligent navigation method, device, computer equipment, and computer-readable storage medium for agricultural machinery. Background Technology

[0002] As agricultural production moves towards infrastructure-based and intelligent operations, the operating scenarios for agricultural machinery are no longer limited to completely open outdoor farmland, but are gradually expanding to semi-enclosed or enclosed environments such as greenhouses, seedling sheds, storage areas, and agricultural product processing and transshipment areas. In these scenarios, agricultural machinery often needs to frequently switch between outdoor farmland and indoor or semi-indoor environments.

[0003] In related technologies, navigation path planning is based on Global Navigation Satellite System (GNSS) positioning data. However, GNSS positioning data is prone to signal attenuation, interruption, or accuracy reduction in indoor or obstructed environments, resulting in low accuracy of the generated navigation path and affecting the stability of navigation and operation control of agricultural machinery and equipment. Summary of the Invention

[0004] Therefore, it is necessary to provide an intelligent navigation method, device, computer equipment, and computer-readable storage medium for agricultural machinery that can improve the accuracy of navigation path generation, in order to address the aforementioned technical problems.

[0005] Firstly, this application provides an intelligent navigation method for agricultural machinery, the method comprising:

[0006] Acquire lidar point cloud data, camera image data, inertial measurement unit data, and global satellite navigation system positioning data from agricultural machinery and equipment;

[0007] The first pose increment is obtained based on lidar point cloud data and inertial measurement unit data; the second pose increment is obtained based on camera image data and inertial measurement unit data; the first pose increment and the second pose increment are fused to obtain the local pose of the agricultural machinery; the local pose is the pose of the agricultural machinery in the local coordinate system.

[0008] The local pose and global satellite navigation system positioning data are fused to obtain the global pose of agricultural machinery and equipment; the global pose is the pose of agricultural machinery and equipment in the global coordinate system.

[0009] Determine the environmental semantic information of agricultural machinery and equipment;

[0010] Based on global pose and environmental semantic information, a navigation path is generated for agricultural machinery and equipment; the navigation path is used to instruct the agricultural machinery and equipment to navigate.

[0011] In one embodiment, the first pose increment and the second pose increment are fused to obtain the local pose of the agricultural machinery, including:

[0012] Acquire point cloud registration residuals, number of point cloud matching points, number of point cloud feature points, photometric error, motion intensity information of inertial measurement unit, and continuous pre-integration of inertial measurement unit;

[0013] By using the attention weight allocation network, based on the first pose increment, point cloud registration residual, number of point cloud matching points, second pose increment, number of point cloud feature points, photometric error, and motion intensity information of the inertial measurement unit, the first weight coefficient corresponding to the first pose increment and the second weight coefficient corresponding to the second pose increment are obtained.

[0014] Based on the first weighting coefficient and the second weighting coefficient, the first pose increment and the second pose increment are weighted and summed to obtain the fused pose increment.

[0015] The fused pose increment and the inertial measurement unit are continuously pre-integrated and input to the Kalman filter to obtain the local pose.

[0016] In one embodiment, the local pose and global navigation satellite system positioning data are fused to obtain the global pose of the agricultural machinery, including:

[0017] Transform the local pose from the local coordinate system to the global coordinate system to obtain the predicted pose;

[0018] Calculate the residual between the global navigation satellite system positioning data and the predicted pose to obtain the pose residual;

[0019] Calculate the set of quality indicators for global navigation satellite system positioning data;

[0020] If each quality indicator in the quality indicator set meets the corresponding preset indicator threshold and the pose residual is less than the preset residual threshold, the observation noise covariance of the global satellite navigation system positioning data and the local pose covariance of the local pose are obtained. The observation noise covariance and the local pose covariance are input into the gating network to obtain the third weight coefficient corresponding to the global satellite navigation system positioning data and the fourth weight coefficient corresponding to the pose residual.

[0021] The global pose is obtained by weighting and summing the positioning data and pose residuals of the global satellite navigation system based on the third and fourth weighting coefficients.

[0022] In one embodiment, determining the environmental semantic information of agricultural machinery includes:

[0023] Target object detection is performed on camera image data to obtain the target object region;

[0024] The LiDAR point cloud data is projected onto the image coordinate system of the camera image data, and the projected LiDAR point cloud data is associated with the target object area to obtain environmental semantic information; or...

[0025] The camera image data is input into the depth estimation model to obtain a depth map; based on the depth map and the camera's intrinsic parameters, simulated point cloud data is obtained; the simulated point cloud data is projected onto the image coordinate system where the camera image data is located, and the projected simulated point cloud data is associated with the target object region to obtain environmental semantic information.

[0026] In one embodiment, a navigation path for agricultural machinery is generated based on global pose and environmental semantic information, including:

[0027] Based on environmental semantic information, a semantic cost map is constructed; the semantic cost map represents the cost required for agricultural machinery to move to different locations;

[0028] Based on the semantic cost map, the target passable area is obtained; the target passable area is a passable area with a cost less than a preset threshold and continuous distribution.

[0029] Based on the semantic label information of the semantic cost map, the field boundary area and the inter-row gap area are identified from the target traversable area;

[0030] Extract the center passage lines of the field ridge boundary area and the inter-row gap area to obtain multiple channels;

[0031] Based on the positional relationship between agricultural machinery and equipment and each channel, determine the transition path between each channel;

[0032] Based on each channel and transition path, a navigation path for agricultural machinery and equipment is generated.

[0033] In one embodiment, after generating the navigation path for the agricultural machinery, the method further includes:

[0034] Based on the navigation path and global pose, generate motion control commands;

[0035] According to motion control commands, control agricultural machinery and equipment to move along the navigation path.

[0036] In one embodiment, after acquiring lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data of agricultural machinery and equipment, the method further includes:

[0037] Add timestamps to lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data;

[0038] Based on the timestamp, the lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data are time-aligned.

[0039] Secondly, this application also provides an intelligent navigation device for agricultural machinery, the device comprising:

[0040] The data acquisition module is used to acquire lidar point cloud data, camera image data, inertial measurement unit data, and global satellite navigation system positioning data from agricultural machinery and equipment.

[0041] The local pose acquisition module is used to obtain a first pose increment based on lidar point cloud data and inertial measurement unit data; to obtain a second pose increment based on camera image data and inertial measurement unit data; and to fuse the first pose increment and the second pose increment to obtain the local pose of the agricultural machinery equipment; the local pose is the pose of the agricultural machinery equipment in the local coordinate system.

[0042] The global pose acquisition module is used to fuse local pose and global satellite navigation system positioning data to obtain the global pose of agricultural machinery and equipment; the global pose is the pose of agricultural machinery and equipment in the global coordinate system.

[0043] The environmental semantic information determination module is used to determine the environmental semantic information of agricultural machinery and equipment;

[0044] The navigation path generation module is used to generate navigation paths for agricultural machinery based on global pose and environmental semantic information; the navigation path is used to instruct the agricultural machinery to navigate.

[0045] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of the first aspect.

[0046] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps of the first aspect.

[0047] The aforementioned intelligent navigation method, device, computer equipment, and computer-readable storage medium for agricultural machinery acquire LiDAR point cloud data, camera image data, inertial measurement unit (IMU) data, and Global Navigation Satellite System (GNSS) positioning data from the agricultural machinery. Based on the LiDAR point cloud data and IMU data, a first pose increment is obtained. Based on the camera image data and IMU data, a second pose increment is obtained. The first and second pose increments are fused to obtain the local pose of the agricultural machinery; the local pose is the pose of the agricultural machinery in a local coordinate system. The local pose and GNSS positioning data are fused to obtain the global pose of the agricultural machinery; the global pose is the pose of the agricultural machinery in a global coordinate system. Environmental semantic information of the agricultural machinery is determined. Based on the global pose and environmental semantic information, a navigation path for the agricultural machinery is generated; the navigation path is used to instruct the agricultural machinery to navigate. As can be seen from the above, this application, by fusing local pose and global navigation satellite system (GNSS) positioning data, can avoid the global pose being skewed by short-term erroneous GNSS positioning data in the event of signal quality fluctuations, short-term anomalies, or indoor / outdoor scene switching of GNSS positioning data. Furthermore, when GNSS positioning data is reliable, it can effectively suppress the cumulative drift of local pose, thereby improving the generation accuracy of global pose and thus improving the generation accuracy of navigation path. Attached Figure Description

[0048] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0049] Figure 1 This is an application environment diagram of an intelligent navigation method for agricultural machinery in one embodiment;

[0050] Figure 2 This is a flowchart illustrating an intelligent navigation method for agricultural machinery in one embodiment;

[0051] Figure 3 This is a schematic diagram of the process for obtaining a local pose in one embodiment;

[0052] Figure 4 This is a schematic diagram of the process for obtaining the global pose in one embodiment;

[0053] Figure 5 This is a timing diagram of the interaction between a remote user, a terminal, and agricultural machinery in one embodiment.

[0054] Figure 6 This is a structural block diagram of an intelligent navigation device for agricultural machinery in one embodiment;

[0055] Figure 7 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0057] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0058] The intelligent navigation method for agricultural machinery provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 acquires lidar point cloud data, camera image data, inertial measurement unit (IMU) data, and GPS positioning data from agricultural machinery. Based on the lidar point cloud data and IMU data, it obtains a first pose increment; based on the camera image data and IMU data, it obtains a second pose increment; it fuses the first and second pose increments to obtain the local pose of the agricultural machinery; the local pose is the pose of the agricultural machinery in a local coordinate system; it fuses the local pose and GPS positioning data to obtain the global pose of the agricultural machinery; the global pose is the pose of the agricultural machinery in a global coordinate system; it determines the environmental semantic information of the agricultural machinery; and based on the global pose and environmental semantic information, it generates a navigation path for the agricultural machinery; the navigation path is used to instruct the agricultural machinery to navigate. The terminal can be a control unit within the agricultural machinery or a remote control unit of the agricultural machinery. Server 104 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server that provides cloud computing services.

[0059] In one embodiment, such as Figure 2 As shown, an intelligent navigation method for agricultural machinery is provided. This embodiment applies this method to... Figure 1 Taking terminal 102 as an example, the method includes the following steps:

[0060] Step S210: Acquire lidar point cloud data, camera image data, inertial measurement unit data, and global satellite navigation system positioning data of agricultural machinery and equipment.

[0061] Agricultural machinery and equipment include, but are not limited to, seeding machinery and equipment, plant protection machinery and equipment, and sprinkler irrigation machinery and equipment.

[0062] Among them, LiDAR point cloud data refers to the three-dimensional point cloud data of the environment surrounding the device acquired through LiDAR (Light Detection and Ranging).

[0063] Among them, camera image data refers to ambient visible light images or video sequences acquired through camera acquisition devices.

[0064] Among them, inertial measurement unit data refers to the angular velocity and angular velocity of the equipment collected by the inertial measurement unit (IMU), which are used to calculate the attitude changes and short-term motion state of the equipment.

[0065] Global Navigation Satellite System (GNSS) positioning data refers to the absolute position information of the device in the world coordinate system, which is collected by the GNSS.

[0066] In this embodiment, the output data of each sensor is acquired in real time through a data interface or bus installed on the terminal device. These sensors include a lidar unit, a camera, an inertial measurement unit, and a global navigation satellite system receiver.

[0067] Step S220: Obtain the first pose increment based on the lidar point cloud data and the inertial measurement unit data; obtain the second pose increment based on the camera image data and the inertial measurement unit data; fuse the first pose increment and the second pose increment to obtain the local pose of the agricultural machinery equipment; the local pose is the pose of the agricultural machinery equipment in the local coordinate system.

[0068] Among them, the local coordinate system refers to the coordinate system established with agricultural machinery and equipment as the center, which changes as the agricultural machinery and equipment moves.

[0069] In this embodiment, LiDAR-Inertial Odometry (LIO) can be used to fuse LiDAR point cloud data and IMU inertial data to obtain the first pose increment. LiDAR-Inertial Odometry is a method for estimating the continuous motion pose of a device by tightly coupling and fusing LiDAR point cloud data with IMU inertial data.

[0070] Specifically, the LiDAR point cloud data of the current frame is registered with a local sparse map to obtain the first estimated pose increment of the agricultural machinery. The local sparse map is constructed based on historical LiDAR point cloud data. During the time interval between two frames of LiDAR point cloud data, the inertial measurement unit data is integrated to obtain the first initial pose increment of the agricultural machinery. A Kalman filter or smoothing optimization method is used to tightly couple the first estimated pose increment and the first initial pose increment of the agricultural machinery to obtain the first pose increment.

[0071] Visual-Inertial Odometry (VIO) can be used to estimate the pose changes of agricultural machinery by fusing camera image data and inertial measurement unit (IMU) data. VIO is a method for calculating odometer position by fusing camera image information with IMU inertial data.

[0072] Specifically, within the time interval between two frames of image data, the inertial measurement unit data is integrated to obtain the second initial pose increment of the agricultural machinery. Feature points of the local sparse map are projected onto the camera image data to obtain projection points. The luminance (pixel intensity) of the image region surrounding the projection points is compared with the luminance information associated with the feature points of the local sparse map to obtain the luminance error. By minimizing the luminance error, the pose is estimated to obtain the second estimated pose increment of the agricultural machinery. A Kalman filter or smoothing optimization method is used to tightly couple the second estimated pose increment and the second initial pose increment of the agricultural machinery to obtain the second pose increment.

[0073] Weighting coefficients are assigned to the first pose increment and the second pose increment. Based on the assigned weighting coefficients, the first pose increment and the second pose increment are fused to obtain the local pose of the agricultural machinery equipment.

[0074] Step S230: The local pose and the global satellite navigation system positioning data are fused to obtain the global pose of the agricultural machinery and equipment; the global pose is the pose of the agricultural machinery and equipment in the global coordinate system.

[0075] The global coordinate system refers to an absolute reference system fixed to the environment or the world.

[0076] In this embodiment of the application, considering that the estimation of local pose has cumulative error and will produce local pose drift after a long time, the pose of agricultural machinery and equipment can be corrected by fusing local pose and global satellite navigation system positioning data to obtain a drift-free global pose estimate.

[0077] Step S240: Determine the environmental semantic information of agricultural machinery and equipment.

[0078] Among them, environmental semantic information includes, but is not limited to, obstacles in the environment, passable areas, and the location information of the target object.

[0079] In this embodiment of the application, target detection can be performed on camera image data and lidar point cloud data to obtain environmental semantic information of agricultural machinery and equipment.

[0080] Step S250: Generate a navigation path for agricultural machinery based on global pose and environmental semantic information; the navigation path is used to instruct the agricultural machinery to navigate.

[0081] In this embodiment of the application, a path planning algorithm can be used to automatically generate navigation paths for agricultural machinery based on global pose and environmental semantic information.

[0082] The aforementioned intelligent navigation method for agricultural machinery involves acquiring lidar point cloud data, camera image data, inertial measurement unit (IMU) data, and global navigation satellite system (GNSS) positioning data from the agricultural machinery. Based on the lidar point cloud data and IMU data, a first pose increment is obtained. Based on the camera image data and IMU data, a second pose increment is obtained. The first and second pose increments are fused to obtain the local pose of the agricultural machinery, which is the pose of the agricultural machinery in a local coordinate system. The local pose and GNSS positioning data are then fused to obtain the global pose of the agricultural machinery, which is the pose of the agricultural machinery in a global coordinate system. Environmental semantic information of the agricultural machinery is determined. Based on the global pose and environmental semantic information, a navigation path for the agricultural machinery is generated. The navigation path is used to instruct the agricultural machinery to navigate. As can be seen from the above, this application, by fusing local pose and global navigation satellite system (GNSS) positioning data, can avoid the global pose being skewed by short-term erroneous GNSS positioning data in the event of signal quality fluctuations, short-term anomalies, or indoor / outdoor scene switching of GNSS positioning data. Furthermore, when GNSS positioning data is reliable, it can effectively suppress the cumulative drift of local pose, thereby improving the generation accuracy of global pose and thus improving the generation accuracy of navigation path.

[0083] In one embodiment, after acquiring lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data of agricultural machinery and equipment, the method further includes:

[0084] Step S211: Add timestamps to the lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data;

[0085] Step S212: Time-align the lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data according to the timestamp.

[0086] In this embodiment, since different sensors have different sampling frequencies and timestamp bases, time alignment of multi-sensor data is required to ensure data fusion consistency. Specifically, the terminal device can use a high-precision clock or GPS timing as a global time base to add accurate timestamps to camera image data, LiDAR point cloud data, inertial measurement unit data, and global navigation satellite system positioning data. The data collected by the inertial measurement unit, LiDAR, and camera are all hard-triggered sampling via a clock signal generated by the industrial control board. Simultaneously, the industrial control board sends a trigger signal to the system to synchronize the sampling time of the sensor data; that is, the data uses the trigger time of the sampling signal as the corresponding timestamp. Data from different sensors is buffered, interpolated, or discarded based on the timestamps to match sensor information within the same time window.

[0087] In practical applications, after receiving camera image data, the attitude change at the time of image acquisition is estimated by finding the inertial measurement unit data closest to that timestamp and interpolating them. Then, the LiDAR point cloud data at the corresponding time is subjected to motion correction (removing distortions caused by motion during scanning) to obtain LiDAR point cloud data at the same time and in the same coordinate system as the image.

[0088] The embodiments of this application can ensure the effectiveness of subsequent sensor data fusion by continuously monitoring the data streams of each sensor and performing real-time time alignment.

[0089] In one embodiment, such as Figure 3 As shown, the first pose increment and the second pose increment are fused to obtain the local pose of the agricultural machinery, including:

[0090] Step S310: Obtain point cloud registration residual, number of point cloud matching points, number of point cloud feature points, photometric error, motion intensity information of inertial measurement unit, and continuous pre-integration of inertial measurement unit.

[0091] Among them, point cloud registration residual refers to the difference between the actual distance calculated for each matching point pair and the expected distance (usually zero) when the pose transformation parameters are optimized by the least squares method during the point cloud registration process.

[0092] Among them, the number of point cloud matching points refers to the number of point pairs that successfully find corresponding relationships in point cloud registration.

[0093] The number of feature points in a point cloud refers to the number of feature points in a local sparse map.

[0094] Among them, photometric error refers to the difference between the actual observed value and the predicted value of the image pixel brightness.

[0095] Among them, the motion intensity information of the inertial measurement unit refers to the acceleration and angular velocity data measured by the IMU, which reflects the motion intensity of the object (such as rapid acceleration and sharp turns).

[0096] Among them, continuous pre-integration of the inertial measurement unit refers to continuously integrating the acceleration and angular velocity data of the IMU between two keyframes to obtain the estimated value of the relative pose transformation (rotation, translation).

[0097] In this embodiment, when obtaining the first pose increment, it is necessary to perform point cloud registration between the current frame's LiDAR point cloud data and the local sparse map. The point cloud registration residual and the number of matched points can be obtained from the point cloud registration process. When obtaining the second pose increment, feature point projection from the local sparse map is required, from which the number of feature points and photometric error can be obtained. Continuous integration is performed on the IMU's acceleration and angular velocity data to obtain the inertial measurement unit's continuous pre-integration.

[0098] Step S320: Through the attention weight allocation network, based on the first pose increment, point cloud registration residual, number of point cloud matching points, second pose increment, number of point cloud feature points, photometric error, and motion intensity information of the inertial measurement unit, the first weight coefficient corresponding to the first pose increment and the second weight coefficient corresponding to the second pose increment are obtained.

[0099] The sum of the first weighting coefficient and the second weighting coefficient is 1.

[0100] In this embodiment, the first pose increment, point cloud registration residual, number of matching points in the point cloud, second pose increment, number of feature points in the point cloud, photometric error, and motion intensity information of the inertial measurement unit are normalized and extracted through an embedding layer to obtain a feature vector of uniform scale. The feature vector is then input into an attention weight allocation network, which outputs a first weight coefficient and a second weight coefficient.

[0101] Step S330: Based on the first weighting coefficient and the second weighting coefficient, the first pose increment and the second pose increment are weighted and summed to obtain the fused pose increment.

[0102] In this embodiment, the first weighting coefficient is multiplied by the first pose increment to obtain a first product result. The second weighting coefficient is multiplied by the second pose increment to obtain a second product result. The first product result and the second product result are added together to obtain the fused pose increment.

[0103] Step S340: The fused pose increment and the inertial measurement unit are continuously pre-integrated and input to the Kalman filter to obtain the local pose.

[0104] The Kalman filter can be an Error-State Iteration Kalman Filter (ESIKF). ESIKF achieves high-precision estimation of the state of a nonlinear system by iteratively updating the error state.

[0105] In this embodiment of the application, the Kalman filter calculates the pose increment at the current moment based on the continuous pre-integration of the inertial measurement unit at the previous moment, and obtains the local pose at the current moment based on the fused pose increment at the current moment and the pose increment at the current moment.

[0106] In one embodiment, such as Figure 4 As shown, the global pose of agricultural machinery is obtained by fusing local pose and global navigation satellite system positioning data, including:

[0107] Step S410: Transform the local pose from the local coordinate system to the global coordinate system to obtain the predicted pose;

[0108] Step S420: Calculate the residual between the global satellite navigation system positioning data and the predicted pose to obtain the pose residual;

[0109] Step S430: Calculate the set of quality indicators for global navigation satellite system positioning data;

[0110] Step S440: If each quality index in the quality index set meets the corresponding preset index threshold and the pose residual is less than the preset residual threshold, obtain the observation noise covariance of the global satellite navigation system positioning data and the local pose covariance of the local pose. Input the observation noise covariance and the local pose covariance into the gating network to obtain the third weight coefficient corresponding to the global satellite navigation system positioning data and the fourth weight coefficient corresponding to the pose residual.

[0111] Step S450: Based on the third and fourth weighting coefficients, the global satellite navigation system positioning data and pose residuals are weighted and summed to obtain the global pose.

[0112] In the Kalman filter, the output local pose covariance is a key parameter describing the uncertainty of pose estimation, reflecting the degree of confidence of the Kalman filter in the current pose estimation.

[0113] Among them, the observation noise covariance of GNSS positioning data is a key parameter describing the uncertainty of GNSS observations. The main diagonal elements represent the variance of each observation (such as pseudorange and carrier phase), describing its dispersion, while the off-diagonal elements are the covariance, describing the linear correlation between different observations.

[0114] The quality index set is used to evaluate the reliability of global satellite navigation system positioning data. The quality index set includes multiple quality indicators, but is not limited to the number of satellites, DOP value, RT fixed / floating state, carrier-to-noise ratio, positioning solution type, and velocity / heading stability.

[0115] Each quality indicator has a corresponding preset threshold. For example, the preset threshold for the number of satellites is 4, and the preset threshold for the carrier-to-noise ratio is 40dB-Hz.

[0116] In this embodiment of the application, since the positioning data of the Global Navigation Satellite System may have large short-term errors due to occlusion, multipath, antenna attitude change or differential lock loss, and the estimation of local pose has cumulative drift over a long period of time, in order to address these two types of errors, the local pose and the positioning data of the Global Navigation Satellite System are fused to ensure that the global pose is not skewed by the erroneous GNSS positioning data and to suppress the divergence of local pose drift.

[0117] Specifically, the local pose is aligned with the global navigation satellite system (GNSS) positioning data by timestamp. When the frequencies of the two are different or there is a delay, the local pose is interpolated or the GNSS positioning data is compensated for the delay to ensure they correspond at the same time. A coordinate transformation is then performed on the local pose to obtain the predicted pose in the world coordinate system.

[0118] Calculate the pose residual between the predicted pose and the GNSS positioning data. If the pose residual is greater than or equal to a preset residual threshold (which can be adaptively set based on local pose covariance), the GNSS positioning data is determined to be abnormal. If at least one quality indicator in the quality indicator set does not meet the corresponding preset indicator threshold, the GNSS positioning data is determined to be abnormal. When GNSS positioning data is abnormal, the predicted pose can be used as the global pose.

[0119] If the pose residual is less than a preset residual threshold, and all quality indicators in the quality indicator set meet their corresponding preset thresholds, the observation noise covariance is calculated for the GNSS positioning data, and the local pose covariance based on the local pose output is obtained through Kalman filtering. Based on a gating network, weight coefficients are assigned to the observation noise covariance and the local pose covariance to obtain the third weight coefficient corresponding to the GNSS positioning data and the fourth weight coefficient corresponding to the pose residual.

[0120] Multiply the third weighting coefficient by the global navigation satellite system positioning data to obtain the third product. Multiply the fourth weighting coefficient by the pose residual to obtain the fourth product. Summate the third and fourth products to obtain the global pose.

[0121] In one embodiment, determining the environmental semantic information of agricultural machinery includes:

[0122] Step S241: Detect the target object in the camera image data to obtain the target object region.

[0123] The target area includes, but is not limited to, obstacle areas and passable areas.

[0124] In this embodiment, camera image data can be input into a target object detection model to obtain the target object region. The target object detection model can be the SAM3 model (Segment Anything Model 3). The SAM3 model is a general-purpose vision-based model capable of pixel-level segmentation of any target in an image or video based on input prompts.

[0125] For example, a user can suggest a "yellow school bus" or provide an example of a certain type of obstacle to the model, and the SAM3 model will find all instances in the image and provide a pixel-level mask.

[0126] The SAM3 model supports segmentation based on concept cues in both images and videos, and can even track multiple target instances simultaneously. In practical applications, concept cues such as "pedestrians," "vehicles," "drivable roads," "buildings," and "machinery" are pre-set for typical obstacles and regions, allowing the model to segment the image and obtain the pixel regions of these elements. Furthermore, for specific objects (such as fruit trees or weeds in agriculture, or shelf labels in warehousing), corresponding concepts / examples can be input for the model to recognize these objects. Because the recognition targets can be customized according to the needs of different applications without retraining the model, only the cues need to be changed, the efficiency of object detection is greatly improved.

[0127] Step S242: Project the LiDAR point cloud data onto the image coordinate system where the camera image data is located, and associate the projected LiDAR point cloud data with the target object area to obtain environmental semantic information; or,

[0128] In this embodiment, LiDAR point cloud data is projected onto the image coordinate system of a camera and associated with a mask region (target object region) generated by a target object detection model. This assigns a semantic label to each point in the LiDAR point cloud data, obtaining environmental semantic information. For example, points belonging to the "obstacle" mask region are marked as obstacle points, and points belonging to the "ground" or "road" region are marked as passable ground points. This allows for the separation of the 3D point set of obstacles and the point set of open, passable areas from the LiDAR point cloud data.

[0129] Step S243: Input the camera image data into the depth estimation model to obtain a depth map; obtain simulated point cloud data based on the depth map and the camera's intrinsic parameters; project the simulated point cloud data onto the image coordinate system where the camera image data is located, and associate the projected simulated point cloud data with the target object region to obtain environmental semantic information.

[0130] The depth estimation model can be the Depth Anything 3 model, which is a depth estimation model that can predict scene depth information from single or multiple images and is used to generate spatial structure information when real ranging sensors are lacking.

[0131] In this embodiment, when LiDAR point cloud data is missing, a depth estimation model is invoked to estimate the depth of the camera image data, obtaining a depth map. Combined with the camera's intrinsic parameters, the depth map is reconstructed into a point cloud, obtaining simulated point cloud data. This simulated point cloud data is projected onto the camera's image coordinate system and associated with the mask region (target object region) generated by the target object detection model, thereby assigning a semantic label to each point in the simulated point cloud data and obtaining environmental semantic information.

[0132] In an optional embodiment, when camera image data is missing, environmental semantic information can be obtained directly based on LiDAR point cloud data. For example, outliers in the point cloud can be identified as obstacles using clustering algorithms, and flat ground and non-ground areas can be distinguished using ground extraction algorithms (such as RANSAC plane fitting).

[0133] In one embodiment, a navigation path for agricultural machinery is generated based on global pose and environmental semantic information, including:

[0134] Step S251: Construct a semantic cost map based on environmental semantic information; the semantic cost map represents the cost required for agricultural machinery to move to different locations.

[0135] Semantic cost maps are an environmental representation that incorporates semantic information into traditional cost maps. They are used to describe the access costs, obstacle attributes, and operational constraints in different areas. Traditional cost maps divide the environment into grids or voxels and assign access costs to each cell, and are used for path planning and obstacle avoidance decisions.

[0136] In this embodiment, the semantic cost map uses three-dimensional voxels as representations to mark obstacles and passable areas into corresponding grid cells. Specifically, for the location of an identified obstacle, the corresponding grid cell is assigned a high-cost or impassable marker. For freely passable areas, a low-cost marker is assigned. For unknown or unexplored areas, a neutral or unexplored marker can be assigned.

[0137] Optionally, if the environmental semantic information provides richer semantics (such as different obstacle types, dynamic obstacle prediction of movement, etc.), the semantic cost map can also be dynamically updated accordingly. For example, the grid state of moving obstacles (pedestrians, vehicles, etc.) can be changed over time, and specific dangerous areas (cliffs, water bodies) can be assigned an infinitely high cost to absolutely prohibit entry.

[0138] Optionally, data from sensors can be fused to perform low-level updates to the semantic cost map. For example, new LiDAR point cloud data can be used to update the occupied grid, with grids hit by the point cloud marked as occupied and areas not hit gradually marked as free or unknown. This ensures that even objects that the aforementioned object detection model fails to identify (such as very small obstacles) can still be represented in the map, and the occupied grid can provide a low-level safety supplement.

[0139] Optionally, the semantic cost map is maintained as a local area centered on the robot's current location (e.g., a range of several tens of meters in front), and is continuously updated as the agricultural machinery moves, thus limiting the map size and detail. Simultaneously, historical information is periodically decayed; that is, the confidence level is reduced for obstacle data that has been outside the sensor's perception range for an extended period, to prevent residual ghosts of moving obstacles from affecting subsequent navigation path generation.

[0140] Step S252: Obtain the target passable area based on the semantic cost map; the target passable area is a passable area with a cost less than a preset threshold and continuous distribution.

[0141] Step S253: Based on the semantic label information of the semantic cost map, identify the field boundary area and the inter-row gap area from the target passable area.

[0142] The preset threshold can be set according to actual needs.

[0143] In this embodiment of the application, the semantic cost map not only includes geometric information of obstacles and passable areas, but also clearly marks agricultural semantic elements such as crop bodies, field ridge boundaries and passable areas between rows.

[0144] By screening for low-cost and continuously distributed passable areas, and combining semantic tags to prioritize areas identified as inter-row gaps or ridge passages, we can avoid entering crop growing areas or high-risk zones.

[0145] Step S254: Extract the center passage lines of the field ridge boundary area and the inter-row gap area to obtain multiple channels.

[0146] In this embodiment of the application, for passable areas that are distributed in strips, the geometric shape of the area is analyzed, and its central passage line is extracted as the basic path for agricultural machinery to travel, thereby forming a set of channels that are consistent with the direction of the field ridges and are parallel or approximately parallel to each other.

[0147] Step S255: Determine the transition path between each channel based on the positional relationship between the agricultural machinery and each channel.

[0148] In this embodiment of the application, based on the generation of each channel, the channels are further automatically sorted according to the current position of the agricultural machinery and the spatial relationship between each channel, and a transition path between channels is generated, so that the agricultural machinery can smoothly switch from one channel to the next channel while meeting its own turning radius and width constraints.

[0149] Step S256: Generate navigation paths for agricultural machinery and equipment based on each channel and transition path.

[0150] In the embodiments of this application, one or more continuous navigation paths can be automatically formed based on each channel and the transition path between channels.

[0151] In one embodiment, after generating the navigation path for the agricultural machinery, the method further includes:

[0152] Step S260: Generate motion control commands based on the navigation path and global pose;

[0153] Step S270: Control the agricultural machinery to move along the navigation path according to the motion control command.

[0154] In the embodiments of this application, a closed-loop control algorithm, such as PID control or model predictive control (MPC), is used to calculate the steering and acceleration / deceleration commands required by the agricultural machinery based on the navigation path and global pose, so that the agricultural machinery gradually approaches and follows the navigation path.

[0155] In practical applications, taking the commonly used method "pure tracking" as an example, a "target point" is selected ahead of the navigation path. Then, the steering angle to be set is calculated using geometric relationships, causing the agricultural machinery to move towards the target point. The steering control output is typically calculated based on the speed difference between the steering servo or differential wheel; speed control adjusts the throttle or motor output by comparing the current speed with the planned desired speed. PID loops can be used to smoothly track speed and direction commands, reducing overshoot and oscillations.

[0156] To facilitate understanding of the above method embodiments, an intelligent navigation system for agricultural machinery is provided. The terminal runs this intelligent navigation system, which includes a sensor acquisition module, a time synchronization module, a local fusion positioning module, a global fusion positioning module, a multimodal recognition module, a semantic cost map module, a path planning module, a control module, a communication module, and a task scheduling module. Each module interacts with data through clearly defined interfaces, forming a complete closed loop from perception and positioning to decision-making and execution. Throughout the process, sensor data is fused to obtain high-precision positioning results and environmental understanding, which are then used to plan safe and efficient navigation paths. The control module executes path tracking to achieve autonomous movement and operation.

[0157] Specifically, the sensor acquisition module includes various sensors (such as cameras, lidar, inertial measurement units, and global navigation satellite systems) to collect environmental and self-motion data. After timestamp calibration by the time synchronization module, these data are sent in parallel for subsequent processing. The local fusion positioning module uses lidar inertial odometry and visual inertial odometry to calculate the local pose (position and orientation) of the agricultural machinery relative to its environment in real time. The global fusion positioning module combines GNSS positioning data to correct accumulated errors in local pose and obtain the global pose. The multimodal recognition module fuses camera image data and lidar point cloud data to identify surrounding obstacles, passable areas, and specific operational targets, among other environmental semantic information. The semantic cost map module maps environmental semantic information to a semantic cost map, representing the passability cost of different areas in the environment and planning for agricultural tasks. The path planning module calculates an obstacle-avoiding and efficient navigation path in the established global coordinate system based on the semantic cost map and the task objective. Finally, the control module receives the planned navigation path and global pose, and converts them into control commands for the actuators of the agricultural machinery (including drive motors, steering mechanisms, and work execution devices), driving the agricultural machinery to move along the navigation path. Simultaneously, the task scheduling module is responsible for triggering the path planning module to plan new navigation paths based on pre-set task queues or external commands (received via the communication module), and reporting the task status to the remote end via the communication module after the task is completed.

[0158] like Figure 5 The diagram illustrates an interaction sequence diagram between a remote user, a terminal, and agricultural machinery. For routine operations, the remote user sends tasks, work areas, and parameter configurations to the terminal. The terminal runs an intelligent navigation system, which abstracts the user-configured work plan into executable task units through a task scheduling module, such as "spraying a certain plot," "weeding a certain plot," "inspection and sampling," and "transfer to a warehouse / charging point," while maintaining task queues, priorities, and execution conditions. Based on the global pose output by the global fusion positioning module, the current plot or passage is determined. The semantic cost map module and path planning module automatically generate and distribute the work path. The control module is responsible for path tracking and the linkage of the execution mechanisms. The task scheduling module continuously monitors task completion and abnormal events: when it detects obstruction of the passage ahead, blockage of the operating mechanism, sensor degradation, decreased positioning reliability, or triggering of the safe zone, the module can automatically pause the current task, trigger local replanning, or switch to a safety strategy (slow down, detour, wait in place, return to the safe point), and record the event for subsequent traceability; when the task meets the completion conditions, such as coverage reaching the threshold, completion of work in the target area, or completion of inspection point visits, the task scheduling module automatically switches to the next task and performs necessary status resets.

[0159] For initial operations in unfamiliar scenarios, the task scheduling module provides a "first exploration and mapping, then routine operations" scheduling process to reduce reliance on manual preparation when first entering a new site. When users first enter an unfamiliar site, they only need to define the work area, usually by inputting polygon boundaries or site outlines. The task scheduling module then triggers the exploration and mapping task: automatically planning an exploration path within the work area, driving the positioning and mapping module to collect sensor data and build a priori map of the site, including the structure of accessible passages, boundaries, obstacle distribution, and key landmark information. During the exploration process, the module continuously evaluates the mapping quality, including loop consistency, coverage, and location reliability. When the preset mapping completion conditions are met, the map is solidified as the "scene prior" for that site. Subsequently, when performing periodic tasks such as spraying, weeding, and inspection in the same scene, the task scheduling module prioritizes loading this priori map for quick initialization of positioning and planning, and corrects and completes the map through incremental updates during operation. For example, newly appearing temporary obstacles, passage changes, or terrain changes are written into the incremental layer, thereby achieving continuous map iteration without repeating full mapping. If the cumulative incremental changes exceed the threshold or the inconsistency between the map and reality increases, the module can automatically trigger a local reconstruction or re-exploration process to ensure the effectiveness of the map and the reliability of the planning in the long run.

[0160] Addressing the characteristics of "periodic tasks" and "24 / 7 operation" in agricultural scenarios, the task scheduling module supports a time- and condition-triggered scheduling mechanism. It can automatically generate task plans based on operational cycles, such as daily inspections, weekly spraying, and fertilization according to growth stages, and dynamically adjust the execution window in conjunction with external environmental factors. Specifically, the module can access meteorological information, such as rainfall, wind speed, temperature, and humidity, and crop status information, such as soil moisture, pest and disease risk, and crop growth indicators. Under unsuitable operating conditions, tasks are automatically postponed or canceled, while tasks are automatically started during suitable periods and multiple plots are scheduled for continuous operation, reducing manual intervention. Simultaneously, the task scheduling module monitors equipment maintenance status and incorporates it into scheduling constraints. When insufficient battery / fuel, expired maintenance cycles, abnormal temperatures of critical components, fault codes, or insufficient consumables are detected, the module prioritizes triggering replenishment and maintenance tasks. For example, it automatically returns to the charging station / refueling point / liquid replenishment point, enters maintenance parking mode, and resumes unfinished tasks or continues execution from the most recent breakpoint after replenishment, thereby achieving long-term stable operation of agricultural machinery.

[0161] To address situations where sensors are missing or degraded, when a sensor becomes unavailable (e.g., due to LiDAR obstruction, malfunction, or lack of equipment), the intelligent navigation system automatically utilizes other sensors or alternative algorithms to compensate for the data loss, ensuring that basic functions are not affected. Specifically, when a sensor is missing, it automatically switches to a tolerance-tolerant scheme of monocular depth estimation or point cloud geometry processing, achieving continuous acquisition of environmental semantic information while ensuring safety. This redundant design at the sensor layer improves the system's robustness.

[0162] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0163] Based on the same inventive concept, this application also provides an intelligent navigation device for agricultural machinery to implement the intelligent navigation method for agricultural machinery as described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more embodiments of the intelligent navigation device for agricultural machinery provided below can be found in the limitations of the intelligent navigation method for agricultural machinery described above, and will not be repeated here.

[0164] In one exemplary embodiment, such as Figure 6 As shown, an intelligent navigation device for agricultural machinery is provided, the device comprising:

[0165] The data acquisition module 610 is used to acquire lidar point cloud data, camera image data, inertial measurement unit data, and global satellite navigation system positioning data of agricultural machinery and equipment.

[0166] The local pose acquisition module 620 is used to obtain a first pose increment based on lidar point cloud data and inertial measurement unit data; obtain a second pose increment based on camera image data and inertial measurement unit data; fuse the first pose increment and the second pose increment to obtain the local pose of the agricultural machinery; the local pose is the pose of the agricultural machinery in the local coordinate system.

[0167] The global pose acquisition module 630 is used to fuse local pose and global satellite navigation system positioning data to obtain the global pose of agricultural machinery and equipment; the global pose is the pose of agricultural machinery and equipment in the global coordinate system;

[0168] The environmental semantic information determination module 640 is used to determine the environmental semantic information of agricultural machinery and equipment.

[0169] The navigation path generation module 650 is used to generate navigation paths for agricultural machinery based on global pose and environmental semantic information; the navigation path is used to instruct the agricultural machinery to navigate.

[0170] In one embodiment, the first pose increment and the second pose increment are fused to obtain the local pose of the agricultural machinery, including:

[0171] Acquire point cloud registration residuals, number of point cloud matching points, number of point cloud feature points, photometric error, motion intensity information of inertial measurement unit, and continuous pre-integration of inertial measurement unit;

[0172] By using the attention weight allocation network, based on the first pose increment, point cloud registration residual, number of point cloud matching points, second pose increment, number of point cloud feature points, photometric error, and motion intensity information of the inertial measurement unit, the first weight coefficient corresponding to the first pose increment and the second weight coefficient corresponding to the second pose increment are obtained.

[0173] Based on the first weighting coefficient and the second weighting coefficient, the first pose increment and the second pose increment are weighted and summed to obtain the fused pose increment.

[0174] The fused pose increment and the inertial measurement unit are continuously pre-integrated and input to the Kalman filter to obtain the local pose.

[0175] In one embodiment, the local pose and global navigation satellite system positioning data are fused to obtain the global pose of the agricultural machinery, including:

[0176] Transform the local pose from the local coordinate system to the global coordinate system to obtain the predicted pose;

[0177] Calculate the residual between the global navigation satellite system positioning data and the predicted pose to obtain the pose residual;

[0178] Calculate the set of quality indicators for global navigation satellite system positioning data;

[0179] If each quality indicator in the quality indicator set meets the corresponding preset indicator threshold and the pose residual is less than the preset residual threshold, the observation noise covariance of the global satellite navigation system positioning data and the local pose covariance of the local pose are obtained. The observation noise covariance and the local pose covariance are input into the gating network to obtain the third weight coefficient corresponding to the global satellite navigation system positioning data and the fourth weight coefficient corresponding to the pose residual.

[0180] The global pose is obtained by weighting and summing the positioning data and pose residuals of the global satellite navigation system based on the third and fourth weighting coefficients.

[0181] In one embodiment, determining the environmental semantic information of agricultural machinery includes:

[0182] Target object detection is performed on camera image data to obtain the target object region;

[0183] The LiDAR point cloud data is projected onto the image coordinate system of the camera image data, and the projected LiDAR point cloud data is associated with the target object area to obtain environmental semantic information; or...

[0184] The camera image data is input into the depth estimation model to obtain a depth map; based on the depth map and the camera's intrinsic parameters, simulated point cloud data is obtained; the simulated point cloud data is projected onto the image coordinate system where the camera image data is located, and the projected simulated point cloud data is associated with the target object region to obtain environmental semantic information.

[0185] In one embodiment, a navigation path for agricultural machinery is generated based on global pose and environmental semantic information, including:

[0186] Based on environmental semantic information, a semantic cost map is constructed; the semantic cost map represents the cost required for agricultural machinery to move to different locations;

[0187] Based on the semantic cost map, the target passable area is obtained; the target passable area is a passable area with a cost less than a preset threshold and continuous distribution.

[0188] Based on the semantic label information of the semantic cost map, the field boundary area and the inter-row gap area are identified from the target traversable area;

[0189] Extract the center passage lines of the field ridge boundary area and the inter-row gap area to obtain multiple channels;

[0190] Based on the positional relationship between agricultural machinery and equipment and each channel, determine the transition path between each channel;

[0191] Based on each channel and transition path, a navigation path for agricultural machinery and equipment is generated.

[0192] In one embodiment, after generating the navigation path for the agricultural machinery, the method further includes:

[0193] Based on the navigation path and global pose, generate motion control commands;

[0194] According to motion control commands, control agricultural machinery and equipment to move along the navigation path.

[0195] In one embodiment, after acquiring lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data of agricultural machinery and equipment, the method further includes:

[0196] Add timestamps to lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data;

[0197] Based on the timestamp, the lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data are time-aligned.

[0198] The various modules in the intelligent navigation device of the aforementioned agricultural machinery can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0199] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows. Figure 7 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores intelligent navigation data for agricultural machinery. The I / O interfaces allow the processor to exchange information with external devices. The communication interface allows communication with external terminals via a network connection. When the computer program is executed by the processor, it implements an intelligent navigation method for agricultural machinery.

[0200] Those skilled in the art will understand that Figure 7 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the intelligent navigation method for agricultural machinery described above. The steps of the intelligent navigation method for agricultural machinery described here can be the steps in the intelligent navigation method for agricultural machinery of the various embodiments described above.

[0201] In one embodiment, a computer-readable storage medium is provided, storing a computer program that, when executed by a processor, causes the processor to perform the steps of the intelligent navigation method for agricultural machinery described above. The steps of the intelligent navigation method for agricultural machinery described here can be the steps from the intelligent navigation method for agricultural machinery in the various embodiments described above.

[0202] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the steps of the intelligent navigation method for agricultural machinery described above. The steps of the intelligent navigation method for agricultural machinery described here can be the steps in the intelligent navigation method for agricultural machinery of the various embodiments described above.

[0203] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0204] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0205] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0206] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. An intelligent navigation method for agricultural machinery, characterized in that, The method includes: Acquire lidar point cloud data, camera image data, inertial measurement unit data, and global satellite navigation system positioning data from agricultural machinery and equipment; A first pose increment is obtained based on the lidar point cloud data and the inertial measurement unit data; a second pose increment is obtained based on the camera image data and the inertial measurement unit data; the first pose increment and the second pose increment are fused to obtain the local pose of the agricultural machinery; the local pose is the pose of the agricultural machinery in a local coordinate system. The local pose and the global satellite navigation system positioning data are fused to obtain the global pose of the agricultural machinery; the global pose is the pose of the agricultural machinery in the global coordinate system. Determine the environmental semantic information of the agricultural machinery and equipment; Based on the global pose and the environmental semantic information, a navigation path is generated for the agricultural machinery; the navigation path is used to instruct the agricultural machinery to navigate.

2. The method according to claim 1, characterized in that, The step of fusing the first pose increment and the second pose increment to obtain the local pose of the agricultural machinery includes: Acquire point cloud registration residuals, number of point cloud matching points, number of point cloud feature points, photometric error, motion intensity information of inertial measurement unit, and continuous pre-integration of inertial measurement unit; By using an attention weight allocation network, based on the first pose increment, the point cloud registration residual, the number of matching points in the point cloud, the second pose increment, the number of feature points in the point cloud, the photometric error, and the motion intensity information of the inertial measurement unit, a first weight coefficient corresponding to the first pose increment and a second weight coefficient corresponding to the second pose increment are obtained. Based on the first weighting coefficient and the second weighting coefficient, the first pose increment and the second pose increment are weighted and summed to obtain the fused pose increment; The fused pose increment and the inertial measurement unit are continuously pre-integrated and input to the Kalman filter to obtain the local pose.

3. The method according to claim 1, characterized in that, The process of fusing the local pose and the global navigation satellite system positioning data to obtain the global pose of the agricultural machinery includes: The local pose is transformed from the local coordinate system to the global coordinate system to obtain the predicted pose; Calculate the residual between the global navigation satellite system positioning data and the predicted pose to obtain the pose residual; Calculate the set of quality indicators for the global navigation satellite system positioning data; If each quality indicator in the set of quality indicators meets the corresponding preset indicator threshold and the pose residual is less than the preset residual threshold, the observation noise covariance of the global satellite navigation system positioning data and the local pose covariance of the local pose are obtained. The observation noise covariance and the local pose covariance are input into the gating network to obtain the third weight coefficient corresponding to the global satellite navigation system positioning data and the fourth weight coefficient corresponding to the pose residual. The global pose is obtained by weighting and summing the global satellite navigation system positioning data and the pose residual based on the third and fourth weighting coefficients.

4. The method according to claim 1, characterized in that, Determining the environmental semantic information of the agricultural machinery includes: Target object detection is performed on the camera image data to obtain the target object region; The lidar point cloud data is projected onto the image coordinate system where the camera image data is located, and the projected lidar point cloud data is associated with the target object region to obtain the environmental semantic information; or... The camera image data is input into a depth estimation model to obtain a depth map; based on the depth map and the camera's intrinsic parameters, simulated point cloud data is obtained; the simulated point cloud data is projected onto the image coordinate system where the camera image data is located, and the projected simulated point cloud data is associated with the target object region to obtain the environmental semantic information.

5. The method according to claim 1, characterized in that, The step of generating a navigation path for the agricultural machinery based on the global pose and the environmental semantic information includes: Based on environmental semantic information, a semantic cost map is constructed; the semantic cost map represents the cost required for the agricultural machinery to move to different locations; Based on the semantic cost map, the target passable area is obtained; the target passable area is a passable area with a cost less than a preset threshold and continuously distributed. Based on the semantic label information of the semantic cost map, the field boundary area and the inter-row gap area are identified from the target traversable area; Extract the center passage lines of the field ridge boundary area and the inter-row gap area to obtain multiple channels; Based on the positional relationship between the agricultural machinery and each of the channels, the transition path between each of the channels is determined; The navigation path for the agricultural machinery is generated based on each of the channels and the transition path.

6. The method according to any one of claims 1 to 5, characterized in that, After generating the navigation path for the agricultural machinery, the method further includes: Based on the navigation path and the global pose, generate motion control commands; According to the motion control command, the agricultural machinery is controlled to move along the navigation path.

7. The method according to any one of claims 1 to 5, characterized in that, After acquiring lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data from agricultural machinery and equipment, the method further includes: Add timestamps to the lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data; Based on the timestamp, the lidar point cloud data, camera image data, inertial measurement unit data, and global navigation satellite system positioning data are time-aligned.

8. An intelligent navigation device for agricultural machinery, characterized in that, The device includes: The data acquisition module is used to acquire lidar point cloud data, camera image data, inertial measurement unit data, and global satellite navigation system positioning data from agricultural machinery and equipment. The local pose acquisition module is used to obtain a first pose increment based on the lidar point cloud data and the inertial measurement unit data; obtain a second pose increment based on the camera image data and the inertial measurement unit data; and fuse the first pose increment and the second pose increment to obtain the local pose of the agricultural machinery; the local pose is the pose of the agricultural machinery in a local coordinate system. A global pose acquisition module is used to fuse the local pose and the global satellite navigation system positioning data to obtain the global pose of the agricultural machinery; the global pose is the pose of the agricultural machinery in the global coordinate system. An environmental semantic information determination module is used to determine the environmental semantic information of the agricultural machinery and equipment; The navigation path generation module is used to generate a navigation path for the agricultural machinery based on the global pose and the environmental semantic information; the navigation path is used to instruct the agricultural machinery to navigate.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images