Real-time semantic SLAM system in agricultural field environment

Abstract

The invention discloses a real-time semantic SLAM system in an agricultural field environment. The real-time semantic SLAM system comprises a semantic information extraction module, an SLAM preprocessing module, an SLAM front-end module and an SLAM rear-end module, the semantic information extraction module is used for preprocessing an image acquired by an agricultural robot, performing segmentation reasoning on semantic information in the preprocessed input image, and feeding back the extracted semantic information to the SLAM preprocessing module through an ROS system; the SLAM preprocessing module stores the semantic information extracted by the semantic information extraction module, and performs image masking for subsequent point cloud and semantic information matching and semantic map forming; the SLAM front-end module is used for associating the image frames in the SLAM preprocessing module with the semantic information; and the SLAM rear-end module is used for displaying the constructed semantic map and observing semantic information in the agricultural field environment through the semantic map. And real-time and high-precision environment map construction is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of semantic maps, and in particular to a real-time semantic SLAM system in an agricultural field environment. Background Art

[0002] Semantic maps play a vital role in smart industry. By providing detailed 3D crop row data, they can significantly optimize agricultural robot management strategies, thereby improving the economic benefits of agricultural field production. However, in open-air agricultural environments, agricultural robots face challenges such as scene homogeneity, weak motion landmarks, unstable lighting, and other unfavorable factors. At the same time, the bumpy movement during off-road driving can also cause interference. These factors make it particularly complex to achieve accurate and reliable autonomous positioning and real-time environmental semantic map construction in this environment.

[0003] At present, some studies in the agricultural field have explored semantic SLAM technology, but most of the current systems are still offline or non-real-time, which makes it difficult to meet the needs of real-time decision-making and planning in agricultural scenarios.

[0004] Therefore, there is an urgent need to provide a new type of real-time semantic SLAM system in agricultural field environments to solve the problem of map construction faced by existing technologies in complex dynamic agricultural environments. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a real-time semantic SLAM system in an agricultural field environment, which can realize real-time, high-precision environmental map construction and integrate rich semantic information to support agricultural robots to perform autonomous navigation and intelligent decision-making in unstructured vast field environments.

[0006] In order to solve the above technical problems, a technical solution adopted by the present invention is: providing a real-time semantic SLAM system in an agricultural field environment, including a semantic information extraction module, a SLAM preprocessing module, a SLAM front-end module, and a SLAM back-end module; The semantic information extraction module is used to preprocess the images collected by the agricultural robot, segment and reason the semantic information in the preprocessed input images, and feed the extracted semantic information back to the SLAM preprocessing module through the ROS system; The SLAM preprocessing module saves the semantic information extracted by the semantic information extraction module and performs image masking for subsequent point cloud and semantic information matching and semantic map formation; The SLAM front-end module is used to associate the image frames and semantic information in the SLAM pre-processing module; The SLAM backend module is used to display the constructed semantic map, and the semantic information in the agricultural field environment is observed through the semantic map.

[0007] In a preferred embodiment of the present invention, the semantic information extraction module includes an image preprocessing unit and a semantic segmentation unit; The image preprocessing unit is used to preprocess the images collected by the agricultural robot, including loading images, adjusting sizes, normalizing pixel values, and performing data enhancement; The semantic segmentation unit is used to perform segmentation reasoning on the semantic information in the preprocessed input image, and feed back the extracted semantic information to the SLAM preprocessing module through the ROS system.

[0008] Furthermore, the semantic segmentation unit extracts multi-scale features from the preprocessed input image based on the Fast-SCNN network architecture and generates a segmentation mask.

[0009] In a preferred embodiment of the present invention, the SLAM preprocessing module includes an IMU unit, a picture frame unit, and an image mask unit; The IMU unit is used to collect and process raw data from an inertial measurement unit (IMU), including data from an accelerometer and a gyroscope, to assist in subsequent posture estimation; The picture frame unit is used to transmit the picture frame to the semantic information extraction module to extract semantic information; The image mask unit is used to extract semantic information in the image and distinguish "crops" from "non-crops".

[0010] In a preferred embodiment of the present invention, the SLAM front-end module includes an IMU pre-division unit, a descriptor extraction unit, and a semantic information association unit; The IMU prediction subunit is used to use the accelerometer and gyroscope data of the inertial measurement unit IMU to perform real-time prediction and integration of the motion state of the device to generate a preliminary posture estimate; The descriptor extraction unit is used to extract key points and corresponding local feature descriptors from each frame of image; The semantic information association unit is used to associate the image frame in the SLAM processing module with the semantic information extracted by the semantic information extraction module.

[0011] Furthermore, the step of the semantic information associating unit associating the image frame in the SLAM processing module with the semantic information extracted by the semantic information extraction module comprises: First, the preprocessed image is processed using the semantic segmentation model in the semantic information extraction module to generate a category label map and a confidence map for each pixel, wherein the pixel label represents the classification result of crops and non-crops; Then, the image frame and its generated semantic label map and confidence map are saved as the first key frame, and the camera pose information is recorded. For a normal frame, when it is set as a key frame, the image is also processed to generate semantic information, and its semantic label map, confidence map and camera pose information are saved together; Finally, the new keyframes and their semantic information are fused into the map in the SLAM system, and the map is updated through a graph optimization algorithm to ensure the consistency of the information.

[0012] In a preferred embodiment of the present invention, the SLAM backend module includes a semantic map construction unit and an IMU initialization unit; The IMU initialization unit is used to accurately calibrate and initialize the IMU sensor at the initial stage of system startup; The semantic map construction unit is used to display the constructed semantic map, and the semantic information in the agricultural field environment is observed through the semantic map.

[0013] Furthermore, the semantic map construction unit intuitively displays the point cloud data with semantic labels in the constructed semantic map by coloring the point cloud.

[0014] The beneficial effects of the present invention are: (1) By combining semantic segmentation and SLAM technology, the present invention can construct a high-precision environmental map in real time in an agricultural field environment. This high-precision map can provide agricultural robots with detailed environmental information, helping them to better understand and adapt to complex agricultural scenarios; (2) The present invention can provide accurate semantic maps for agricultural robots, which not only supports the robot's autonomous navigation, but also improves its intelligent decision-making ability in actual operations. For example, the robot can plan paths more effectively, avoid obstacles, and improve operating efficiency and safety; (3) The technical solution of the present invention improves the intelligent management level of agricultural fields and promotes the development of smart agriculture. Through automation and intelligent means, the economic benefits and efficiency of agricultural production have been significantly improved, thereby promoting the progress of agricultural science and technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a structural block diagram of a preferred embodiment of the real-time semantic SLAM system in an agricultural field environment of the present invention; Figure 2 is the ROS information system data flow diagram; Figure 3 It is a construction effect diagram of the semantic map. DETAILED DESCRIPTION

[0016] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby making a clearer and more definite definition of the protection scope of the present invention.

[0017] See also Figure 1 , the embodiment of the present invention includes: A real-time semantic SLAM system in an agricultural field environment comprises a semantic information extraction module, a SLAM preprocessing module, a SLAM front-end module and a SLAM back-end module.

[0018] The semantic information extraction module is used to preprocess the images collected by the agricultural robot, and to segment and infer the semantic information in the preprocessed input images, and to feed back the extracted semantic information to the SLAM preprocessing module through the ROS system, including an image preprocessing unit and a semantic segmentation unit.

[0019] like Figure 2 As shown, the semantic extraction module is implemented based on the ROS system. The three functional subsystems of data set playback, picture frame semantic segmentation, simultaneous positioning and semantic map construction are encapsulated in ROS nodes. The three ROS nodes are independent of each other at runtime and the exchange of data sets is completed through the message subscription mechanism.

[0020] Before entering the SLAM preprocessing module, the picture frame first performs segmentation reasoning on the semantic information in the input image in advance in the semantic extraction module, and then feeds the extracted semantic information back to the SLAM preprocessing module. The semantic segmentation unit implements semantic segmentation through a semantic segmentation network. The semantic segmentation network based on Fast-SCNN first needs to perform image preprocessing, including loading images, resizing, normalizing pixel values, and performing data enhancement (such as rotation, flipping, etc.) to improve the diversity and consistency of training data. Then, the feature extractor in the Fast-SCNN network architecture (a lightweight deep network such as MobileNet or Xception) extracts multi-scale features from the preprocessed image. The segmentation head of the network uses depth-separable convolution and hole convolution to generate accurate segmentation masks. In the training stage, a labeled image dataset (in which each pixel is marked as a crop or non-crop) is used to optimize the network parameters, apply loss functions such as cross entropy loss, and perform performance evaluation on the validation set to adjust hyperparameters. Finally, the trained model is applied to the new image for prediction, and the segmentation results are post-processed (such as smoothing and edge repair) to improve the accuracy of distinguishing crops from non-crops. This loose coupling ensures that the semantic extraction module will not affect the operation of the overall SLAM system, and provides good conditions for the further development of subsequent semantic information extraction. The semantic information extraction module uses data enhancement techniques such as random resizing, cropping, flipping and photometric distortion to improve generalization performance, uses SGD optimizer for training, and uses a polynomial learning rate scheduler.

[0021] The SLAM preprocessing module saves the semantic information extracted by the semantic information extraction module and performs image masking for subsequent point cloud and semantic information matching and semantic map formation, including an IMU unit, a picture frame unit, and an image mask unit.

[0022] The IMU unit is used to collect and process raw data from an inertial measurement unit (IMU), including data from an accelerometer and a gyroscope, to assist in subsequent posture estimation.

[0023] The picture frame unit is used to transmit the picture frame to the semantic information extraction module to extract semantic information.

[0024] The image mask unit is used to extract semantic information in the image and distinguish "crops" from "non-crops".

[0025] Specifically, in the SLAM preprocessing module, the picture frame is first transmitted to the semantic extraction module to extract the semantic information in the picture frame, and then the extracted semantic information is saved in the image mask submodule in the SLAM preprocessing module for subsequent point cloud matching with semantic information and semantic map formation. The image mask is used to extract semantic information in the image and distinguish between "crops" and "non-crops". First, the trained semantic segmentation model classifies each pixel in the image as "crops" or "non-crops" and generates a mask map in which each pixel value corresponds to the category predicted by the model. Then, these masks can be used for image processing and analysis to segment out "crop" areas or remove "non-crop" areas from the image. Combined with Figure 2 In this example, the left initial image of the camera in the binocular camera is used for segmentation mask.

[0026] The SLAM front-end module is used to associate the image frames and semantic information in the SLAM pre-processing module, including an IMU pre-score unit, a descriptor extraction unit, and a semantic information association unit. The IMU pre-score unit is used to use the accelerometer and gyroscope data of the inertial measurement unit (IMU) to predict and integrate the motion state of the device in real time and generate a preliminary pose estimate. The descriptor extraction unit is used to extract key points and corresponding local feature descriptors from each frame of the image, and accurately perform feature matching even when the image changes in perspective, illumination, or is partially occluded. The semantic information association module associates the image frames and semantic information in the SLAM pre-processing module. This association is mainly at two times in this module, one is when the first key frame is created, and the other is when a normal frame is set as a key frame in the tracking thread. The present invention also associates semantic information with map points at these two times when the map points are created, which facilitates the subsequent construction of semantic maps.

[0027] In the SLAM system, the process of associating image frames with semantic information (segmenting crops) can be achieved through the following steps: First, the preprocessed image is processed using a semantic segmentation model to generate a class label map and a confidence map for each pixel, where the pixel label represents the classification result of crops and non-crops. Then, the image frame and its generated semantic label map and confidence map are saved as the first keyframe, and the camera pose information is recorded. For a normal frame, when it is set as a keyframe, the image is also processed to generate semantic information, and its semantic label map, confidence map and camera pose information are saved together. Subsequently, the new keyframe and its semantic information are fused into the map in the SLAM system, and the map is updated through a graph optimization algorithm to ensure the consistency of the information. Finally, the updated map containing semantic information is used for further analysis, such as accurately identifying crop areas or other environmental features. Through these steps, the SLAM system can effectively associate image frames with semantic information, thereby improving environmental understanding and positioning accuracy.

[0028] The SLAM backend module is used to display the constructed semantic map, and observe the semantic information in the agricultural field environment through the semantic map, including a semantic map construction unit and an IMU initialization unit. The IMU initialization unit is used to accurately calibrate and initialize the IMU sensor at the initial stage of system startup, including estimating the IMU bias (such as the zero bias of the gyroscope and accelerometer), noise parameters, and initial speed, position, and direction.

[0029] Compared with the traditional SLAM system, the newly added semantic map construction module of the present invention can clearly observe the semantic information in the environment. In terms of semantic map construction, the present invention realizes the precise association between semantic labels and three-dimensional point clouds by integrating the precise semantic information (distinguishing between crops and non-crops) extracted by the previous module with the corresponding video frame data. Figure 2 As shown in the figure, the initial image of the left camera, the initial image of the right camera, the segmentation mask and the original segmentation image are associated and combined with the IMU data to realize the construction of the semantic map. Subsequently, through point cloud coloring, the point cloud data with semantic labels are intuitively displayed in the semantic map, ensuring the accuracy and timeliness of the semantic map. At the same time, it effectively responds to the dynamic changes in the agricultural field environment and provides strong support for subsequent path planning, control tasks and agricultural management decisions.

[0030] like Figure 3 As shown in the figure, the semantic map construction effect is intuitively demonstrated. The constructed semantic point cloud is mainly composed of crop row point clouds (green point clouds), but there are many background point clouds (black point clouds) mixed in the crop row point clouds. The background point cloud is generally distributed in horizontal strips. The black color corresponds to the channels between the ridges of agricultural fields. The black point cloud is non-agricultural crops, including soil, weeds, trees, etc.

[0031] The main goal of the present invention is to achieve real-time, high-precision environmental map construction and integrate rich semantic information to support agricultural robots in unstructured vast field environments for autonomous navigation and intelligent decision-making. By proposing a semantic SLAM technology, the adaptability and stability of the system in the agricultural field environment can be significantly improved, and the intelligent management level of agricultural fields will be greatly improved, thereby promoting the comprehensive development of smart agriculture and significantly improving the economic benefits and efficiency of agricultural production.

[0032] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiments can be implemented by software, or by means of software plus necessary general hardware platforms. Based on such understanding, the technical solutions of the above embodiments can be embodied in the form of software products, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), including several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods of various embodiments of the present invention.

[0033] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A real-time semantic SLAM system in an agricultural field environment, characterized in that: Including semantic information extraction module, SLAM preprocessing module, SLAM front-end module, SLAM back-end module; The semantic information extraction module is used to preprocess the images collected by the agricultural robot, segment and reason the semantic information in the preprocessed input images, and feed the extracted semantic information back to the SLAM preprocessing module through the ROS system; The SLAM preprocessing module saves the semantic information extracted by the semantic information extraction module and performs image masking for subsequent point cloud and semantic information matching and semantic map formation; The SLAM front-end module is used to associate the image frames and semantic information in the SLAM pre-processing module; The SLAM backend module is used to display the constructed semantic map, and the semantic information in the agricultural field environment is observed through the semantic map.

2. The real-time semantic SLAM system in an agricultural field environment according to claim 1, characterized in that: The semantic information extraction module includes an image preprocessing unit and a semantic segmentation unit; The image preprocessing unit is used to preprocess the images collected by the agricultural robot, including loading images, adjusting sizes, normalizing pixel values, and performing data enhancement; The semantic segmentation unit is used to perform segmentation reasoning on the semantic information in the preprocessed input image, and feed back the extracted semantic information to the SLAM preprocessing module through the ROS system.

3. The real-time semantic SLAM system in an agricultural field environment according to claim 2, characterized in that: The semantic segmentation unit extracts multi-scale features from the preprocessed input image based on the Fast-SCNN network architecture and generates a segmentation mask.

4. The real-time semantic SLAM system in an agricultural field environment according to claim 1, characterized in that: The SLAM preprocessing module includes an IMU unit, a picture frame unit, and an image mask unit; The IMU unit is used to collect and process raw data from an inertial measurement unit (IMU), including data from an accelerometer and a gyroscope, to assist in subsequent posture estimation; The picture frame unit is used to transmit the picture frame to the semantic information extraction module to extract semantic information; The image mask unit is used to extract semantic information in the image and distinguish "crops" from "non-crops".

5. The real-time semantic SLAM system in an agricultural field environment according to claim 1, characterized in that: The SLAM front-end module includes an IMU pre-division unit, a descriptor extraction unit, and a semantic information association unit; The IMU prediction subunit is used to use the accelerometer and gyroscope data of the inertial measurement unit IMU to perform real-time prediction and integration of the motion state of the device to generate a preliminary posture estimate; The descriptor extraction unit is used to extract key points and corresponding local feature descriptors from each frame of image; The semantic information association unit is used to associate the image frame in the SLAM processing module with the semantic information extracted by the semantic information extraction module.

6. The real-time semantic SLAM system in an agricultural field environment according to claim 5, characterized in that: The step of the semantic information associating unit associating the image frame in the SLAM and processing module with the semantic information extracted by the semantic information extraction module comprises: First, the preprocessed image is processed using the semantic segmentation model in the semantic information extraction module to generate a category label map and a confidence map for each pixel, wherein the pixel label represents the classification result of crops and non-crops; Then, the image frame and its generated semantic label map and confidence map are saved as the first key frame, and the camera pose information is recorded. For a normal frame, when it is set as a key frame, the image is also processed to generate semantic information, and its semantic label map, confidence map and camera pose information are saved together; Finally, the new keyframes and their semantic information are fused into the map in the SLAM system, and the map is updated through a graph optimization algorithm to ensure the consistency of the information.

7. The real-time semantic SLAM system in an agricultural field environment according to claim 1, characterized in that: The SLAM backend module includes a semantic map construction unit and an IMU initialization unit; The IMU initialization unit is used to accurately calibrate and initialize the IMU sensor at the initial stage of system startup; The semantic map construction unit is used to display the constructed semantic map, and the semantic information in the agricultural field environment is observed through the semantic map.

8. The real-time semantic SLAM system in an agricultural field environment according to claim 7, characterized in that: The semantic map construction unit intuitively displays the point cloud data with semantic labels in the constructed semantic map through point cloud coloring.

Images