Unmanned simultaneous positioning and mapping method and device, medium and equipment

Abstract

The invention discloses an unmanned simultaneous positioning and mapping method and device, a medium and equipment, and relates to the technical field of computers, and the method comprises the following steps: obtaining image data of the surrounding environment of unmanned equipment, carrying out semantic segmentation through employing an improved semantic segmentation model, and determining the semantic information of each pixel; acquiring multi-frame point cloud data, and performing distortion correction on the multi-frame point cloud data by performing pre-integration on the inertial measurement unit data; projecting each frame of point cloud data to the time-space synchronized image data, and determining a semantic tag of a three-dimensional point in each frame of point cloud data; removing three-dimensional points of the dynamic object according to the speed and the space overlapping degree; and optimizing a surrounding environment map and a pose map of the unmanned equipment by selecting a key frame and a loopback candidate frame. According to the method, the semantic segmentation technology and the laser radar inertial odometer are fused for positioning and mapping, so that the positioning and mapping precision in a dynamic similar environment is greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, device, medium, and equipment for simultaneous positioning and mapping of unmanned vehicles. Background Art

[0002] Currently, Simultaneous Localization and Mapping (SLAM) technology can provide reliable surrounding environment information and real-time location for unmanned devices (such as self-driving cars), and is the foundation of autonomous driving technology. Traditional SLAM works well in static environments, but most objects in real-world environments are moving. Dynamic objects cause ghosting, which reduces the positioning accuracy of self-driving cars and leads to errors in the constructed map. In addition, in order to achieve intelligent navigation of self-driving cars, such as "parking the car next to the tree", it is necessary to convert semantic information into a spatial representation to construct a semantic map of higher quality than traditional SLAM maps.

[0003] Cameras can capture rich visual information, from which semantic information can be extracted. They also offer advantages such as low cost and compact size. However, images captured by cameras are susceptible to lighting effects. Unlike cameras, lidar (LiDAR) captures information as a series of scattered point clouds. While these point clouds are sparse and lack texture features, they contain accurate angle and distance information and a precise 3D spatial representation. Therefore, a single sensor has limitations. In autonomous driving scenarios, localization and mapping accuracy are extremely demanding. In scenes with unclear features, high-precision pose estimation using either cameras or lidar alone is difficult. Inertial measurement units (IMUs) can provide highly accurate pose solutions in a short period of time. Currently, lidar (Light Laser Detection and Ranging, LiDAR), IMUs, and cameras are the most widely used sensors in SLAM algorithms. Therefore, fusing multi-source data based on the characteristics of each sensor is a reasonable approach.

[0004] Generally speaking, a typical SLAM system can be divided into two modules: front-end odometry and back-end optimization. The front-end module uses data collected by various sensors to calculate the position and attitude changes between two adjacent frames, ultimately outputting pose information. The back-end module optimizes and corrects the accumulated errors generated by the front-end odometry to improve positioning and mapping accuracy. Loop detection plays a crucial role in the back-end module, as it correlates recent data with historical data to improve the robustness and accuracy of the SLAM system.

[0005] However, traditional SLAM relies primarily on visual images or laser point clouds to perceive the environment, detecting loop closure based on scene similarity. Image-based methods can easily misdetect loop closures in areas with similar textures, leading to mapping failure. Point cloud-based methods also fail in some degraded or similar scenes. Summary of the Invention

[0006] Based on this, it is necessary to provide a method, device, medium and equipment for simultaneous positioning and mapping of unmanned driving to address the above technical problems.

[0007] The present invention adopts the following technical solutions:

[0008] The present invention provides a method for simultaneous positioning and mapping of an unmanned vehicle, comprising:

[0009] The image data of the environment surrounding the unmanned driving device acquired during the current acquisition cycle is input into a pre-trained semantic segmentation model for semantic segmentation, and the semantic information of each pixel in the image data is determined;

[0010] Acquire multi-frame point cloud data of the environment around the unmanned driving device during the current acquisition cycle, and pre-integrate the inertial measurement unit data of the unmanned driving device during the current acquisition cycle to perform distortion correction on the multi-frame point cloud data;

[0011] Determine the image data that is spatiotemporally synchronized with each frame of point cloud data, and project each frame of point cloud data onto the spatiotemporally synchronized image data; determine the semantic labels of the three-dimensional points in each frame of point cloud data based on the semantic information of the pixels in the image data and the projection of the point cloud data on the image;

[0012] Based on the changes in the speed and space occupied by the 3D points corresponding to different semantic labels in each frame of point cloud data, the 3D points corresponding to dynamic objects in each frame of point cloud data are determined and removed to obtain multi-frame target point cloud data containing only static objects;

[0013] Multi-frame target point cloud data is stitched together to construct a local subgraph corresponding to the current acquisition cycle; keyframes are selected from the multi-frame target point cloud data, and for each keyframe, loop candidate frames are determined from historical keyframes through geometric detection; when the loop candidate frame matches the preset semantic label in the keyframe, the environment map and pose graph of the unmanned driving device are optimized based on the loop candidate frame and the keyframe.

[0014] Optionally, the semantic segmentation model is based on the Deeplabv3+ algorithm, with the backbone network of its encoder replaced by the RefineNet network, and depthwise separable dilated convolution is adopted in the ASPP module of the RefineNet network. The convolution layer and the NAM module are sequentially connected after the ASPP module, and then the model is constructed in combination with the encoder of the Deeplabv3+ algorithm;

[0015] The image data of the environment surrounding the unmanned driving device acquired during the current acquisition cycle is input into a pre-trained semantic segmentation model for semantic segmentation, specifically including:

[0016] The image data of the unmanned vehicle's surroundings acquired during the current acquisition cycle is fed into a pre-trained semantic segmentation model. The RefineNet network in the semantic segmentation model extracts and aggregates multi-scale features of environmental objects in the image data. The NAM module then performs channel-based attention weighting to obtain deep features of the environmental objects.

[0017] The decoder fuses the shallow features of the environmental objects in the encoder with the deep features and refines the fused features to obtain the final semantic segmentation result.

[0018] Optionally, the pre-integration of the inertial measurement unit data in the current acquisition cycle of the unmanned driving device specifically includes:

[0019] The inertial measurement unit data in the current acquisition cycle of the unmanned driving device is pre-integrated using the following formula:

[0020]

[0021] in, The location of the unmanned equipment, For the speed of unmanned equipment, is the rotation of the unmanned equipment, t m is the lower bound of the current acquisition cycle, t n is the upper bound of the current acquisition cycle, t is the time in the current acquisition cycle, is the rotation matrix, The acceleration measurement data of the unmanned driving equipment, is the deviation of the acceleration sensor of the unmanned driving equipment, The measurement data of the angular velocity of the unmanned driving equipment, is the deviation of the gyroscope of the unmanned driving device, and Ω is the antisymmetric matrix of the acceleration of the unmanned driving device.

[0022] Optionally, the pre-integration of the inertial measurement unit data within the current acquisition cycle of the unmanned driving device to perform distortion correction on the multi-frame point cloud data specifically includes:

[0023] For each frame of point cloud data, the position change of the lidar on the unmanned driving device during the period of collecting the frame of point cloud data is determined by pre-integrating the inertial measurement unit data of the unmanned driving device during the period of collecting the frame of point cloud data;

[0024] According to the position change of the laser radar during the period of collecting the frame of point cloud data, the position change of the point cloud data at any time during the period of collecting the frame of point cloud data relative to the initial time during the period of collecting the frame of point cloud data is obtained by linear interpolation;

[0025] According to the change in the posture of the point cloud data at any time during the period of collecting the frame of point cloud data relative to the initial time during the period of collecting the frame of point cloud data, the coordinates of each three-dimensional point in the frame of point cloud data are converted to the coordinate system of the first three-dimensional point collected in the frame of point cloud data, thereby completing the distortion correction of the frame of point cloud data.

[0026] Optionally, the method further includes:

[0027] For each frame of distortion-corrected point cloud data, determine the range image of the frame of point cloud data, and determine the 3D points belonging to the ground and the 3D points belonging to non-ground in the point cloud data based on the pitch angle of the range image;

[0028] Remove the 3D points belonging to the ground in the point cloud data of this frame.

[0029] Optionally, determining and removing the three-dimensional points corresponding to dynamic objects in each frame of point cloud data based on the change in speed and space occupied by the three-dimensional points corresponding to different semantic labels in each frame of point cloud data specifically includes:

[0030] Calculate the center of mass of the object formed by the three-dimensional points corresponding to different semantic labels in each frame of point cloud data;

[0031] Determine the semantic label pairs corresponding to the objects that match between the point cloud data frames based on the distances between the centroids of the objects with the same semantic labels between the point cloud data frames.

[0032] According to the time sequence of each frame of point cloud data, the velocity change of the center of mass of different objects is determined, and the 3D point corresponding to the dynamic object is determined according to the preset velocity threshold;

[0033] For each semantic label pair, determine the overlapping area of ​​the 3D points corresponding to the semantic label pair in each frame of point cloud data, and determine the 3D point corresponding to the dynamic object based on a preset area threshold;

[0034] Determine the similarity between the distribution of corresponding 3D points of the semantic label pairs in each frame of point cloud data, and determine the 3D points corresponding to the dynamic objects based on a preset similarity threshold;

[0035] Remove the 3D points corresponding to dynamic objects in each frame of point cloud data.

[0036] Optionally,

[0037] The present invention provides an unmanned driving simultaneous positioning and mapping device, comprising:

[0038] The present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned method of simultaneous positioning and mapping of unmanned driving.

[0039] The present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the method for simultaneous positioning and mapping of an unmanned vehicle is implemented.

[0040] At least one of the above technical solutions adopted by the present invention can achieve the following beneficial effects:

[0041] The present invention first obtains image data of the environment around the unmanned driving device, uses an improved semantic segmentation model to perform semantic segmentation, and determines the semantic information of each pixel; obtains multi-frame point cloud data, and corrects the distortion of the multi-frame point cloud data by pre-integrating the inertial measurement unit data; projects each frame of point cloud data onto the spatiotemporally synchronized image data, and determines the semantic labels of the three-dimensional points in each frame of point cloud data; eliminates the three-dimensional points of dynamic objects based on speed and spatial overlap; and optimizes the map of the environment around the unmanned driving device and the pose graph by selecting key frames and loop candidate frames and performing loop detection based on semantic information.

[0042] The present invention improves the positioning accuracy in similar scenarios by determining the semantic labels of three-dimensional points in each frame of point cloud data and performing loop detection based on the semantic information, while being able to construct a high-quality map with semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0044] Figure 1 A schematic flow chart of a method for simultaneous positioning and mapping of unmanned vehicles provided by the present invention;

[0045] Figure 2 A schematic diagram of a system framework provided by the present invention;

[0046] Figure 3 A schematic diagram of segmentation of ground three-dimensional points of point cloud data provided by the present invention;

[0047] Figure 4 A schematic diagram of a semantic segmentation network model provided by the present invention;

[0048] Figure 5 A schematic diagram of single-frame point cloud semantic segmentation provided by the present invention;

[0049] Figure 6 A flow chart of dynamic object detection and elimination provided by the present invention;

[0050] Figure 7 A schematic diagram of a loop detection process provided by the present invention;

[0051] Figure 8 A schematic diagram of a geometric detection threshold provided by the present invention;

[0052] Figure 9 A schematic diagram of an unmanned driving simultaneous positioning and mapping device provided by the present invention;

[0053] Figure 10 A schematic diagram of a computer device for implementing a method for simultaneous positioning and mapping of unmanned vehicles provided by the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with specific embodiments of the present invention and corresponding drawings. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0055] Over the past few decades, many researchers have devoted themselves to the study of SLAM, achieving fruitful results. Typically, an Extended Kalman Filter (EKF) is first used to estimate the relative positions of objects in an unknown environment, and a random map is built to store these spatial relationship estimates and their uncertainties. Currently, there are many types of SLAM systems, most of which use LiDAR, cameras, and IMUs.

[0056] Currently, LiDAR-based SLAM systems have achieved remarkable research results in both theory and application, thanks to their relatively stable performance both indoors and outdoors, even in complex working conditions. For example, the LOAM algorithm extracts edge points and plane points from the LiDAR point cloud and constructs a point cloud error model based on the point-line and point-plane matching relationships between feature points. Using a frame-to-sub-image matching approach, it achieves high-precision motion estimation and mapping. The Cartographer algorithm also employs a frame-to-sub-image matching strategy, coupled with loop closure detection and graph optimization, achieving excellent results even in real-time. The LeGO-LOAM algorithm, an extension of the LOAM algorithm, adds point cloud segmentation processing to eliminate noise points and incorporates loop closure detection and graph optimization methods for global optimization, improving localization and mapping accuracy. Finally, the LIOM algorithm jointly optimizes LiDAR and IMU observation information, using a CNN segmentation network to detect and filter out dynamic objects in each frame. There's also the tightly coupled LiDAR-IMU method, LIO-SAM, which fuses information from LiDAR, IMU, and GPS sensors by constructing an odometry, pre-integration, GPS, and loop factors, and using loop factor graph optimization. The FAST-LIO and FAST-LIO2 algorithms are also tightly coupled systems. These algorithms use the IMU to compensate for the motion of each point using a strict backpropagation step size and estimate the complete state of the system using a manifold iterative Kalman filter.

[0057] In addition to traditional algorithms, with the continuous advancement of deep learning technology in areas such as semantic segmentation, point cloud semantic information is gradually being applied to the field of laser SLAM to enhance scene understanding and improve localization and mapping accuracy. An instance-level semantic system has been proposed. It first uses the SSD (Single Shot MultiBox Detector) network to perform object detection in a single RGB image (2D). Based on the detection results and depth information, a 3D segmentation algorithm is used to further segment the 3D point cloud. Finally, the camera pose information from ORB-SLAM2 is used to merge the 3D point cloud segmentation results for each frame into a map to create a semantic point cloud map. A Detect-SLAM algorithm has also been proposed, combining SLAM with a deep learning-based neural network for object detection and semantic segmentation. This algorithm filters out dynamic objects, avoiding interference from unreliable features, and constructs an instance-level semantic map of objects with excellent real-time performance. Another algorithm, the DynaSLAM algorithm, combines geometric constraints with the classic semantic segmentation network Mask-RCNN to detect and remove dynamic objects. Based on the SUMA framework, a semantic laser SLAM method SUMA++ was proposed. It uses the RangeNet++ semantic segmentation network to introduce semantic information, remove dynamic objects, and use flood filling and filter techniques to achieve semantic segmentation and noise removal of point clouds. By using the semantic ICP method, point cloud data association and matching are achieved, thereby improving the accuracy of pose estimation and map construction. A SA-LOAM algorithm was also proposed. It is a laser SLAM algorithm based on LOAM and semantic assistance. This method has high pose estimation accuracy and map consistency in large-scale complex scenes. The Det-SLAM system was also proposed. It is a semantic SLAM system based on Detectron2 and ORB-SLAM3 algorithms. ORB-SLAM3 obtains the depth information of feature points through the movement of the camera and uses semantic information to eliminate dynamic points. Compared with using geometric information constraints, it is more accurate.

[0058] Image semantic segmentation processes image information captured by a visual sensor, labeling each pixel in the image according to a predefined semantic segmentation category and label. Essentially, it performs pixel-level detection on the image, clearly segmenting object boundaries. Therefore, semantic segmentation provides richer image information. Traditional semantic segmentation methods include threshold-based, region-based, and edge-based methods. Threshold-based methods, such as the maximum inter-class variance method and the adaptive threshold method, use the image's grayscale properties to calculate one or more grayscale values ​​and compare them with a predefined threshold to achieve image segmentation. Region-based methods utilize image similarity criteria to achieve image segmentation. Key segmentation methods include region growing and region splitting and merging. Edge-based methods detect edges in image grayscale values. The grayscale, color, and other characteristics of pixels along the dividing line between adjacent regions in the image undergo a sudden change. Edge detection operators include the Laplacian operator and the Canny operator.

[0059] In recent years, with the continuous development of deep learning, a growing number of researchers have designed a large number of new and efficient semantic segmentation algorithms. For example, the Fully Convolutional Network (FCN) has been proposed. FCN is a pioneering work in semantic segmentation based on deep learning. It replaces all the fully connected layers of the Convolutional Neural Network (CNN) with convolutions and restores the image dimension through upsampling. Another proposed real-time network, Segnet, effectively addresses the problem of FCN's inability to perform real-time inference. This network uses an encoder-decoder structure and upsampling by downsampling the maximum pooled position, improving the algorithm's computational speed. Deeplabv2 has also been proposed. Building on Deeplabv1, it introduces atrous spatial pyramid pooling (ASPP), improving the network's segmentation performance for objects of varying scales, but still relies on fully connected conditional random fields. Deeplabv3 has also been proposed, introducing a multi-grid strategy and an optimized ASPP structure, eliminating the reliance on fully connected conditional random fields. Finally, the Deeplabv3+ algorithm adopts an encoder-decoder structure, using the Deeplabv3 network structure as the encoder and introducing a decoder to obtain clearer segmentation boundaries. The HRNet network was also proposed, which adds a high-resolution feature main network to a low-resolution feature sub-network in parallel, realizing multi-scale fusion and feature extraction, and improving the prediction accuracy of the heat map.

[0060] Loop closure detection algorithms identify scenes by determining whether the robot has reached a previous location, thereby determining the similarity between scenes. Loop closure detection is crucial for pose estimation and map construction in large-scale scenes. Faulty loop closure detection can lead to the failure of the entire trajectory optimization process.

[0061] Generally, lidar-based loop detection methods can be roughly divided into three categories. The first category is point-to-point matching, which directly finds the position of the current point cloud in the global point cloud and then performs matching, such as the ICP method and its derivatives. The second category is a method similar to the Bag of Words (BoW) model, which uses point cloud feature descriptors to describe each keyframe and stores the results in the bag-of-words model of the lidar point cloud. Local descriptor methods generally describe the local information around the keypoints and use this information for similarity matching. Global descriptor methods, such as the proposed M2DP method, project the entire scanned 3D point cloud onto multiple 2D planes and extract a global position descriptor based on the spatial density distribution characteristics of the laser points on the planes. Methods such as Scan Context, LocNet, ISC, OverlapNet, and Scan Context++ convert 3D point clouds into 2D image representations and then find loops by comparing these 2D images. The third category is some segmentation-based methods. For example, SegMatch segments and clusters point clouds into different elements, extracts their local features, and combines local features with global features to perform fast and accurate scene recognition.

[0062] In recent years, with the continuous advancement of deep learning research in scene understanding, more and more researchers have been enhancing scene representation capabilities by using more advanced semantic features. GOSMatch proposed a new global descriptor that utilizes the spatial relationship between semantic information and proposes a coarse-to-fine strategy to effectively search for loops and achieve accurate 6-DOF initial pose estimation. The proposed SA-LOAM system also integrates a semantic-based loop detection method. This method maintains a semantic map for each frame of the point cloud and quickly identifies similar scenes based on the similarity score of the semantic map.

[0063] The technical solutions provided by various embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0064] Figure 1 The following is a flowchart of a method for simultaneous positioning and mapping of an unmanned vehicle in the present invention, which specifically includes the following steps:

[0065] S101: Inputting the image data of the environment surrounding the unmanned driving device acquired during the current acquisition cycle into a pre-trained semantic segmentation model for semantic segmentation, and determining the semantic information of each pixel in the image data.

[0066] S102: Acquire multi-frame point cloud data of the environment surrounding the unmanned driving device in the current acquisition cycle, and pre-integrate the inertial measurement unit data of the unmanned driving device in the current acquisition cycle to perform distortion correction on the multi-frame point cloud data.

[0067] S103: Determine the image data that is spatiotemporally synchronized with each frame of point cloud data, and project each frame of point cloud data onto the spatiotemporally synchronized image data; determine the semantic labels of the three-dimensional points in each frame of point cloud data based on the semantic information of the pixels in the image data and the projection of the point cloud data on the image.

[0068] S104: Based on the changes in the speed of the objects formed by the three-dimensional points corresponding to different semantic labels in each frame of point cloud data and the occupied space, the three-dimensional points corresponding to the dynamic objects in each frame of point cloud data are determined and removed to obtain multi-frame target point cloud data containing only static objects.

[0069] S105: stitching multi-frame target point cloud data to construct a local subgraph corresponding to the current acquisition cycle; selecting key frames from the multi-frame target point cloud data, and for each key frame, determining a loop candidate frame from historical key frames through geometric detection; when the loop candidate frame matches a preset semantic label in the key frame, optimizing the surrounding environment map and pose graph of the unmanned driving device based on the loop candidate frame and the key frame.

[0070] For the sake of convenience, the following description will only be based on the server as the execution subject. The server mentioned in the present invention can be a server set up on a business platform, or a device such as a desktop computer or a laptop computer that can execute the solution of the present invention.

[0071] The semantic SLAM framework proposed in this paper consists of two parts: a front-end odometer module and a back-end optimization module. Figure 2 As shown, Figure 2 This is a schematic diagram of a system framework in the present invention.

[0072] Front-end odometry module: Pre-integrates IMU data and establishes an error model. Pre-integration of IMU data eliminates motion distortion caused by LiDAR motion. Ground segmentation is then performed on the point cloud data of the surrounding environment acquired by the unmanned vehicle. Semantic segmentation is performed on the image data of the surrounding environment captured by the camera. The segmented image data is synchronized with the point cloud data of the LiDAR point cloud frame in time and space, and the point cloud data is projected onto the image data to obtain semantic information, achieving semantic segmentation of the LiDAR point cloud. Dynamic objects are then detected and removed by calculating the similarity score of the semantic labels. Edge and surface features are then extracted from the point cloud data after dynamic object removal. Keyframes are extracted and subgraphs are constructed.

[0073] The backend optimization module detects key semantic features and associates them with corresponding keyframes based on nearest neighbor timestamps. Within a certain spatial threshold of a keyframe with a key semantic feature, it determines whether other keyframes with the same key semantic feature label exist. If so, this is considered a loop. Finally, pose graph optimization is used to correct global errors, achieving globally consistent pose estimation and constructing a highly accurate and robust 3D semantic map.

[0074] The following describes the steps involved in the above process, including IMU pre-integration:

[0075] The present invention follows the IMU pre-integration derivation to obtain the time t m and t n The relative motion between m is the lower bound of the current acquisition cycle, t n The upper limit of the current acquisition cycle. The location of the unmanned device speed and rotation The state can be determined by IMU in the time interval [t m ,t n ] to propagate the measured values ​​within the current acquisition cycle of the unmanned driving device, the inertial measurement unit data within the current acquisition cycle can be pre-integrated using the following formula:

[0076]

[0077] Where, is the rotation matrix, The acceleration measurement data of the unmanned driving equipment, is the deviation of the acceleration sensor of the unmanned driving equipment, The measurement data of the angular velocity of the unmanned driving equipment, is the deviation of the gyroscope of the unmanned driving device, and Ω is the antisymmetric matrix of the acceleration of the unmanned driving device.

[0078] Point cloud data distortion correction:

[0079] Due to the rotational mechanism within the LiDAR, the LiDAR can become out of sync during measurement, resulting in motion distortion in the acquired point cloud data. This method integrates the data collected by the IMU to determine the current position and velocity of the LiDAR on the unmanned vehicle. The acquired timestamp information is then linearly interpolated to determine the corresponding pose change at each moment. This is then unified into the base coordinate system to correct for point cloud distortion.

[0080] Specifically, for each frame of point cloud data, the inertial measurement unit data of the unmanned driving device during the period of collecting the frame of point cloud data can be pre-integrated to determine the position change of the lidar on the unmanned driving device during the period of collecting the frame of point cloud data. Then, based on the position change of the lidar during the period of collecting the frame of point cloud data, the position change of the point cloud data at any time during the period of collecting the frame of point cloud data relative to the initial time during the period of collecting the frame of point cloud data can be obtained through linear interpolation. Finally, based on the position change of the point cloud data at any time during the period of collecting the frame of point cloud data relative to the initial time during the period of collecting the frame of point cloud data, the coordinates of each 3D point in the frame of point cloud data are converted to the coordinate system of the first 3D point collected in the frame of point cloud data, thereby completing the distortion correction of the frame of point cloud data.

[0081] Through IMU pre-integration, the laser radar can be obtained at [t i ,t i+1 ]The posture change at the moment By linear interpolation, we can get the value of any time t k ∈[t i ,t i+1 ] relative to t i The posture change at the moment is shown in the following formula:

[0082] By changing the posture, t k The three-dimensional point coordinates at the moment are converted to the coordinate system of the first collected three-dimensional point, as shown in the following formula: P' k =T k P k .

[0083] Where, P k It is t k The position of the three-dimensional point at time, P′ k It's P k The position in the first 3D point coordinate system.

[0084] Laser point cloud segmentation:

[0085] When the laser radar collects features of the surrounding environment, it will obtain a large amount of point cloud information, especially in open scenes where there are many ground points generated by roads. Since ground points can be ignored in map reconstruction, and during the inter-frame matching process, ground point information processing consumes a lot of computing power, which reduces the efficiency of the algorithm. Therefore, the present invention performs ground segmentation processing on the original point cloud information input by the laser radar, and performs feature extraction on point cloud information other than ground points. Specifically, for each frame of distortion-corrected point cloud data, the distance image of the frame point cloud data can be determined, and the three-dimensional points belonging to the ground and the three-dimensional points belonging to non-ground in the point cloud data can be determined based on the pitch angle of the distance image, and then the three-dimensional points belonging to the ground in the frame point cloud data are removed. Figure 3 As shown, Figure 3 This is a schematic diagram of segmenting ground three-dimensional points of point cloud data in the present invention. Figure 3 The first row is the original point cloud, the first column of the second row is the 3D points belonging to the ground, and the second column of the second row is the point cloud data after removing the 3D points belonging to the ground.

[0086] The point cloud data collected by the LiDAR at a certain moment is projected onto a two-dimensional image to record the position of the point cloud data in the LiDAR coordinate system. This image is called a range image. By calculating the pitch angle of the range image, each three-dimensional point in the point cloud data is divided into ground points and non-ground points. Select point P in a column of the range image. i (x i ,y i ,z i ) and P j (x j ,y j ,z j ), the pitch angle Angle of the straight line formed by the two points is: Angle = atan 2 (diffz,ds),ds=sqrt(diffx 2 +diffy 2 ). Where, diffx=x j -x i , diffy=y j -y i , diffz=z j -z i .

[0087] Feature extraction:

[0088] This paper uses a feature extraction method similar to LOAM, using curvature as a way to distinguish edge features from surface features. Assume that point i is any point in the k-th frame point cloud, and S is the set of consecutive points i returned by the lidar in the same frame. The curvature calculation formula at point i is as follows: in, Represents the coordinates of the i-th point and the j-th point in the k-th frame point cloud in the lidar coordinate system.

[0089] The 3D points in the scan are sorted according to the c value. If the curvature c is larger, it means that the local plane of the 3D point is more uneven, which conforms to the edge feature, that is, the point is on a sharp edge in the 3D space. If the curvature c is smaller, it means that the local plane of the point is relatively flat, which conforms to the plane feature, that is, the point is on a smooth plane in the 3D space. The edge points and plane points extracted at time k are represented as N ei and N pi . Then the edge points and plane points can form a set of N i, that is, N i ={N ei ,N pi}.

[0090] Frame to subimage registration:

[0091] The transformation of each keyframe can be used to transform the features into the local odometry coordinate system. After the transformation, the keyframes are spliced ​​to form a local subgraph. Using the robot motion state predicted by the IMU, the feature points of the current frame are transformed into the previous frame or the global coordinate system to obtain the transformed feature set. The feature association between the edge features and surface features in the feature set in the local subgraph is then found. The distance d from the point to the line is then constructed. e and point-to-surface d p The distance between the frame and the point in the subgraph is d. e The distance d from the point to the surface p Composed of distance matrix d. Based on the scan matching method, the cost function can be established To solve the optimal pose transformation relationship from frame to sub-image

[0092] For dynamic objects in the driving environment of unmanned driving equipment, semantic segmentation can be performed on the point cloud data first, and then the three-dimensional points belonging to dynamic objects can be identified and eliminated based on the semantic segmentation results.

[0093] When performing semantic segmentation of point cloud data, the point cloud data and the image data after semantic segmentation can be synchronized in time and space to achieve point cloud semantic segmentation.

[0094] When performing semantic segmentation on image data, an existing mature semantic segmentation model can be used. Of course, in order to improve the accuracy of positioning and mapping of unmanned driving equipment, in one or more embodiments of the present invention, the backbone network of its encoder can be replaced with a RefineNet network based on the Deeplabv3+ algorithm, and a depth-separable dilated convolution can be used in the ASPP module of the RefineNet network. The convolution layer and the NAM module are sequentially connected in series after the ASPP module, and then combined with the decoder of the Deeplabv3+ algorithm to construct a semantic segmentation model for the image. Therefore, when the image data of the environment around the unmanned driving equipment obtained in the current acquisition cycle is input into the pre-trained semantic segmentation model for semantic segmentation, the image data of the environment around the unmanned driving equipment obtained in the current acquisition cycle can be input into the pre-trained semantic segmentation model. The RefineNet network in the semantic segmentation model extracts and aggregates the multi-scale features of the environmental objects in the image data, and the NAM module performs attention weighting based on the channel to obtain the deep features of the environmental objects; the decoder fuses the shallow features of the environmental objects in the encoder with the deep features and refines the fused features to obtain the final semantic segmentation result.

[0095] In the image semantic segmentation task, feature extraction plays a key role in the final segmentation effect, so we consider replacing the backbone network of the encoder of the Deeplabv3+ algorithm.

[0096] Typically, the environment around unmanned vehicles is complex during driving, with objects in the environment having different spatial distributions, such as pedestrians and vehicles, and similar objects of the same type being distributed in different states, such as pedestrians and cyclists, roads and curbs. The RefineNet network, through its multi-scale feature fusion capabilities, can effectively integrate information at different scales from the unmanned vehicle's surroundings, resulting in more accurate image segmentation results. At the same time, RefineNet's refined network structure retains sufficient detail to better distinguish similar objects in the unmanned vehicle's surroundings, reducing segmentation errors. Furthermore, the Chained Residual Pooling in RefineNet helps capture a wide range of contextual information, which is crucial for understanding complex scenes and relationships in image data.

[0097] Four sets of comparative experiments were conducted on the cityscapes dataset, testing the performance of different backbone networks against the improved Deeplabv3+ algorithm. Mean intersection over union (mIOU) and mean pixel accuracy (mPA) were used as metrics to evaluate the adaptability of the different backbone networks. The experimental results are shown in Table 1. As can be seen, replacing the backbone network with RefineNet achieved mIoU of 82.63% and mPA of 85.57%. The remaining backbone networks performed worse than RefineNet in both metrics, demonstrating that RefineNet effectively optimizes the model and improves the accuracy of feature extraction.

[0098] Table 1 Comparison results of different backbone networks

[0099]

[0100] Simultaneous localization and mapping for autonomous vehicles often require fast execution on resource-constrained platforms. Depthwise Separable Dilated Convolution (DSDConv) combines the advantages of dilated and depthwise separable convolutions, significantly reducing the number of parameters by decomposing standard convolution into depthwise and pointwise convolutions. This significantly reduces the number of parameters and computational complexity for semantic segmentation of image data. Depthwise convolution processes each input channel independently, while pointwise convolution integrates features through 1×1 convolutions.

[0101] In addition, dilated convolution allows the use of smaller convolution kernels while increasing the receptive field, that is, dilated convolution can expand the receptive field without incurring additional computational cost, which enables the network to capture more contextual information while maintaining the resolution of the feature map, which is very useful for the task of semantic segmentation of images of the environment around unmanned vehicles.

[0102] This method reduces the number of parameters and computational complexity, improving model efficiency. Two sets of comparative experiments were conducted based on the RefineNet backbone network. The experimental results are shown in Table 2. As can be seen, the introduction of depthwise separable dilated convolution improves the mean Intersection Over Union (MIoU) while significantly reducing the single image prediction time (SPPT).

[0103] Table 2 Comparison of ASPP improvement experiments

[0104]

[0105] The essence of the attention mechanism is to enable convolutional neural networks to focus on important feature vectors and ignore unimportant feature information by calculating corresponding weights. This approach improves the accuracy of fitting results while avoiding interference from useless features, and also improves computational efficiency to a certain extent. This paper introduces the NAM module, based on the ASPP module of the RefineNet backbone network. The NAM module captures global contextual information through non-local operations, which is crucial for understanding the complex layout and relationships in autonomous driving scenarios. It can improve the model's understanding of global information, thereby improving segmentation accuracy. Furthermore, by introducing a non-local attention mechanism into feature maps, the NAM module enhances feature representation capabilities, enabling the model to better capture subtle semantic information, which is particularly important for image segmentation tasks. Furthermore, in autonomous driving environments, different objects may have long-range dependencies. The NAM module can effectively model these long-range dependencies, helping the network better understand the scene. Furthermore, reducing misclassification is crucial in autonomous driving image segmentation. By considering global context, the NAM module helps reduce misclassification problems caused by insufficient local information.

[0106] This paper introduces four different attention mechanisms for comparison. The experimental results are shown in Table 3. The NAM module improves performance over both SE and CBAM. Compared to ECA, while slightly slower, it achieves a 2.32% improvement in mIOU. Therefore, this paper selects the more powerful NAM module to assist the model in better completing semantic segmentation tasks.

[0107] Table 3 Comparison results of different attention mechanisms

[0108]

[0109] In summary, the present invention adopts the representative algorithm Deeplabv3+ as the basic framework of image segmentation for improvement and optimization. First, the RefineNet network is used as the backbone network of the encoder of the Deeplabv3+ algorithm, and the depth-separable dilated convolution is introduced in the ASPP module to reduce the number of model parameters while improving the prediction speed. After the backbone network extracts the deep feature map and the shallow feature map, the lightweight and efficient attention module NAM is introduced to enhance the channel characteristics of the network model. The traditional Deeplabv3+ algorithm uses the linear interpolation upsampling method to deal with the problem of different dimensions of deep feature maps and shallow feature maps, but when the unmanned vehicle collects environmental image information, there are many nonlinear relationships between pixels with large scale changes. Deconvolution achieves high-precision upsampling through parameter learning, so the present invention uses deconvolution to replace linear interpolation. As Figure 4 As shown, Figure 4 This is a schematic diagram of a semantic segmentation network model in the present invention.

[0110] External calibration can be used to obtain the rotation and translation matrices between the LiDAR and camera, and the LiDAR and IMU, projecting the observations of each sensor into the same coordinate system for spatial synchronization. Camera intrinsic calibration can be performed using Zhang's calibration method, while external calibration of the LiDAR and IMU can follow hand-eye calibration. Calibration tool kits can be used for joint calibration of the LiDAR and camera.

[0111] The laser radar and camera establish a mapping relationship between point cloud data and image data after semantic segmentation through external parameter calibration, that is, the pixel coordinates corresponding to each 3D point cloud are obtained. After the 2D image collected by the camera is processed by semantic segmentation, the semantic labels of the same type of objects are consistent, and the pixel coordinates correspond to the semantic labels one by one, thus establishing a mapping relationship between 3D point cloud data and semantic labels, thereby obtaining a 3D point cloud with semantic information. Figure 5 As shown, Figure 5 This is a schematic diagram of single-frame point cloud semantic segmentation in the present invention. Figure 5 The upper part is the original point cloud data, and the lower part is the corresponding point cloud data with semantic information. The three-dimensional points corresponding to objects with different semantics are marked with different colors. Figure 5 As can be seen from the figure, the point cloud data after semantic segmentation can clearly distinguish trees, cars, lawns and the ground.

[0112] Culling dynamic objects:

[0113] In the previous section, semantic segmentation has been performed on the point cloud data, and objects such as cars and people have semantic labels in the point cloud. The present invention calculates the center of mass of potential dynamic objects with semantic labels, and determines and removes dynamic objects through the following steps. The dynamic object detection and removal flow chart of the present invention is as follows: Figure 6 As shown, Figure 6 This is a flow chart of dynamic object detection and elimination in the present invention.

[0114] Specifically, the center of mass of the object formed by the three-dimensional points corresponding to different semantic labels in each frame of point cloud data can be calculated separately. Then, based on the distance between the center of mass of the objects corresponding to the same semantic label between each frame of point cloud data, the semantic label pairs corresponding to the objects that match between each frame of point cloud data are determined. At the same time, according to the time sequence of each frame of point cloud data, the speed change of the center of mass of different objects is determined, and the three-dimensional points corresponding to dynamic objects are determined according to the preset speed threshold. Then, for each semantic label pair, the overlapping area of ​​the three-dimensional points corresponding to the semantic label pair in each frame of point cloud data is determined, and the three-dimensional points corresponding to dynamic objects are determined according to the preset area threshold. And the similarity between the distribution of the three-dimensional points corresponding to the semantic label pair in each frame of point cloud data is determined, and the three-dimensional points corresponding to dynamic objects are determined according to the preset similarity threshold. Finally, the three-dimensional points corresponding to dynamic objects in each frame of point cloud data are removed.

[0115] a. Calculate the centroid of the semantic labels of various potential dynamic objects in the current frame and the reference frame respectively.

[0116] b. Calculate the distance between the centroids of the same semantic labels between each frame of point cloud data, and match the labels with the closest distance. If a new object appears in the scene or an error occurs in semantic recognition, an incorrect correspondence will occur. Therefore, the present invention calculates its speed and constrains it to reduce incorrect correspondences. The present invention uses the speed of the centroid to remove some more obvious incorrect matching relationships. A larger preset speed threshold can be set, and those that exceed the preset speed threshold will be removed.

[0117] c. Calculate the overlap area between the 3D points corresponding to the semantic label pairs in each frame of point cloud data. If there is no overlap or the 3D points in the overlap are too sparse, the object is considered to have changed position and is directly identified as a dynamic object without further evaluation. Then, project the 3D points corresponding to the semantic labels in the overlap onto a 2D grid and assign a value to each grid cell to describe the pseudo occupancy.

[0118] d. Calculate the similarity score between corresponding semantic labels. In a given two-dimensional network, it is necessary to define and calculate the similarity distance between two corresponding relationships. Use the Spearman correlation coefficient to calculate the column vectors corresponding to the two semantic labels. and The distance between them. Then sum up all the distances as the similarity distance between the two corresponding relations. Finally, according to the number of columns V s The similarity distance is normalized. The calculation formula of the similarity score is as follows: Among them, σ() represents the standard deviation, cov() represents the covariance, C C Represents the current semantic label, R X represents the target semantic label, V S Represents the number of vector pairs.

[0119] If the corresponding label similarity score is greater than the preset similarity threshold, it is considered to be static in the current frame and the reference frame. Otherwise, it is considered to be a dynamic object and is removed.

[0120] Loopback detection:

[0121] Loop detection optimization plays a great role in obtaining a globally consistent map. In combination with the geometry-based method, this paper designs a new loop detection method, such as Figure 7 As shown, Figure 7This is a schematic diagram of a loop detection process in the present invention. The principle of this method is that if a key semantic feature is detected in a key frame and is within the spatial threshold range, and the same type of key semantic feature exists in the previous trajectory, then it is considered that the unmanned vehicle has passed through the previous trajectory again. The trajectory points corresponding to the two key semantic features form a closed loop. Given a key frame semantic point cloud P L , select representative semantic features according to semantic labels, such as buildings, traffic lights, traffic signs, etc. Then the extracted point cloud Mapped onto the xy plane, assuming One point in The transformed three-dimensional point can be expressed as:

[0122] Where s i is the semantic label of the 3D point, l i is the distance from the 3D point to the scanning origin, δ i is the azimuth of the point.

[0123] Next, the key frame is selected. The first frame of point cloud data is used as the key frame, and the rest are judged based on the relative posture between consecutive frames. Let the posture of the m-1 key frame be T m-1 And the pose of the mth key frame is T m , its relative posture is T m-1,m According to the conversion formula of rotation matrix and Euler angle, the relative pose T m-1,m The rotation matrix R in m-1,m Converted to incremental attitude angle {Δδ x ,Δδ y ,Δδ z}, if the subsequent frame meets one of the following two threshold conditions, the point cloud data of the frame is extracted as the key frame. (1) Δδ z Greater than the angle threshold (2) Posture increment on the XOY plane Greater than the distance threshold

[0124] After the keyframes are selected, a suitable loop candidate frame is selected for each keyframe from the historical keyframes through geometric detection. When processing the current keyframe, it needs to meet three threshold conditions at the same time to be selected as a loop candidate frame. (1) The spatial position is less than the regional threshold l r (2) The cycle length must be greater than the length threshold l d (3) The cosine value of the angle between the normal vectors of the two frame point clouds must be greater than the angle threshold l α .like Figure 8 As shown, Figure 8 Schematic diagram of a geometric detection threshold in the present invention.

[0125] Afterwards, the present invention uses key semantic features to perform loop detection on the key frame and its corresponding loop candidate frame. All key semantic feature detection results are mapped to the key frame closest in time. After this step, each key frame has its own key semantic feature label. The key frame with the key semantic feature label is compared with the key semantic feature labels in all loop candidate frames. If the same key semantic feature label exists, it is considered that a loop is formed between the two frames. That is to say, if the unmanned vehicle detects a traffic light at the time and place of the key frame, and the historical data also detects a traffic light within the spatial threshold, it can be determined that the unmanned vehicle has passed the same traffic light and returned to the loop closure point.

[0126] After detecting the loop closure, it is determined whether this is a reverse cycle based on the direction of the odometer. If it is a reverse cycle, the yaw angle θ is quickly solved by differential translation. The point cloud is then accurately transformed through ICP matching calculation. Finally, for the accuracy of matching, the present invention performs re-matching verification on the point cloud. The present invention calculates the ratio of the area of ​​the overlapping area, which is the ratio of the area of ​​the overlapping area to the total area of ​​the two point clouds. If the ratio is less than the set threshold, the loop is discarded. In the experiment of the present invention, this threshold is set to 0.75. Finally, the optimal loop candidate key frame is selected by comparing the loop candidate key frame with the currently aligned key frame. The key frame and the loop candidate key frame are added as nodes to the SLAM graph optimization.

[0127] Map building:

[0128] If a frame of point cloud data is detected as a key frame, the environment map of the current frame needs to be updated. The main method is to convert the real-time point cloud information of the key frame into the world coordinate system through coordinate transformation. When the k+1 frame is selected as the sub-key frame, the continuous sub-key frame optimization is performed using ICP, and the relative posture change between the sub-key frames is T k+1 The relative change from the kth sub-keyframe to the world coordinate system is The relative change from the k+1th sub-keyframe to the world coordinate system can be obtained as Finally, the point cloud coordinates of the k+1 sub-keyframe are converted to the world coordinate system, and the converted sub-keyframes are merged into a voxel map {M} i middle.

[0129] Pose graph optimization:

[0130] Pose graph optimization aims to eliminate the cumulative error of the LiDAR odometry and maintain global consistency. Each keyframe pose is used as the variable to be optimized, and the relative poses and loops between adjacent keyframes in the LiDAR odometry are used as constraints to construct a pose graph model, that is, to solve the least squares problem, as shown in the following formula:

[0131] Since the objective function is a nonlinear function, the Levenberg-Marquardt (LM) optimization method can be used to solve it and obtain the optimal pose of each key frame. The optimized pose is then used to correct the point cloud map to ensure the global consistency of the map.

[0132] The present invention improves the efficiency and performance of the semantic segmentation model by improving the semantic segmentation model. Through IMU pre-integration, the laser radar point cloud data is distorted and kinematic constraints are provided between the state quantities of consecutive frames, thereby realizing tight coupling between the IMU and the laser radar. The improved semantic segmentation model is used to obtain image semantic information, and the improved semantic segmentation model is synchronized with the laser radar point cloud data in time and space to obtain a three-dimensional point cloud with semantic information, thereby realizing the removal of dynamic three-dimensional points. Key semantic features are used to perform loop detection and verification on key frames and their corresponding loop candidate frames, and the optimal loop candidate key frames and key frames are selected as nodes to be added to the SLAM graph optimization, thereby greatly improving the positioning accuracy and robustness of the SLAM system.

[0133] This paper proposes a semantic SLAM framework that integrates lidar, IMU, and camera. By introducing semantic information, it enables the detection and elimination of dynamic target objects. A semantically informed loop detection method is also proposed, which effectively detects loops and eliminates accumulated errors, thereby improving the positioning accuracy of the entire system in complex scenarios and constructing high-quality maps with semantic information. An improved image semantic segmentation algorithm based on Deeplabv3+ is proposed. By replacing the backbone network and introducing depthwise separable dilated convolutions in the ASPP module, along with an attention mechanism, the performance of the image segmentation algorithm is improved and the accuracy of point cloud semantic segmentation is guaranteed.

[0134] When applying the method for simultaneous positioning and mapping of unmanned vehicles provided by the present invention, it is not necessary to Figure 1 The steps are executed in the order shown. The specific execution order of the steps can be determined according to needs, and the present invention does not limit this.

[0135] The above is a method for simultaneous positioning and mapping of unmanned vehicles provided by one or more embodiments of the present invention. Based on the same idea, the present invention also provides a corresponding device for simultaneous positioning and mapping of unmanned vehicles, such as Figure 9 shown.

[0136] Figure 9 A schematic diagram of an unmanned driving simultaneous positioning and mapping device provided by the present invention includes:

[0137] The image segmentation module 201 is used to input the image data of the environment around the unmanned driving device acquired during the current acquisition cycle into a pre-trained semantic segmentation model for semantic segmentation, and respectively determine the semantic information of each pixel in the image data;

[0138] The correction module 202 is used to obtain multiple frames of point cloud data of the environment around the unmanned driving device in the current acquisition cycle, and pre-integrate the inertial measurement unit data of the unmanned driving device in the current acquisition cycle to perform distortion correction on the multiple frames of point cloud data;

[0139] The point cloud segmentation module 203 is used to determine image data that is spatiotemporally synchronized with each frame of point cloud data, and project each frame of point cloud data onto the spatiotemporally synchronized image data; determine the semantic labels of the three-dimensional points in each frame of point cloud data based on the semantic information of the pixels in the image data and the projection of the point cloud data on the image;

[0140] The dynamic culling module 204 is configured to determine and remove the 3D points corresponding to dynamic objects in each frame of point cloud data based on the speed and space occupied by the 3D points corresponding to different semantic labels in each frame of point cloud data, thereby obtaining multi-frame target point cloud data containing only static objects.

[0141] The loop detection module 205 is used to stitch together multiple frames of target point cloud data to construct a local subgraph corresponding to the current acquisition cycle; select key frames from the multiple frames of target point cloud data, and for each key frame, determine a loop candidate frame from the historical key frames through geometric detection; when the loop candidate frame matches the preset semantic label in the key frame, the environment map and pose graph of the unmanned driving device are optimized based on the loop candidate frame and the key frame.

[0142] For the specific definition of the device for simultaneous positioning and mapping of unmanned vehicles, please refer to the definition of the method for simultaneous positioning and mapping of unmanned vehicles above, and will not be repeated here. The various modules in the above-mentioned device for simultaneous positioning and mapping of unmanned vehicles can be implemented in whole or in part by software, hardware, and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0143] The present invention also provides a computer-readable storage medium, which stores a computer program, which can be used to execute the above Figure 1 The provided unmanned driving simultaneous positioning and mapping method.

[0144] The present invention also provides Figure 10 The structural diagram of the computer equipment shown in FIG. Figure 10As shown in the figure, at the hardware level, the computer device includes a processor, an internal bus, a network interface, a memory, and a non-volatile memory. Of course, it may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it to achieve the above Figure 1 The provided unmanned driving simultaneous positioning and mapping method.

[0145] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0146] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of the present invention.

Claims

1. A method for simultaneous positioning and mapping of unmanned vehicles, characterized in that: include: The image data of the environment surrounding the unmanned driving device acquired during the current acquisition cycle is input into a pre-trained semantic segmentation model for semantic segmentation, and the semantic information of each pixel in the image data is determined; Acquire multi-frame point cloud data of the environment around the unmanned driving device during the current acquisition cycle, and pre-integrate the inertial measurement unit data of the unmanned driving device during the current acquisition cycle to perform distortion correction on the multi-frame point cloud data; Determine the image data that is spatiotemporally synchronized with each frame of point cloud data, and project each frame of point cloud data onto the spatiotemporally synchronized image data; determine the semantic labels of the three-dimensional points in each frame of point cloud data based on the semantic information of the pixels in the image data and the projection of the point cloud data on the image; Based on the changes in the speed and space occupied by the 3D points corresponding to different semantic labels in each frame of point cloud data, the 3D points corresponding to dynamic objects in each frame of point cloud data are determined and removed to obtain multi-frame target point cloud data containing only static objects; Splice multiple frames of target point cloud data to construct a local sub-image corresponding to the current acquisition cycle; Keyframes are selected from multi-frame target point cloud data. For each keyframe, loop candidate frames are determined from historical keyframes through geometric detection. When the loop candidate frame matches the preset semantic label in the keyframe, the surrounding environment map and pose graph of the unmanned driving device are optimized based on the loop candidate frame and the keyframe.

2. The method for simultaneous positioning and mapping of an unmanned vehicle according to claim 1, wherein: The semantic segmentation model is based on the Deeplabv3+ algorithm. The backbone network of its encoder is replaced with the RefineNet network. Depthwise separable dilated convolution is used in the ASPP module of the RefineNet network. The convolutional layer and the NAM module are connected in series after the ASPP module, and then combined with the decoder of the Deeplabv3+ algorithm to construct the model. The image data of the environment surrounding the unmanned driving device acquired during the current acquisition cycle is input into a pre-trained semantic segmentation model for semantic segmentation, specifically including: The image data of the unmanned vehicle's surroundings acquired during the current acquisition cycle is fed into a pre-trained semantic segmentation model. The RefineNet network in the semantic segmentation model extracts and aggregates multi-scale features of environmental objects in the image data. The NAM module then performs channel-based attention weighting to obtain deep features of the environmental objects. The decoder fuses the shallow features of the environmental objects in the encoder with the deep features and refines the fused features to obtain the final semantic segmentation result.

3. The method for simultaneous positioning and mapping of an unmanned vehicle according to claim 1, wherein: The pre-integration of the inertial measurement unit data within the current acquisition cycle of the unmanned driving device specifically includes: The inertial measurement unit data in the current acquisition cycle of the unmanned driving device is pre-integrated using the following formula: in, The location of the unmanned equipment, For the speed of unmanned equipment, is the rotation of the unmanned equipment, t m is the lower bound of the current acquisition cycle, t n is the upper bound of the current acquisition cycle, t is the time in the current acquisition cycle, is the rotation matrix, The acceleration measurement data of the unmanned driving equipment, is the deviation of the acceleration sensor of the unmanned driving equipment, The measurement data of the angular velocity of the unmanned driving equipment, is the deviation of the gyroscope of the unmanned driving device, and Ω is the antisymmetric matrix of the acceleration of the unmanned driving device.

4. The method for simultaneous positioning and mapping of an unmanned vehicle according to claim 3, wherein: The pre-integration of the inertial measurement unit data within the current acquisition cycle of the unmanned driving device to perform distortion correction on the multi-frame point cloud data specifically includes: For each frame of point cloud data, the position change of the lidar on the unmanned driving device during the period of collecting the frame of point cloud data is determined by pre-integrating the inertial measurement unit data of the unmanned driving device during the period of collecting the frame of point cloud data; According to the position change of the laser radar during the period of collecting the frame of point cloud data, the position change of the point cloud data at any time during the period of collecting the frame of point cloud data relative to the initial time during the period of collecting the frame of point cloud data is obtained by linear interpolation; According to the change in the posture of the point cloud data at any time during the period of collecting the frame of point cloud data relative to the initial time during the period of collecting the frame of point cloud data, the coordinates of each three-dimensional point in the frame of point cloud data are converted to the coordinate system of the first three-dimensional point collected in the frame of point cloud data, thereby completing the distortion correction of the frame of point cloud data.

5. The method for simultaneous positioning and mapping of an unmanned vehicle according to claim 1, wherein: The method further comprises: For each frame of distortion-corrected point cloud data, determine the range image of the frame of point cloud data, and determine the 3D points belonging to the ground and the 3D points belonging to non-ground in the point cloud data based on the pitch angle of the range image; Remove the 3D points belonging to the ground in the point cloud data of this frame.

6. The method for simultaneous positioning and mapping of an unmanned vehicle according to claim 1, wherein: The method of determining and removing the three-dimensional points corresponding to the dynamic objects in each frame of point cloud data according to the change in the speed of the objects formed by the three-dimensional points corresponding to different semantic labels in each frame of point cloud data and the change in the space occupied by the objects specifically includes: Calculate the center of mass of the object formed by the three-dimensional points corresponding to different semantic labels in each frame of point cloud data; Determine the semantic label pairs corresponding to the objects that match between the point cloud data frames based on the distances between the centroids of the objects with the same semantic labels between the point cloud data frames. According to the time sequence of each frame of point cloud data, the velocity change of the center of mass of different objects is determined, and the 3D point corresponding to the dynamic object is determined according to the preset velocity threshold; For each semantic label pair, determine the overlapping area of ​​the 3D points corresponding to the semantic label pair in each frame of point cloud data, and determine the 3D point corresponding to the dynamic object based on a preset area threshold; Determine the similarity between the distribution of corresponding 3D points of the semantic label pairs in each frame of point cloud data, and determine the 3D points corresponding to the dynamic objects based on a preset similarity threshold; Remove the 3D points corresponding to dynamic objects in each frame of point cloud data.

7. The method for simultaneous positioning and mapping of an unmanned vehicle according to claim 1, wherein: The method comprises selecting key frames from multiple frames of target point cloud data, determining loop candidate frames from historical key frames through geometric detection for each key frame, and optimizing the environment map and pose graph of the unmanned driving device based on the loop candidate frames and the key frames when the loop candidate frames match the preset semantic labels in the key frames. Specifically, the method comprises: Select key frames from multi-frame point cloud data, and detect preset key semantic labels from the semantic labels of the three-dimensional points of each key frame; For each keyframe, determine the loop candidate frame of the keyframe from the historical keyframes based on geometric constraints; Determine whether the key frame and the corresponding loop candidate frame have the same preset key semantic label; If so, it indicates that there is a loop, and the surrounding environment map and pose graph of the unmanned driving device are optimized based on the loop candidate frame and the key frame.

8. An unmanned driving simultaneous positioning and mapping device, characterized in that: include: An image segmentation module is used to input the image data of the environment around the unmanned driving device acquired during the current acquisition cycle into a pre-trained semantic segmentation model for semantic segmentation, and to determine the semantic information of each pixel in the image data; The correction module is used to obtain multi-frame point cloud data of the environment around the unmanned driving device in the current acquisition cycle, and pre-integrate the inertial measurement unit data in the current acquisition cycle of the unmanned driving device to perform distortion correction on the multi-frame point cloud data; The point cloud segmentation module is used to determine the image data that is spatiotemporally synchronized with each frame of point cloud data, and project each frame of point cloud data onto the spatiotemporally synchronized image data; based on the semantic information of the pixels in the image data and the projection of the point cloud data on the image, the semantic labels of the 3D points in each frame of point cloud data are determined; The dynamic culling module is used to determine and remove the 3D points corresponding to dynamic objects in each frame of point cloud data based on the speed and space occupied by the 3D points corresponding to different semantic labels in each frame of point cloud data, thereby obtaining multi-frame target point cloud data containing only static objects; The loop detection module is used to stitch together multiple frames of target point cloud data to construct a local sub-graph corresponding to the current acquisition cycle; Keyframes are selected from multi-frame target point cloud data. For each keyframe, loop candidate frames are determined from historical keyframes through geometric detection. When the loop candidate frame matches the preset semantic label in the keyframe, the surrounding environment map and pose graph of the unmanned driving device are optimized based on the loop candidate frame and the keyframe.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and capable of running on the processor, wherein when the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

Images