A multimodal SLAM method, system, device, medium and program product

Abstract

The present invention discloses a multimodal SLAM method, system, device, medium and program product. First, a lightweight semantic segmentation network is used to segment the current image frame to obtain the semantic mask of the current image frame, and then correction and projection are performed on the three-dimensional point cloud to obtain the point cloud map of the current image frame in the radar coordinate system, and point cloud segmentation is performed to obtain multiple point cloud sets with different semantic classification labels; then the saliency mask of the current image frame is predicted and the visual feature points of the current image frame are extracted to obtain the depth value of the visual feature points; finally, the visual feature points are tracked and a local map is constructed; at the same time, the Scan‑Context descriptor and similarity distance are used to perform loop detection and global BA on the point cloud set and construct a global map. This application comprehensively utilizes multiple sensor data to improve the perception ability of environmental features and effectively deal with the problems of insufficient perception and reduced positioning accuracy in complex environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of simultaneous positioning and mapping, and specifically relates to a multimodal SLAM method, system, equipment, medium and program product. Background Art

[0002] In modern autonomous navigation and environmental perception technologies, multimodal simultaneous localization and mapping (SLAM) methods play a vital role. This method integrates data from different sensors, such as lidar, cameras, and inertial measurement units (IMUs), to achieve comprehensive perception and precise positioning of the surrounding environment. However, traditional multimodal SLAM methods still face many challenges in practical applications.

[0003] First, moving objects in dynamic environments (such as pedestrians and vehicles) can interfere with the SLAM system, resulting in inaccurate positioning and map construction. Although existing technologies solve the mapping problem in dynamic environments by combining semantic segmentation with SLAM, this method only integrates semantic information into SLAM in a loosely coupled manner, and fails to fully utilize semantic information to achieve more accurate dynamic target detection. Secondly, rapid changes in the environment (such as changes in light and weather) will affect the data quality of the sensor, further reducing the reliability and stability of the SLAM system; current technologies usually treat these data equally when processing them, without effectively distinguishing based on data quality, resulting in low-quality data and high-quality data being treated equally, thereby weakening the system's ability to adapt to environmental changes. In addition, traditional methods often rely on preset parameters and models, and it is difficult to cope with the diverse needs in different scenarios and complex environments. Therefore, there is an urgent need for a multimodal SLAM method that can better adapt to dynamic changes and complex environments to improve the stability and robustness of the system. Summary of the invention

[0004] The main purpose of the present invention is to overcome the shortcomings and deficiencies of the prior art and to provide a multimodal SLAM method, system, device, medium and program product. By introducing advanced semantic segmentation and saliency prediction technology, as well as instance loop detection based on point cloud segmentation, the present invention comprehensively utilizes data from multiple sensors to improve the perception ability of environmental features, and effectively addresses the problems of insufficient perception and reduced positioning accuracy of traditional methods in complex environments.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] The first object of the present invention is to provide a multimodal SLAM method, comprising the following steps:

[0007] S1, using the monocular camera and multi-line radar mounted on the mobile platform to obtain the current image frame and current point cloud frame of the surrounding environment;

[0008] S2. Use a lightweight semantic segmentation network to segment the current image frame to obtain a semantic mask of the current image frame; at the same time, use the camera pinhole model and the transformation matrix to correct the semantic mask of the current image frame and project it onto the three-dimensional point cloud to obtain a point cloud map of the current image frame in the radar coordinate system, and then use a lightweight semantic segmentation network to segment the point cloud map of the current image frame in the radar coordinate system to obtain multiple point cloud sets with different semantic classification labels; the lightweight semantic segmentation network is a semantic segmentation model based on lightweight Transformer multi-attention fusion;

[0009] S3, performing object masking on the current image frame and predicting a saliency mask of the current image frame in combination with the semantic mask of the current image frame;

[0010] S4, dynamically adjusting the extraction threshold of the visual ORB feature extraction algorithm according to the saliency mask of the current image frame, and extracting the visual feature points of the current image frame;

[0011] S5. Use a fast depth registration algorithm based on three-axis projection clustering to associate the visual feature points of the current image frame with the radar points of the current point cloud frame to obtain the depth value of the visual feature points;

[0012] S6, tracking visual feature points and building a local map;

[0013] S7. Use Scan-Context descriptor and similarity distance to perform loop detection and global BA on the point cloud set and build a global map.

[0014] As a preferred technical solution, the obtaining of the semantic mask of the current image frame is specifically as follows:

[0015] S201, inputting the current image frame into a lightweight semantic segmentation network in the form of a list, downsampling through a general 3×3 convolution downsampling module, and extracting the image tensor of the current image frame;

[0016] S202, passing the picture tensor of the current image frame into a feature extraction backbone network composed of four encoder modules connected in sequence, and obtaining an output tensor of the corresponding encoder module; the encoder module includes a general linear multi-head attention module and an adaptive global vertical and horizontal attention module;

[0017] S203, inputting the output tensor of the second encoder module in the feature extraction backbone network into the multi-scale feature extraction module and the depth-separable asymmetric fusion module to perform multi-scale feature extraction and feature fusion, to obtain a uniform feature tensor and a first fusion feature;

[0018] S204, performing multi-scale fusion of the first fusion feature and the output result of the feature extraction backbone network, and obtaining a semantic mask of the current image frame after processing by an upsampling network connected in sequence by four general upsampling modules; the four encoder modules and the four upsampling modules are correspondingly skipped;

[0019] The step of obtaining a plurality of point cloud sets having different semantic classification labels is as follows:

[0020] S205, applying the camera pinhole model and the transformation matrix, and using the back projection of visual geometry to map the semantic mask of the current image frame to the three-dimensional point cloud of the radar coordinate system, to obtain the point cloud map of the current image frame in the radar coordinate system; the mapping process is:

[0021] The pixel coordinates of the semantic mask of the current image frame are converted to the camera coordinates to obtain the point cloud image of the current image frame in the camera coordinate system;

[0022] Use the transformation matrix to convert the point cloud image of the current image frame in the camera coordinate system into a point cloud image in the radar coordinate system;

[0023] S206, using a lightweight semantic segmentation network to perform point cloud segmentation on the point cloud image of the current image frame in the radar coordinate system, to obtain a laser radar point cloud frame having multiple point cloud sets;

[0024] S207 , assigning corresponding semantic classification labels to each segmented point cloud set according to the semantic mask of the current image frame, to obtain multiple point cloud sets with different semantic classification labels.

[0025] As a preferred technical solution, the adaptive global vertical and horizontal attention module performs feature enhancement on the output of the general linear multi-head attention module, and the process is:

[0026] After channel separation of the output tensor X of the linear multi-head attention module, it is sent to the pooling layer module including the global average pooling layer, the adaptive average pooling layer a and the adaptive average pooling layer b to extract the pooling features; on the other hand, it is sent to the convolution module including the depth convolution 3×3 and the depth convolution 5×5 for channel excitation compression operation, and then multiplied with the output tensor X of the linear multi-head attention module, and finally added with the pooling features output by the pooling layer module to obtain the pixel value domain separation feature;

[0027] After horizontal and vertical adaptive pooling of the output tensor X of the linear multi-head attention module, vector stacking and fusion are performed, and the input is sequentially connected in asymmetric 3×1 convolution and asymmetric 1×5 convolution for spatial association learning. Finally, use Sigmoid The function outputs and broadcasts the channel to generate the attention matrix;

[0028] Multiply the pixel value domain separation feature with the attention matrix to get the output tensor of the encoder module;

[0029] The process of obtaining the uniform feature tensor and the first fusion feature is as follows:

[0030] The output tensor of the second encoder module in the feature extraction backbone network is used as the input tensor X2 of the multi-scale feature extraction module for channel separation. On the one hand, it is input into the receptive field expansion network composed of deep dilated convolution a, deep dilated convolution b and 1×1 convolution to obtain the dilated scale feature; on the other hand, it is input into the scale-preserving network composed of asymmetric convolution 1×3, asymmetric convolution 3×1 and 1×1 convolution to obtain the retained scale feature; finally, the dilated scale feature and the retained scale feature are stacked and shuffled to obtain a uniform feature tensor containing multiple scales;

[0031] The uniform feature tensor is input into the depthwise separable asymmetric fusion module, and is compressed through a branch network consisting of a depthwise asymmetric 1×3 convolution, a depthwise asymmetric 3×1 convolution and a normalized point-by-point convolution, and a branch network consisting of a simple normalized point-by-point convolution. Two compressed feature tensors are obtained and then channel stacking is performed to obtain the first fusion feature.

[0032] As a preferred technical solution, the prediction of the saliency mask of the current image frame is specifically as follows:

[0033] S301, graying the current image frame to obtain a grayscale image;

[0034] S302, using a Gaussian filter to smooth the grayscale image;

[0035] S303, using the Sobel operator to calculate the gradient values ​​of the smoothed grayscale image in the horizontal and vertical directions, and obtain the gradient intensity and direction of each pixel;

[0036] S304, performing non-maximum suppression processing on the smoothed grayscale image, comparing the gradient values ​​of each pixel point and the offset point in the gradient direction, and retaining the pixel points with the maximum gradient value in the gradient direction;

[0037] S305: Classify the pixels in the grayscale image after non-maximum suppression processing, including strong edge pixels. Strong , weak edge Weak and non-marginal Non-edge ; When the gradient value of the pixel is greater than or equal to the high gradient value threshold M H , it is classified as a strong edge Strong ; When the gradient value of the pixel is lower than the low threshold value of the gradient value M L , it is classified as non-marginal Non-edge ; When the gradient value of the pixel is between the high gradient threshold M H and gradient value low threshold M L When it is between Weak ;

[0038] S306, using a connectivity analysis algorithm, in the grayscale image after non-maximum suppression processing, connecting the strong edge and the weak edge with the set Manhattan distance to form a complete edge, and obtaining an edge line map of the current frame image;

[0039] S307, setting the grayscale values ​​of the pixels classified as sky, ground and person in the semantic mask of the current image frame to 0, and setting the grayscale values ​​of the remaining pixels to 255, to obtain a semantic binary image of the current image frame;

[0040] S308, in the edge line map of the current frame image, the gray values ​​of the pixels at positions corresponding to the pixels with gray values ​​of 0 in the semantic binary map are also set to 0, and the remaining pixels remain unchanged, to obtain a geometric focal length map of the current frame image;

[0041] S309: Use a Gaussian filter to perform a diffusion operation on the geometric focal length map to obtain a saliency mask of the current image frame.

[0042] As a preferred technical solution, the extraction of visual feature points of the current image frame is specifically as follows:

[0043] S401, constructing a Gaussian image pyramid of the current image frame and a Gaussian image pyramid of a saliency mask of the current image frame;

[0044] S402, calculating the number of visual feature points that can be extracted per unit area of ​​the Gaussian image pyramid of the current image frame and the number of visual feature points that need to be extracted from each layer of the Gaussian image pyramid of the current image frame, the calculation process is:

[0045] Calculate the total image area of ​​the Gaussian image pyramid of the current image frame S ;

[0046] Calculate the number of visual feature points that can be extracted per unit area of ​​the Gaussian image pyramid of the current image frame;

[0047] Calculate the number of visual feature points that need to be extracted from each layer of the Gaussian image pyramid of the current image frame;

[0048] S403, dividing each layer image of the Gaussian image pyramid of the current image frame into blocks to obtain a plurality of squares;

[0049] S404, traverse each square of each layer of the Gaussian image pyramid of the current image frame, and calculate the extraction threshold of the visual ORB feature extraction algorithm according to the square and the Gaussian image pyramid of the saliency mask of the current image frame, the process is:

[0050] Read the Gaussian image pyramid of the current image frame i Tier j The square corresponds to the Gaussian image pyramid of the saliency mask in the current image frame. i The image block relative to the layer image p ij And calculate the image block p ij The average gray value of

[0051] Calculate the Gaussian image pyramid of the current image frame i Layer Image j The extraction threshold of the visual ORB feature extraction algorithm for squares;

[0052] S405, extracting visual feature points from squares in each layer of the Gaussian image pyramid of the current image frame using the visual ORB feature extraction algorithm according to the extraction threshold of the visual ORB feature extraction algorithm and the number of visual feature points to be extracted from each layer of the Gaussian image pyramid of the current image frame, and saving the extracted visual feature points into a feature point set with the image layer number of the Gaussian image pyramid of the current image frame where the square is located as a label;

[0053] S406, performing quadtree homogenization on the visual feature points of each layer of the image in the feature point set, removing some visual feature points of poor quality, and saving the remaining visual feature points in the key point set in the same way as the visual feature points of the current image frame.

[0054] As a preferred technical solution, the depth value of the visual feature point is obtained as follows:

[0055] S501, aligning the coordinate systems of the two sensors through the external parameters of the monocular camera and the multi-line radar;

[0056] S502, calculating the coordinates of the visual feature points of the current image frame and the radar points of the current point cloud frame in the camera coordinate system;

[0057] S503, projecting the coordinates of the visual feature points and the radar points in the camera coordinate system onto the surface of a unit sphere centered on the camera, and normalizing them into three-dimensional coordinates on the surface of the unit sphere;

[0058] S504: traverse each visual feature point on the surface of the unit sphere to obtain its depth value.

[0059] As a preferred technical solution, traversing each visual feature point on the surface of the unit sphere to obtain its depth value is specifically as follows:

[0060] The visual feature points on the surface of the unit sphere j and each radar point is projected to the xoy plane, yoz plane and zox plane of the three-dimensional coordinate system where the unit sphere is located; the number of projections of the visual feature points and the radar points is in a one-to-many form, that is, the projection of one visual feature point will be linked to the projections of multiple radar points;

[0061] The elbow method is used to determine the number of clusters in each plane for the radar points in each plane. k , and set the radar point clustering distance function;

[0062] Based on visual feature points j And the coordinates of each radar point in each plane, calculate the distance from each radar point in each plane to the visual feature point j The Euclidean distance and using k Mean clustering, get the k Clusters and k Cluster head radar points;

[0063] From the calculated k The cluster head radar points are selected t Visual feature points j The closest cluster head radar point is Euclidean distance, and t The cluster head radar points are connected to form a visual feature point in each plane. j The maximum connected edge graph of ;

[0064] The maximum connected edge graph formed on each plane is projected along the z-axis, x-axis, and y-axis to obtain three radar point sets on the surface of the unit sphere. U 1. U 2 and U 3;

[0065] The three radar point sets U 1. U 2 and U 3. Perform intersection operation to obtain the best radar point set U = U 1∩ U 2∩ U 3;

[0066] Calculate the optimal radar point set U The average depth value of all radar points in is taken as the visual feature point j The depth value of .

[0067] As a preferred technical solution, the visual feature points are tracked and a local map is constructed, specifically:

[0068] S601: Check whether the number of visual feature points with depth values ​​in the current image frame is greater than a threshold ɑ ; If satisfied, mark the current image frame as the initial image frame; if not satisfied, check the next image frame until the initial image frame is found;

[0069] S602, converting the pixel coordinates of the visual feature points with depth values ​​in the initial image frame into 3D points in the camera coordinate system;

[0070] S603, setting the initial camera pose to the unit matrix, that is, the camera coordinate system coincides with the world coordinate system, so the 3D point in the world coordinate system is directly equal to the 3D point in the camera coordinate system;

[0071] S604, storing all visual feature points with depth values ​​in the initial image frame and their corresponding 3D points in the world coordinate system as initial map points;

[0072] S605, projecting the 3D point in the world coordinate system in the previous initial image frame into the initial image frame to obtain its projection coordinates;

[0073] S606: Taking the visual feature point with a depth value in the initial image frame as the center of the circle, a circle with a radius of β The circular search area is calculated, and the Euclidean distance between the projection coordinates of each 3D point in the previous initial image frame and the coordinates of the center of the circle is calculated. The pixel pair with the smallest Euclidean distance is recorded. If the minimum Euclidean distance is less than the threshold θ , then the pixel point is taken as a matching point pair, and the number of matching point pairs is recorded;

[0074] S607: If the number of matching point pairs between the initial image frame and the previous initial image frame is greater than the threshold η , then triangulate each matching point pair to generate 3D map points and then obtain a local map.

[0075] As a preferred technical solution, the loop detection, global BA and construction of a global map are specifically as follows:

[0076] S701, calculate the Scan-Context descriptor matrix of the current point cloud frame, and divide the current point cloud frame into different sector grids according to radial and annular directions, where the number of radial and annular sectors is respectively N r and N s ; Convert the current point cloud frame to a ( N r , N s) of the Scan-Context descriptor matrix; the element values ​​in the Scan-Context descriptor matrix represent the maximum height of all radar points in the fan-shaped grid;

[0077] S702, reduce the dimension of the Scan-Context descriptor matrix of the current point cloud frame to obtain a matrix of size N r *1 one-dimensional Ring Key vector; the value of the Ring Key vector is the average of the maximum height of each row in the Scan-Context descriptor matrix, represented by a ring coding function;

[0078] S703, in the KD tree constructed by the historical key point cloud frame, searching for a similar reference point cloud frame by calculating the distance between the Ring Key values ​​in the Ring Key vector;

[0079] S704, performing a distance comparison between the Scan context descriptor matrix of the searched reference point cloud frame and the Scan context descriptor matrix of the current point cloud frame, finding the Scan context descriptor matrix that is most similar to the Scan context descriptor matrix of the current point cloud frame, and if the distance comparison score is higher than the set distance threshold, it is considered that a loop is found, and the reference point cloud frame corresponding to the most similar Scan context descriptor matrix is ​​used as the final loop point cloud frame;

[0080] S705, determining whether the number of visual feature points of the current image frame is greater than the set number of feature points, and if so, marking the current image frame as a key image frame; obtaining map points in the local map and reprojecting according to the map points, projecting the coordinates of the visual feature points of the current image frame in the world coordinate system through the estimated camera position and the map point position in the local map, and then calculating the difference between the coordinates of the visual feature points of the current image frame in the world coordinate system and the three-dimensional coordinates projected into the local map, and expressing it using an error function;

[0081] S706, using the Gauss-Newton method to calculate the Jacobian matrix and the Hessian matrix of the error function, and iteratively updating the variable optimization objective function;

[0082] S707, monitor the local BA, loop detection and cycle counter of the local map. If the robot detects that it has returned to the previously visited area after the local map is built, the loop detection is triggered. When the cycle counter timing meets the condition, it will enter step S708;

[0083] S708, perform global BA operation on all existing key frames and map point errors; then construct the key frames and map points into a graph structure, and use the g2o graph optimization method to further optimize the global map;

[0084] S709, if it is detected that the robot returns to an area that it has visited before, add new constraints to further optimize the global map and eliminate the accumulated error;

[0085] S710: Compare the key frame list, merge the newly acquired data with the existing map, and update the map information.

[0086] The second object is to provide a multimodal SLAM system, comprising a data acquisition unit, a semantic segmentation unit, a saliency prediction unit, a feature point extraction unit, a depth value calculation unit, a tracking unit and a global construction unit;

[0087] The data acquisition unit is used to obtain a current image frame and a current point cloud frame of the surrounding environment using a monocular camera and a multi-line radar mounted on a mobile platform;

[0088] The semantic segmentation unit is used to segment the current image frame using a lightweight semantic segmentation network to obtain a semantic mask of the current image frame; at the same time, the camera pinhole model and the transformation matrix are used to correct the semantic mask of the current image frame and project it onto the three-dimensional point cloud to obtain a point cloud map of the current image frame in the radar coordinate system, and then the lightweight semantic segmentation network is used to segment the point cloud map of the current image frame in the radar coordinate system to obtain multiple point cloud sets with different semantic classification labels; the lightweight semantic segmentation network is a semantic segmentation model based on lightweight Transformer multi-attention fusion;

[0089] The saliency prediction unit is used to perform object masking on the current image frame and predict the saliency mask of the current image frame in combination with the semantic mask of the current image frame;

[0090] The feature point extraction unit is used to dynamically adjust the extraction threshold of the visual ORB feature extraction algorithm according to the saliency mask of the current image frame to extract the visual feature points of the current image frame;

[0091] The depth value calculation unit is used to use a fast depth registration algorithm based on three-axis projection clustering to associate the visual feature points of the current image frame with the radar points of the current point cloud frame to obtain the depth value of the visual feature points;

[0092] The tracking unit is used to track the visual feature points and construct a local map;

[0093] The global construction unit is used to perform loop detection and global BA on the point cloud set using the Scan-Context descriptor and the similarity distance and to construct a global map.

[0094] The third object is to provide an electronic device, comprising:

[0095] at least one processor; and a memory in communication with the at least one processor; wherein,

[0096] The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor to enable the at least one processor to perform the above-mentioned multimodal SLAM method.

[0097] The fourth purpose is to provide a computer-readable storage medium storing a program, which, when executed by a processor, implements the above-mentioned multimodal SLAM method.

[0098] The fifth purpose is to provide a computer program product, comprising a computer program or instructions, which, when executed by a processor, implement the above-mentioned multimodal SLAM method.

[0099] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0100] 1. This invention introduces advanced semantic segmentation and saliency prediction technology, which can accurately extract key features in the environment. Semantic segmentation can effectively distinguish different types of objects and scenes, and provide detailed semantic information, which helps to fully understand the environment; saliency prediction technology can quickly identify important feature areas in the environment, enhancing the system's perception ability.

[0101] 2. The present invention uses loop detection assisted by point cloud segmentation and visual feature extraction technology combined with saliency information to effectively improve positioning accuracy and map construction accuracy. Point cloud segmentation assisted loop detection technology can identify similar features in the environment, thereby more accurately determining its own position; at the same time, visual feature extraction combined with saliency information can better adapt to environmental changes and improve positioning stability and accuracy.

[0102] 3. The present invention adopts a multimodal SLAM method, making full use of the data of laser radar and visual sensors, and improving the robustness and adaptability of the system in different environments. By comprehensively utilizing the data of multiple sensors, the system can more effectively cope with dynamic scene interference and environmental structure changes, and maintain excellent performance even in complex and changing environments.

[0103] 4. The present invention realizes adaptive feature extraction and environmental perception, significantly reducing the need for human intervention, which not only reduces the complexity and cost of operation and maintenance, but also reduces the impact of human factors on system performance, thereby improving the stability and reliability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0104] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0105] Figure 1 The present invention provides a flowchart of a multimodal SLAM method.

[0106] Figure 2 It is a framework diagram of a lightweight semantic segmentation network in an embodiment of the present invention.

[0107] Figure 3 Schematic diagram of an adaptive global vertical and horizontal attention module in an embodiment of the present invention.

[0108] Figure 4 Schematic diagram of a multi-scale feature extraction module in an embodiment of the present invention.

[0109] Figure 5 Schematic diagram of a depth-separable asymmetric fusion module in an embodiment of the present invention.

[0110] Figure 6 FIG. 4 is a schematic diagram of a saliency mask extracted in an embodiment of the present invention.

[0111] Figure 7 Schematic diagram of the Scan-Context descriptor of point cloud data in an embodiment of the present invention.

[0112] Figure 8 Schematic diagram of the structure of a multimodal SLAM system in an embodiment of the present invention.

[0113] Fig. 9 The figure is a schematic diagram of the structure of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0114] In order to enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0115] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0116] Example 1

[0117] like Figure 1 As shown, this embodiment provides a multimodal SLAM method, comprising the following steps:

[0118] S1, using the monocular camera and multi-line radar mounted on the mobile platform to obtain the current image frame and current point cloud frame of the surrounding environment;

[0119] S2. Use a lightweight semantic segmentation network to segment the current image frame to obtain the semantic mask of the current image frame; at the same time, use the camera pinhole model and transformation matrix to correct the semantic mask of the current image frame and project it onto the three-dimensional point cloud to obtain the point cloud map of the current image frame in the radar coordinate system, and then use a lightweight semantic segmentation network to segment the projected point cloud map of the current image frame in the radar coordinate system to obtain multiple point cloud sets with different semantic classification labels; among them, the lightweight semantic segmentation network is a semantic segmentation model based on lightweight Transformer multi-attention fusion;

[0120] S3, performing object masking on the current image frame and predicting a saliency mask of the current image frame in combination with the semantic mask of the current image frame;

[0121] S4, dynamically adjusting the extraction threshold of the visual ORB feature extraction algorithm according to the saliency mask of the current image frame, thereby achieving the effect of suppressing the generation of visual feature points in the dynamic area and promoting the generation of visual feature points in the static area, and extracting the visual feature points of the current image frame;

[0122] S5. Use a fast depth registration algorithm based on three-axis projection clustering to associate the visual feature points of the current image frame with the radar points of the current point cloud frame to obtain the depth value of the visual feature points;

[0123] S6, tracking visual feature points and building a local map;

[0124] S7. Use Scan-Context descriptor and similarity distance to perform loop detection and global BA on the point cloud set and build a global map.

[0125] In this embodiment, the mobile platform is a robot, which is equipped with a monocular camera to capture image data of the surrounding environment and a multi-line radar to obtain distance data of the surrounding environment; however, the mobile platform is not limited to this, and other mobile platforms that need to achieve autonomous positioning and environmental perception also fall within the scope of protection of this application.

[0126] Furthermore, in step S2, a semantic segmentation model based on lightweight Transformer multi-attention fusion is first used to obtain the semantic mask of the current image frame, such as Figure 2 As shown, specifically:

[0127] S201, input the current image frame in RGB format obtained from the monocular camera into the lightweight semantic segmentation network in the form of a list, downsample it through a general 3×3 convolution downsampling module, compress the image size to reduce the image resolution, extract the spatial features of the image, and obtain the image tensor of the current image frame, which is expressed as:

[0128] T = Conv 3×3 ( Tensor input ),

[0129] in, T is the image tensor output by the downsampling module, Conv 3×3 is a general 3×3 convolution, Tensor input It is the input of the downsampling module;

[0130] S202, passing the image tensor of the current image frame into a feature extraction backbone network composed of four encoder modules connected in sequence, and obtaining the output tensor of the corresponding encoder module; wherein the encoder module includes a general linear multi-head attention module and an adaptive global vertical and horizontal attention module. Specifically, the adaptive global vertical and horizontal attention module is used to enhance the features of the output of the general linear multi-head attention module, such as Figure 3 As shown, the process is:

[0131] After channel separation of the output tensor X of the linear multi-head attention module, the separated tensor is sent to the pooling layer module including the global average pooling layer, the adaptive average pooling layer a and the adaptive average pooling layer b to isolate the low-frequency information between the tensor channels and extract the different low-frequency background features of the tensor to obtain the pooling feature AvgPool On the other hand, the separated tensor is sent to the convolution module containing the depth convolution 3×3 and the depth convolution 5×5 for channel excitation compression operation, and then multiplied with the output tensor X of the linear multi-head attention module to emphasize the obvious regional features in the image, and finally multiplied with the output of the pooling layer module AvgPoolAdd them together to separate the pixel value range of the background features and the obvious areas in the original tensor, and obtain the pixel value range separation features. The operation process is expressed as:

[0132] AvgPool = Concat ( GAvgPool ( x 1), Adaptavg a ( x 2), Adaptavg b ( x 3)),

[0133] Dwconv = Concat ( Dwconv 3×3 ( x 4), Dwconv 5×5 ( x 5)),

[0134] X concat = Dwconv * X + AvgPool ,

[0135] in,{ x 1, x 2, x 3, x 4, x 5}∈ X ,Depend on X The channel separation is obtained; GAvgPool () is the global average pooling layer; Adaptavg a () is the adaptive average pooling layer a, Adaptavg b () is the adaptive average pooling layer b; Concat () is the channel stacking operation, Dwconv 3×3 is the depth convolution 3×3, Dwconv 5×5 is a depthwise convolution of 5×5, Dwconv is the output of the convolution module, X concat Separate features for pixel value domain;

[0136] After horizontal and vertical adaptive pooling of the output tensor X of the linear multi-head attention module, vector stacking and fusion are performed, and the input is sequentially connected in asymmetric 3×1 convolution and asymmetric 1×5 convolution for spatial association learning. Finally, use Sigmoid () function outputs and performs channel broadcasting to generate the attention matrix, which is expressed as:

[0137] X attention = Sigmoid ( Conv 1×5 ( Conv 3×1 ( Concat ( AvgPool h×3 ( X ) T , AvgPool 3×w ( X ))))),

[0138] in, X attention is the attention matrix, Conv 1×5 is an asymmetric 1×5 convolution, Conv 3×1 is an asymmetric 3×1 convolution, AvgPool h×3 is horizontal adaptive pooling, AvgPool 3×w It is vertical adaptive pooling;

[0139] Separate features from pixel value range X concat With the attention matrix X attention Multiply to get the output tensor of the encoder module X output , expressed as:

[0140] X output = X concat * X attention .

[0141] S203, inputting the output tensor of the second encoder module in the feature extraction backbone network into the multi-scale feature extraction module and the depth-separable asymmetric fusion module to perform multi-scale feature extraction and feature fusion, to obtain a uniform feature tensor and a first fusion feature;

[0142] Specifically, Figure 4 As shown, the process of obtaining the uniform feature tensor and the first fusion feature is:

[0143] The output tensor of the second encoder module in the feature extraction backbone network is used as the input tensor X2 of the multi-scale feature extraction module for channel separation. On the one hand, it is input into the receptive field expansion network composed of deep dilated convolution a, deep dilated convolution b and 1×1 convolution to expand the receptive field of the image tensor to obtain more scale features and obtain the expanded scale features. Branch ex On the other hand, the input is a scale-preserving network composed of asymmetric convolution 1×3, asymmetric convolution 3×1 and 1×1 convolution to preserve the feature integrity at the current scale and obtain the scale-preserving feature. Branch st ; Finally, the expanded scale features and the retained scale features are stacked and shuffled to obtain a uniform feature tensor containing multiple scales. Y equ ; The process of the multi-scale feature extraction module is described as:

[0144] Branch ex = ConvBN ( Ddwconv b ( Ddwconv a ( x 1))),

[0145] Branch st = ConvBN ( Conv 3×1 ( Conv 1×3 ( x 2))),

[0146] Y equ = Shuffle ( Concat ( Branch ex , Branch st )),

[0147] in, x 1. x 2 is the tensor after channel separation of input tensor X2, Ddwconv a = Ddwconv b = BN ( dwconv 3×3 ()) is the normalized result of the depth dilated convolution, dwconv 3×3 is a 3×3 depth-expanded convolution, ConvBN = BN ( Conv1×1 ()) is the normalized result of 1×1 convolution, BN () is the normalization operation, Concat ( ) is the channel stacking operation, Shuffle () shuffles the channels;

[0148] like Figure 5 As shown, the uniform feature tensor Y equ Input the deep separable asymmetric fusion module, and perform feature compression through a branch network consisting of deep asymmetric 1×3 convolution, deep asymmetric 3×1 convolution and normalized point-by-point convolution, and a branch network consisting of simple normalized point-by-point convolution, to obtain two compressed feature tensors, which are then channel stacked to obtain the first fusion feature. Y m , expressed as:

[0149] Branch cp1 = BN ( Conv 1×1 ( dwconv 3×1 ( dwconv 1×3 ( Y equ )))),

[0150] Branch cp2 = BN ( Conv 1×1 ( Y equ )),

[0151] Y m = Concat ( Branch cp1 , Branch cp2 ),

[0152] in, Branch cp1 is the first compressed feature tensor, Branch cp2 is the second compressed feature tensor, dwconv 3×1 is a depthwise asymmetric 3×1 convolution, dwconv 1×3 It is a depthwise asymmetric 1×3 convolution.

[0153] S204, multi-scale fusion of the first fusion feature and the output result of the feature extraction backbone network is performed, and the semantic mask of the current image frame is obtained after being processed by the upsampling network connected in sequence by four general upsampling modules; Figure 3 As shown, the four encoder modules are jump-connected to the four up-sampling modules. In this embodiment, the up-sampling modules adopt a bilinear interpolation method.

[0154] At the same time, the same lightweight semantic segmentation network is used for point cloud segmentation to obtain multiple point cloud sets with different semantic classification labels, specifically:

[0155] S205, applying the camera pinhole model and the transformation matrix, and using the back projection of visual geometry to map the semantic mask of the current image frame to the three-dimensional point cloud of the radar coordinate system, to obtain the point cloud map of the current image frame in the radar coordinate system; the mapping process is:

[0156] The pixel coordinates of the semantic mask of the current image frame are converted to the camera coordinates to obtain the point cloud image of the current image frame in the camera coordinate system. The formula is:

[0157] ,

[0158] in,( u , v ) is the coordinate of a pixel in the semantic mask of the current image frame in the pixel coordinate system, ( X C , Y C , Z C )for( u , v ) corresponds to the coordinates in the camera coordinate system, f x , f y , c x , c y All are intrinsic parameters of the camera;

[0159] Using the transformation matrix T CW , T LW Convert the point cloud image of the current image frame in the camera coordinate system to the point cloud image in the radar coordinate system. The conversion formula is:

[0160] ,

[0161] in,( X L , Y L ,Z L )yes( X C , Y C , Z C ) corresponds to the coordinates in the radar coordinate system, T CW is the transformation matrix from the camera coordinate system to the world coordinate system, T LW is the transformation matrix from radar coordinate system to world coordinate system;

[0162] S206, using a lightweight semantic segmentation network to perform point cloud segmentation on the point cloud image of the current image frame in the radar coordinate system, to obtain a laser radar point cloud frame having multiple point cloud sets;

[0163] S207, assigning corresponding semantic classification labels to each segmented point cloud set according to the semantic mask of the current image frame, and obtaining multiple point cloud sets with different semantic classification labels. The LiDAR point cloud frame with semantic information can be used in subsequent loop detection to increase loop constraints.

[0164] Furthermore, in step S3, the saliency mask of the current image frame is predicted, specifically:

[0165] S301, grayscale the current image frame to obtain a grayscale image, and the grayscale processing formula is:

[0166] I g =0.299× I R +0.587× I G +0.114× I B ,

[0167] in, I g Represents the pixel value of the current image frame after grayscale conversion. I R , I G , I B are the pixel values ​​on the red, green and blue channels of the corresponding pixels in the current image frame respectively;

[0168] S302, use a Gaussian filter to smooth the grayscale image to reduce the impact of noise, the formula is:

[0169] ,

[0170] in,( x ,y ) is the coordinate of the pixel in the grayscale image, G () represents Gaussian function, σ represents the standard deviation of the Gaussian filter;

[0171] S303, using the Sobel operator to calculate the gradient values ​​of the smoothed grayscale image in the horizontal and vertical directions, and obtain the gradient intensity and direction of each pixel point; the Sobel operator formula is:

[0172] ,

[0173] in, G x and G y Respectively represent the pixel points in the grayscale image after smoothing ( x , y ) is the gradient value in the horizontal and vertical directions, * is the convolution operator, I gray ( x , y ) is the pixel point in the grayscale image after smoothing ( x , y )’s grayscale value;

[0174] Then the pixel point in the smoothed grayscale image ( x , y ) has a gradient strength and direction of:

[0175] ,

[0176] ,

[0177] in, M ( x , y ) is the pixel point in the grayscale image after smoothing ( x , y ), θ ( x , y ) is the pixel point in the grayscale image after smoothing ( x , y )’s gradient direction;

[0178] S304, performing non-maximum suppression processing on the smoothed grayscale image, comparing the gradient values ​​of each pixel point and the offset point in the gradient direction, retaining the pixel point with the maximum gradient value in the gradient direction, which is conducive to eliminating the blur effect on the edge; the non-maximum suppression formula is:

[0179] ,

[0180] in, M '( x , y ) is the pixel point in the grayscale image after smoothing ( x , y ) The gradient value after non-maximum suppression, M 1 and M 2 are respectively based on the gradient direction θ ( x , y ) calculates the gradient values ​​of adjacent pixels;

[0181] when θ ( x , y )When the value is close to 0°, the pixel point to be compared is ( x , y -1) and ( x , y +1), i.e. M 1=( x , y -1), M 2=( x , y +1);

[0182] when θ ( x , y )When the value is close to 45°, the pixel points to be compared are ( x -1, y +1) and ( x +1, y -1), that is M 1=( x -1, y +1), M 2=( x +1, y -1);

[0183] when θ ( x , y )When the value is close to 90°, the pixel point to be compared is ( x -1, y )and( x +1, y ),Right now M 1=( x -1, y ), M 2=( x +1, y );

[0184] when θ( x , y )When the value is close to 135°, the pixel points to be compared are ( x -1, y -1) and ( x +1, y +1), i.e. M 1=( x -1, y -1), M 2=( x +1, y +1);

[0185] S305: Classify the pixels in the grayscale image after non-maximum suppression processing, including strong edge pixels. Strong , weak edge Weak and non-marginal Non-edge ; When the gradient value of the pixel is greater than or equal to the gradient value high threshold M H , it is classified as a strong edge Strong ; When the gradient value of the pixel is lower than the low threshold value of the gradient value M L , it is classified as non-marginal Non-edge ; When the gradient value of the pixel is between the high gradient threshold M H and gradient value low threshold M L When it is between Weak ; The classification description is:

[0186] ,

[0187] in, Canny ( x , y ) is the pixel point in the grayscale image after non-maximum suppression processing ( x , y ), M H is the high threshold of the gradient value;

[0188] S306, using a connectivity analysis algorithm, in the grayscale image after non-maximum suppression processing, connecting the strong edge and the weak edge with the set Manhattan distance to form a complete edge, and obtaining an edge line map of the current frame image;

[0189] S307, setting the grayscale values ​​of the pixels classified as sky, ground and person in the semantic mask of the current image frame to 0, and setting the grayscale values ​​of the remaining pixels to 255, to obtain a semantic binary image of the current image frame;

[0190] S308, in the edge line map of the current frame image, the grayscale values ​​of the pixels at positions corresponding to the pixels with grayscale values ​​of 0 in the semantic binary map are also set to 0, and the remaining pixels remain unchanged, to obtain a geometric focal length map of the current frame image, which is expressed as:

[0191] ,

[0192] in, Fixation ( x , y ) is the pixel point in the geometric focal length map ( x , y ), I gray_edge ( x , y ) is the pixel point in the edge line map ( x , y ), I gray_semantic ( x , y ) is the pixel point in the semantic binary map ( x , y )’s grayscale value;

[0193] S309, using the Gaussian filter in step S302 to perform a diffusion operation on the geometric focal length map, and through the diffusion process, the saliency is diffused outward from the extracted geometric features to obtain a saliency mask of the current image frame, such as Figure 6 shown.

[0194] Furthermore, in step S4, the extraction threshold of the visual ORB feature extraction algorithm is dynamically adjusted according to the saliency mask of the current image frame to extract the visual feature points of the current image frame, specifically:

[0195] S401, constructing a Gaussian image pyramid ImagePyramid of the current image frame and a Gaussian image pyramid MaskPyramid of a saliency mask of the current image frame;

[0196] S402, calculate the number of visual feature points that can be extracted per unit area of ​​ImagePyramid N average The number of visual feature points that need to be extracted from each layer of the ImagePyramid image is calculated as follows:

[0197] Calculate total image area for ImagePyramid S :

[0198] ,

[0199] in, C is the area of ​​the current image frame,s is the scaling factor of ImagePyramid, m The number of layers of ImagePyramid; the area of ​​the current image frame C The product of the width and height of the current image frame (i.e. the product of the width and height of the image at layer 0 of the ImagePyramid);

[0200] Calculate the number of visual feature points that can be extracted per unit area of ​​ImagePyramid N average :

[0201] ,

[0202] in, N The total number of visual feature points that need to be extracted preset by ImagePyramid;

[0203] Calculate the number of visual feature points that need to be extracted from each layer of ImagePyramid:

[0204] ,

[0205] in, N i For ImagePyramid i The number of visual feature points to be extracted from the layer image;

[0206] S403, divide each layer of the ImagePyramid image into blocks to obtain multiple squares; the side length of each square is W , the calculation formula is:

[0207] ,

[0208] in, W i ImagePyramid i The side length of each square in the layer image; width i and height i They are ImagePyramid i The width and height of the layer image;

[0209] S404, traverse each square of each layer of the ImagePyramid image, and calculate the extraction threshold of the visual ORB feature extraction algorithm according to the square and the MaskPyramid. The process is:

[0210] Read ImagePyramid i Tier jThe squares correspond to the i The image block relative to the layer image p ij ; Calculate image blocks p ij The average gray value :

[0211] ,

[0212] in, N For image blocks p ij The total number of pixels, Lm ( z ) represents an image block p ij Middle z The gray value of pixels;

[0213] Calculate ImagePyramid i Layer Image j The extraction threshold of the visual ORB feature extraction algorithm based on squares T ij :

[0214] ,

[0215] in, n For ImagePyramid i Layer Image j The total number of pixels in a square, LI ( z ) is the ImagePyramid i Layer Image j In the square z The gray value of pixels;

[0216] S405: Extraction threshold according to visual ORB feature extraction algorithm T ij The number of visual feature points to be extracted from each layer of the ImagePyramid image is used, and the visual ORB feature extraction algorithm is used to extract visual feature points from the squares in each layer of the ImagePyramid image. The extracted visual feature points are saved in the feature point set allKeyPoints with the image layer number of the ImagePyramid where the square is located as the label;

[0217] S406, performing quadtree homogenization on the visual feature points of each layer of the image in the feature point set allKeyPoints, removing some visual feature points of poor quality, and saving the remaining visual feature points in the key point set KeyPoints in the same way as the visual feature points of the current image frame.

[0218] Furthermore, in step S5, the visual feature points of the current image frame are associated with the radar points of the current point cloud frame to obtain the depth values ​​of the visual feature points, specifically:

[0219] S501, aligning the coordinate systems of the two sensors through the external parameters of the monocular camera and the multi-line radar;

[0220] S502, calculating the coordinates of the visual feature points of the current image frame and the radar points of the current point cloud frame in the camera coordinate system, the calculation formula is:

[0221] , , z = 1,

[0222] in,( x , y , z ) is the coordinate of the visual feature point or radar point in the camera coordinate system, ( u , v ) is the coordinate of the visual feature point or radar point in the corresponding frame, ( f x , f y ) is the focal length of the camera, ( c x , c y ) is the camera principal point;

[0223] S503, projecting the coordinates of the visual feature points and the radar points in the camera coordinate system onto the surface of a unit sphere centered on the camera, and normalizing them into three-dimensional coordinates on the surface of the unit sphere ( x ' , y ' , z ' ); the normalized formula is:

[0224] ,

[0225] , , ;

[0226] S504: traverse each visual feature point on the surface of the unit sphere to obtain its depth value.

[0227] More specifically, traverse each visual feature point on the surface of the unit sphere to obtain its depth value, specifically:

[0228] First, the visual feature points on the surface of the unit sphere j And each radar point is projected to the xoy plane, yoz plane and zox plane of the three-dimensional coordinate system of the unit sphere respectively; among them, the number of projections of visual feature points and radar points is in a one-to-many form, that is, the projection of one visual feature point will be connected to the projections of multiple radar points; then the visual feature point j The projection relationship with the radar point is:

[0229] Visual feature points j And the radar point is projected onto the xoy plane:

[0230] ,

[0231] in, Lidar x is the constraint for projecting radar points onto the xoy plane (i.e., only radar points that meet the constraint are projected). z is the three-dimensional coordinate of the radar point on the surface of the unit sphere z coordinate, I , II , III , IV , V , VI , VII and VIII Represents eight quadrants in a three-dimensional stereoscopic coordinate system;

[0232] Visual feature points j And the radar point is projected onto the yoz plane:

[0233] ,

[0234] in, Lidar y is the constraint for projecting radar points onto the yoz plane (i.e., only radar points that meet the constraint are projected). x is the three-dimensional coordinate of the radar point on the surface of the unit sphere x coordinate;

[0235] Visual feature points j And the radar point is projected onto the zox plane:

[0236] ,

[0237] in, Lidar z is the constraint for projecting radar points onto the zox plane (i.e., only radar points that meet the constraint are projected). yis the three-dimensional coordinate of the radar point on the surface of the unit sphere y Coordinates; after projection, there is only one visual feature point on the xoy plane, yoz plane, and zox plane j and several radar points.

[0238] Then, the elbow method is used to determine the number of clusters in each plane for the radar points in each plane. k , and set the radar point clustering distance function, expressed as:

[0239] ,

[0240] in, α and β are all weight functions, ( x i , y i ) is the radar point i The coordinates on a plane, ( x j , y j ) is the visual feature point j At the point with the radar i The corresponding coordinates on the plane are depth i is the radar point in the current point cloud frame i The depth value of

[0241] Next, according to the visual feature points j And the coordinates of each radar point in each plane, calculate the distance from each radar point in each plane to the visual feature point j The Euclidean distance and using k Mean clustering, get the k Clusters and k Cluster head radar points;

[0242] From the calculated k The cluster head radar points are selected t Visual feature points j The Euclidean distance to the nearest cluster head radar point is t The cluster head radar points are connected to form a visual feature point in each plane. j The maximum connected edge graph of ;

[0243] The maximum connected edge graph formed on each plane is projected along the z-axis, x-axis, and y-axis to obtain three radar point sets on the surface of the unit sphere. U 1. U 2 and U 3;

[0244] The three radar point sets U 1.U 2 and U 3. Perform intersection operation to obtain the best radar point set U = U 1∩ U 2∩ U 3;

[0245] Finally calculate the best radar point set U The average depth value of all radar points in is taken as the visual feature point j The depth value of .

[0246] Furthermore, in step S6, the visual feature points are tracked and a local map is constructed, specifically:

[0247] S601: Check whether the number of visual feature points with depth values ​​in the current image frame is greater than a threshold ɑ ; If satisfied, mark the current image frame as the initial image frame; if not satisfied, check the next image frame until the initial image frame is found;

[0248] S602, converting the pixel coordinates of the visual feature points with depth values ​​in the initial image frame into 3D points in the camera coordinate system, and the conversion formula is:

[0249] ,

[0250] in,( f x , f y ) is the focal length of the camera, ( c x , c y ) is the camera principal point, ( u i , v i ) is the visual feature point in the initial image frame i The pixel coordinates of d i is the visual feature point in the initial image frame i The depth value of

[0251] S603, set the camera initial position T init Set to the identity matrix I , that is, the camera coordinate system coincides with the world coordinate system, expressed as:

[0252] ,

[0253] Therefore, the 3D point in the world coordinate system is directly equal to the 3D point in the camera coordinate system, so the visual feature point in the initial image frame iA 3D point in the world coordinate system is represented by:

[0254] ;

[0255] S604, storing all visual feature points with depth values ​​in the initial image frame and their corresponding 3D points in the world coordinate system as initial map points;

[0256] S605, projecting the 3D point in the world coordinate system in the previous initial image frame into the initial image frame to obtain its projection coordinates, and the projection formula is:

[0257] ,

[0258] in, The previous initial image frame t In the world coordinate system i The projection coordinates of the 3D points in the initial image frame, The previous initial image frame t In the world coordinate system i The three-dimensional coordinates of the 3D points, K is the camera intrinsic parameter matrix;

[0259] S606: Taking the visual feature point with a depth value in the initial image frame as the center of the circle, a circle with a radius of β The circular search area is calculated, and the Euclidean distance between the projection coordinates of each 3D point in the previous initial image frame and the coordinates of the center of the circle is calculated. The pixel pair with the smallest Euclidean distance is recorded. If the minimum Euclidean distance is less than the threshold θ , then the pixel point is taken as a matching point pair, and the number of matching point pairs is recorded;

[0260] S607: If the number of matching point pairs between the initial image frame and the previous initial image frame is greater than the threshold η , then triangulate each matching point pair to generate 3D map points and then obtain a local map.

[0261] Furthermore, in step S7, the Scan-Context description submatrix is ​​first used to express a frame of point cloud, and then the Scan-Context description submatrix distance is calculated, and the closest point cloud frame is found in the historical key point cloud frame to determine whether a loop phenomenon occurs, specifically:

[0262] S701, such as Figure 7 As shown, the Scan-Context (SC) descriptor matrix of the current point cloud frame is calculated, and the current point cloud frame is divided into different sector grids according to radial and annular directions, where the number of radial and annular directions is respectively N r and Ns ; Convert the current point cloud frame to a ( N r , N s ) of the Scan-Context descriptor matrix; the element value in the Scan-Context descriptor matrix represents the maximum height of all radar points in the sector grid;

[0263] S702: In order to speed up the search, the Scan-Context descriptor matrix of the current point cloud frame is reduced in dimension to obtain a matrix of size N s *1 one-dimensional Ring Key vector; the value of the Ring Key vector is the average of the maximum height of each row in the Scan-Context descriptor matrix, using the ring encoding function ψ express:

[0264] ,

[0265] in, ψ j For the j The Ring Key value of the row, r ij For each in the Scan-Context description matrix j No. i The maximum height of the radar points in the column, N s The number of columns in the Scan context description submatrix.

[0266] S703, in the KD tree constructed by the historical key point cloud frame, searching for a similar reference point cloud frame by calculating the distance between the Ring Key values ​​in the Ring Key vector;

[0267] S704, compare the distance between the Scan context descriptor matrix of the reference point cloud frame and the Scan context descriptor matrix of the current point cloud frame, find the Scan context descriptor matrix that is most similar to the Scan context descriptor matrix of the current point cloud frame, and if the distance comparison score is higher than the set distance threshold, it is considered that the loop is found, and the reference point cloud frame corresponding to the most similar Scan context descriptor matrix is ​​used as the final loop point cloud frame; the distance comparison score formula is:

[0268] ,

[0269] in, I q is the laser point cloud data of the current point cloud frame, Ic is the laser point cloud data of the historical point cloud frame, is the first in the Scan-Context description sub-matrix of the current point cloud frame j The element value of the column, is the first in the Scan-Context descriptor matrix of the reference point cloud frame j The element value of the column;

[0270] S705, determine whether the number of visual feature points of the current image frame is greater than the set number of feature points, and if so, mark the current image frame as a key image frame; obtain the map points in the local map and reproject according to the map points, project the coordinates of the visual feature points of the current image frame in the world coordinate system through the estimated camera position and the map point position in the local map, and then calculate the difference between the coordinates of the visual feature points of the current image frame in the world coordinate system and the three-dimensional coordinates projected into the local map, and use the error function to represent it:

[0271] e ij = π ( K [ R i X j + t i ])- u ij ,

[0272] in, e ij It is i The first keyframe j Reprojection error of visual feature points; π () is the projection function, K is the camera intrinsic parameter matrix, R i and t i They are i The rotation matrix and translation vector of the keyframe, X j It is j The visual feature points are projected to the three-dimensional coordinates of the local map. u ij It is i The first keyframe j The coordinates of the visual feature points in the world coordinate system;

[0273] S706, using the Gauss-Newton method to calculate the Jacobian matrix and Hessian matrix of the error function, and iteratively updating the variable optimization objective function, expressed as:

[0274] ;

[0275] S707, monitor the local BA, loop detection and cycle counter of the local map. If the robot detects that it has returned to the previously visited area after the local map is built, the loop detection is triggered. When the cycle counter timing meets the condition, it will enter step S708;

[0276] S708, perform global BA operation on all existing key frames and map point errors; then construct the key frames and map points into a graph structure, and use the g2o graph optimization method to further optimize the global map;

[0277] S709: If it is detected that the robot returns to an area that it has visited before, new constraints are added to further optimize the global map and eliminate the accumulated error.

[0278] S710: Compare the key frame list, merge the newly acquired data with the existing map, and update the map information.

[0279] It should be noted that, for the sake of convenience, the aforementioned method embodiments are all expressed as a series of action combinations, but those skilled in the art should know that the present invention is not limited to the described order of actions, because according to the present invention, certain steps can be performed in other orders or simultaneously.

[0280] Example 2

[0281] Based on the same idea as the multimodal SLAM method in the above embodiment, the present invention also provides a multimodal SLAM system, which can be used to perform the above multimodal SLAM method. For ease of explanation, in the structural schematic diagram of the multimodal SLAM system embodiment, only the parts related to the embodiment of the present invention are shown, and those skilled in the art will understand that the illustrated structure does not constitute a limitation on the device, and may include more or less parts than the illustrated parts, or combine certain parts, or arrange different parts.

[0282] like Figure 8 As shown, another embodiment of the present invention provides a multimodal SLAM system 800, including a data acquisition unit 801, a semantic segmentation unit 802, a saliency prediction unit 803, a feature point extraction unit 804, a depth value calculation unit 805, a tracking unit 806 and a global construction unit 807;

[0283] The data acquisition unit 801 is used to obtain the current image frame and the current point cloud frame of the surrounding environment using the monocular camera and the multi-line radar mounted on the mobile platform;

[0284] The semantic segmentation unit 802 is used to segment the current image frame using a lightweight semantic segmentation network to obtain a semantic mask of the current image frame; at the same time, the camera pinhole model and the transformation matrix are used to correct the semantic mask of the current image frame and project it onto the three-dimensional point cloud to obtain a point cloud map of the current image frame in the radar coordinate system, and then the lightweight semantic segmentation network is used to segment the point cloud map of the current image frame in the radar coordinate system to obtain multiple point cloud sets with different semantic classification labels; the lightweight semantic segmentation network is a semantic segmentation model based on lightweight Transformer multi-attention fusion;

[0285] The saliency prediction unit 803 is used to perform object masking on the current image frame and predict the saliency mask of the current image frame in combination with the semantic mask of the current image frame;

[0286] The feature point extraction unit 804 is used to dynamically adjust the extraction threshold of the visual ORB feature extraction algorithm according to the saliency mask of the current image frame to extract the visual feature points of the current image frame;

[0287] The depth value calculation unit 805 is used to use a fast depth registration algorithm based on three-axis projection clustering to associate the visual feature points of the current image frame with the radar points of the current point cloud frame to obtain the depth value of the visual feature points;

[0288] The tracking unit 806 is used to track the visual feature points and construct a local map;

[0289] The global construction unit 807 is used to perform loop detection and global BA on the point cloud set using the Scan-Context descriptor and similarity distance and to construct a global map.

[0290] It should be noted that the multimodal SLAM system of the present invention corresponds one-to-one to the multimodal SLAM method of the present invention. The technical features and beneficial effects described in the embodiments of the multimodal SLAM method are applicable to the embodiments of the multimodal SLAM system. For specific contents, please refer to the description in the embodiments of the method of the present invention. They will not be repeated here. This is hereby declared.

[0291] In addition, in the implementation of the multimodal SLAM system in the above-mentioned embodiment, the logical division of each program module is only an example. In actual application, the above-mentioned functions can be assigned to different program modules as needed, for example, for the configuration requirements of the corresponding hardware or the convenience of software implementation. That is, the internal structure of the multimodal SLAM system is divided into different program modules to complete all or part of the functions described above.

[0292] Example 3

[0293] See also Fig. 9In one embodiment, an electronic device 900 for implementing a multimodal SLAM method is provided. The electronic device may include a first processor 901, a first memory 902 and a bus, and may also include a computer program stored in the first memory 902 and executable on the first processor 901, such as a multimodal SLAM program 903.

[0294] Among them, the first memory 902 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, mobile hard disk, multimedia card, card-type memory (for example: SD or DX memory, etc.), magnetic memory, disk, optical disk, etc. The first memory 902 can be an internal storage unit of the electronic device 900 in some embodiments, such as a mobile hard disk of the electronic device 900. The first memory 902 can also be an external storage device of the electronic device 900 in other embodiments, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 900. Further, the first memory 902 can also include both an internal storage unit of the electronic device 900 and an external storage device. The first memory 902 can not only be used to store application software and various types of data installed in the electronic device 900, such as the code of the multimodal SLAM program 903, etc., but also can be used to temporarily store data that has been output or is to be output.

[0295] In some embodiments, the first processor 901 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The first processor 901 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes various functions and processes data of the electronic device 900 by running or executing programs or modules stored in the first memory 902, and calling data stored in the first memory 902.

[0296] Fig. 9 Only an electronic device with components is shown, and those skilled in the art will understand that Fig. 9 The structure shown does not constitute a limitation on the electronic device 900 , and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0297] The multimodal SLAM program 903 stored in the first memory 902 in the electronic device 900 is a combination of multiple instructions. When running in the first processor 901, it can achieve:

[0298] S1, using the monocular camera and multi-line radar mounted on the mobile platform to obtain the current image frame and current point cloud frame of the surrounding environment;

[0299] S2. Use a lightweight semantic segmentation network to segment the current image frame to obtain the semantic mask of the current image frame; at the same time, use the camera pinhole model and transformation matrix to correct the semantic mask of the current image frame and project it onto the three-dimensional point cloud to obtain the point cloud map of the current image frame in the radar coordinate system, and then use a lightweight semantic segmentation network to segment the point cloud map of the current image frame in the radar coordinate system to obtain multiple point cloud sets with different semantic classification labels; among them, the lightweight semantic segmentation network is a semantic segmentation model based on lightweight Transformer multi-attention fusion;

[0300] S3, performing object masking on the current image frame and predicting a saliency mask of the current image frame in combination with the semantic mask of the current image frame;

[0301] S4, dynamically adjusting the extraction threshold of the visual ORB feature extraction algorithm according to the saliency mask of the current image frame, and extracting the visual feature points of the current image frame;

[0302] S5. Use a fast depth registration algorithm based on three-axis projection clustering to associate the visual feature points of the current image frame with the radar points of the current point cloud frame to obtain the depth value of the visual feature points;

[0303] S6, tracking visual feature points and building a local map;

[0304] S7. Use Scan-Context descriptor and similarity distance to perform loop detection and global BA on the point cloud set and build a global map.

[0305] Furthermore, if the module / unit integrated in the electronic device 900 is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium. The computer-readable medium may include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a mobile hard disk, a magnetic disk, an optical disk, a computer memory, and a read-only memory (ROM).

[0306] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0307] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0308] The above embodiments are preferred implementation modes of the present invention, but the implementation modes of the present invention are not limited to the above embodiments. Any other changes, modifications, substitutions, combinations, and simplifications that do not deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the protection scope of the present invention.

Claims

1. A multimodal SLAM method, characterized in that: The steps include: S1, using the monocular camera and multi-line radar mounted on the mobile platform to obtain the current image frame and current point cloud frame of the surrounding environment; S2. Use a lightweight semantic segmentation network to segment the current image frame to obtain a semantic mask of the current image frame; at the same time, use the camera pinhole model and the transformation matrix to correct the semantic mask of the current image frame and project it onto the three-dimensional point cloud to obtain a point cloud map of the current image frame in the radar coordinate system, and then use a lightweight semantic segmentation network to segment the point cloud map of the current image frame in the radar coordinate system to obtain multiple point cloud sets with different semantic classification labels; the lightweight semantic segmentation network is a semantic segmentation model based on lightweight Transformer multi-attention fusion; S3, performing object masking on the current image frame and predicting a saliency mask of the current image frame in combination with the semantic mask of the current image frame; S4, dynamically adjusting the extraction threshold of the visual ORB feature extraction algorithm according to the saliency mask of the current image frame, and extracting the visual feature points of the current image frame; S5. Use a fast depth registration algorithm based on three-axis projection clustering to associate the visual feature points of the current image frame with the radar points of the current point cloud frame to obtain the depth value of the visual feature points; S6, tracking visual feature points and building a local map; S7. Use Scan-Context descriptor and similarity distance to perform loop detection and global BA on the point cloud set and build a global map.

2. The multimodal SLAM method according to claim 1, wherein: The obtaining of the semantic mask of the current image frame is specifically as follows: S201, inputting the current image frame into a lightweight semantic segmentation network in the form of a list, down-sampling through a general 3×3 convolution down-sampling module, and extracting the image tensor of the current image frame; S202, passing the picture tensor of the current image frame into a feature extraction backbone network composed of four encoder modules connected in sequence, and obtaining an output tensor of the corresponding encoder module; the encoder module includes a general linear multi-head attention module and an adaptive global vertical and horizontal attention module; S203, inputting the output tensor of the second encoder module in the feature extraction backbone network into the multi-scale feature extraction module and the depth-separable asymmetric fusion module to perform multi-scale feature extraction and feature fusion, to obtain a uniform feature tensor and a first fusion feature; S204, performing multi-scale fusion of the first fusion feature and the output result of the feature extraction backbone network, and obtaining a semantic mask of the current image frame after processing by an upsampling network connected in sequence by four general upsampling modules; the four encoder modules and the four upsampling modules are correspondingly skipped; The step of obtaining a plurality of point cloud sets having different semantic classification labels is as follows: S205, applying the camera pinhole model and the transformation matrix, and using the back projection of visual geometry to map the semantic mask of the current image frame to the three-dimensional point cloud of the radar coordinate system, to obtain the point cloud map of the current image frame in the radar coordinate system; the mapping process is: The pixel coordinates of the semantic mask of the current image frame are converted to the camera coordinates to obtain the point cloud image of the current image frame in the camera coordinate system; Use the transformation matrix to convert the point cloud image of the current image frame in the camera coordinate system into a point cloud image in the radar coordinate system; S206, using a lightweight semantic segmentation network to perform point cloud segmentation on the point cloud image of the current image frame in the radar coordinate system, to obtain a laser radar point cloud frame having multiple point cloud sets; S207 , assigning corresponding semantic classification labels to each segmented point cloud set according to the semantic mask of the current image frame, to obtain multiple point cloud sets with different semantic classification labels.

3. The multimodal SLAM method according to claim 2, wherein: The adaptive global vertical and horizontal attention module performs feature enhancement on the output of the general linear multi-head attention module. The process is as follows: After channel separation of the output tensor X of the linear multi-head attention module, it is sent to the pooling layer module including the global average pooling layer, the adaptive average pooling layer a and the adaptive average pooling layer b to extract the pooling features; on the other hand, it is sent to the convolution module including the depth convolution 3×3 and the depth convolution 5×5 for channel excitation compression operation, and then multiplied with the output tensor X of the linear multi-head attention module, and finally added with the pooling features output by the pooling layer module to obtain the pixel value domain separation feature; After horizontal and vertical adaptive pooling of the output tensor X of the linear multi-head attention module, vector stacking and fusion are performed, and the input is sequentially connected in asymmetric 3×1 convolution and asymmetric 1×5 convolution for spatial association learning. Finally, use Sigmoid The function outputs and broadcasts the channel to generate the attention matrix; Multiply the pixel value domain separation feature with the attention matrix to get the output tensor of the encoder module; The process of obtaining the uniform feature tensor and the first fusion feature is as follows: The output tensor of the second encoder module in the feature extraction backbone network is used as the input tensor X2 of the multi-scale feature extraction module for channel separation. On the one hand, it is input into the receptive field expansion network composed of deep dilated convolution a, deep dilated convolution b and 1×1 convolution to obtain the dilated scale feature; on the other hand, it is input into the scale-preserving network composed of asymmetric convolution 1×3, asymmetric convolution 3×1 and 1×1 convolution to obtain the retained scale feature; finally, the dilated scale feature and the retained scale feature are stacked and shuffled to obtain a uniform feature tensor containing multiple scales; The uniform feature tensor is input into the depthwise separable asymmetric fusion module, and is compressed through a branch network consisting of a depthwise asymmetric 1×3 convolution, a depthwise asymmetric 3×1 convolution and a normalized point-by-point convolution, and a branch network consisting of a simple normalized point-by-point convolution. Two compressed feature tensors are obtained and then channel stacking is performed to obtain the first fusion feature.

4. The multimodal SLAM method according to claim 1, wherein: The predicting of the saliency mask of the current image frame is specifically: S301, graying the current image frame to obtain a grayscale image; S302, using a Gaussian filter to smooth the grayscale image; S303, using the Sobel operator to calculate the gradient values ​​of the smoothed grayscale image in the horizontal and vertical directions, and obtain the gradient intensity and direction of each pixel; S304, performing non-maximum suppression processing on the smoothed grayscale image, comparing the gradient values ​​of each pixel point and the offset point in the gradient direction, and retaining the pixel points with the maximum gradient value in the gradient direction; S305: Classify the pixels in the grayscale image after non-maximum suppression processing, including strong edge pixels. Strong , weak edge Weak and non-marginal Non-edge ; When the gradient value of the pixel is greater than or equal to the gradient value high threshold M H , it is classified as a strong edge Strong ; When the gradient value of the pixel is lower than the low threshold value of the gradient value M L , it is classified as non-marginal Non-edge ; When the gradient value of the pixel is between the high gradient threshold M H and gradient value low threshold M L When it is between Weak ; S306, using a connectivity analysis algorithm, in the grayscale image after non-maximum suppression processing, connecting the strong edge and the weak edge with the set Manhattan distance to form a complete edge, and obtaining an edge line map of the current frame image; S307, setting the grayscale values ​​of the pixels classified as sky, ground and person in the semantic mask of the current image frame to 0, and setting the grayscale values ​​of the remaining pixels to 255, to obtain a semantic binary image of the current image frame; S308, in the edge line map of the current frame image, the gray values ​​of the pixels at positions corresponding to the pixels with gray values ​​of 0 in the semantic binary map are also set to 0, and the remaining pixels remain unchanged, to obtain a geometric focal length map of the current frame image; S309: Use a Gaussian filter to perform a diffusion operation on the geometric focal length map to obtain a saliency mask of the current image frame.

5. The multimodal SLAM method according to claim 1, wherein: The extracting of visual feature points of the current image frame is specifically as follows: S401, constructing a Gaussian image pyramid of the current image frame and a Gaussian image pyramid of a saliency mask of the current image frame; S402, calculating the number of visual feature points that can be extracted per unit area of ​​the Gaussian image pyramid of the current image frame and the number of visual feature points that need to be extracted from each layer of the Gaussian image pyramid of the current image frame, the calculation process is: Calculate the total image area of ​​the Gaussian image pyramid of the current image frame S ; Calculate the number of visual feature points that can be extracted per unit area of ​​the Gaussian image pyramid of the current image frame; Calculate the number of visual feature points that need to be extracted from each layer of the Gaussian image pyramid of the current image frame; S403, dividing each layer image of the Gaussian image pyramid of the current image frame into blocks to obtain a plurality of squares; S404, traverse each square of each layer of the Gaussian image pyramid of the current image frame, and calculate the extraction threshold of the visual ORB feature extraction algorithm according to the square and the Gaussian image pyramid of the saliency mask of the current image frame, the process is: Read the Gaussian image pyramid of the current image frame i Tier j The square corresponds to the Gaussian image pyramid of the saliency mask in the current image frame. i The image block relative to the layer image p ij And calculate the image block p ij The average gray value of Calculate the Gaussian image pyramid of the current image frame i Layer Image j The extraction threshold of the visual ORB feature extraction algorithm for squares; S405, extracting visual feature points from squares in each layer of the Gaussian image pyramid of the current image frame using the visual ORB feature extraction algorithm according to the extraction threshold of the visual ORB feature extraction algorithm and the number of visual feature points to be extracted from each layer of the Gaussian image pyramid of the current image frame, and saving the extracted visual feature points into a feature point set with the image layer number of the Gaussian image pyramid of the current image frame where the square is located as a label; S406, performing quadtree homogenization on the visual feature points of each layer of the image in the feature point set, removing some visual feature points of poor quality, and saving the remaining visual feature points in the key point set in the same way as the visual feature points of the current image frame.

6. The multimodal SLAM method according to claim 1, wherein: The obtaining of the depth value of the visual feature point is specifically as follows: S501, aligning the coordinate systems of the two sensors through the external parameters of the monocular camera and the multi-line radar; S502, calculating the coordinates of the visual feature points of the current image frame and the radar points of the current point cloud frame in the camera coordinate system; S503, projecting the coordinates of the visual feature points and the radar points in the camera coordinate system onto the surface of a unit sphere centered on the camera, and normalizing them into three-dimensional coordinates on the surface of the unit sphere; S504: traverse each visual feature point on the surface of the unit sphere to obtain its depth value.

7. The multimodal SLAM method according to claim 6, wherein: The method of traversing each visual feature point on the surface of the unit sphere to obtain its depth value is specifically as follows: The visual feature points on the surface of the unit sphere j and each radar point is projected to the xoy plane, yoz plane and zox plane of the three-dimensional coordinate system where the unit sphere is located; the number of projections of the visual feature points and the radar points is in a one-to-many form, that is, the projection of one visual feature point will be linked to the projections of multiple radar points; The elbow method is used to determine the number of clusters in each plane for the radar points in each plane. k , and set the radar point clustering distance function; Based on visual feature points j And the coordinates of each radar point in each plane, calculate the distance from each radar point in each plane to the visual feature point j The Euclidean distance and using k Mean clustering, get the k Clusters and k Cluster head radar points; From the calculated k The cluster head radar points are selected t Visual feature points j The Euclidean distance to the nearest cluster head radar point is t The cluster head radar points are connected to form a visual feature point in each plane. j The maximum connected edge graph of ; The maximum connected edge graph formed on each plane is projected along the z-axis, x-axis, and y-axis to obtain three radar point sets on the surface of the unit sphere. U 1. U 2 and U 3; The three radar point sets U 1. U 2 and U 3. Perform intersection operation to obtain the best radar point set U = U 1∩ U 2∩ U 3; Calculate the optimal radar point set U The average depth value of all radar points in is used as the visual feature point j The depth value of .

8. The multimodal SLAM method according to claim 1, wherein: The tracking of visual feature points and construction of a local map are specifically as follows: S601: Check whether the number of visual feature points with depth values ​​in the current image frame is greater than a threshold ɑ ; If satisfied, the current image frame is marked as the initial image frame; If not satisfied, check the next image frame until the initial image frame is found; S602, converting the pixel coordinates of the visual feature points with depth values ​​in the initial image frame into 3D points in the camera coordinate system; S603, setting the initial camera pose to the unit matrix, that is, the camera coordinate system coincides with the world coordinate system, so the 3D point in the world coordinate system is directly equal to the 3D point in the camera coordinate system; S604, storing all visual feature points with depth values ​​in the initial image frame and their corresponding 3D points in the world coordinate system as initial map points; S605, projecting the 3D point in the world coordinate system in the previous initial image frame into the initial image frame to obtain its projection coordinates; S606: Taking the visual feature point with a depth value in the initial image frame as the center of the circle, a circle with a radius of β The circular search area is calculated, and the Euclidean distance between the projection coordinates of each 3D point in the previous initial image frame and the coordinates of the center of the circle is calculated. The pixel pair with the smallest Euclidean distance is recorded. If the minimum Euclidean distance is less than the threshold θ , then the pixel point is taken as a matching point pair, and the number of matching point pairs is recorded; S607: If the number of matching point pairs between the initial image frame and the previous initial image frame is greater than the threshold η , then triangulate each matching point pair to generate 3D map points and then obtain a local map.

9. The multimodal SLAM method according to claim 1, wherein: The loop detection, global BA and construction of the global map are specifically as follows: S701, calculate the Scan-Context descriptor matrix of the current point cloud frame, and divide the current point cloud frame into different sector grids according to radial and annular directions, where the number of radial and annular sectors is respectively N r and N s ; Convert the current point cloud frame to a ( N r , N s ) of the Scan-Context descriptor matrix; the element values ​​in the Scan-Context descriptor matrix represent the maximum height of all radar points in the fan-shaped grid; S702, reduce the dimension of the Scan-Context descriptor matrix of the current point cloud frame to obtain a matrix of size N r *1 one-dimensional RingKey vector; the value of the Ring Key vector is the average of the maximum height of each row in the Scan-Context descriptor matrix, represented by a ring coding function; S703, in the KD tree constructed by the historical key point cloud frame, searching for a similar reference point cloud frame by calculating the distance between the Ring Key values ​​in the Ring Key vector; S704, performing a distance comparison between the Scan context descriptor matrix of the searched reference point cloud frame and the Scan context descriptor matrix of the current point cloud frame, finding the Scan context descriptor matrix that is most similar to the Scan context descriptor matrix of the current point cloud frame, and if the distance comparison score is higher than the set distance threshold, it is considered that a loop is found, and the reference point cloud frame corresponding to the most similar Scan context descriptor matrix is ​​used as the final loop point cloud frame; S705, determining whether the number of visual feature points of the current image frame is greater than the set number of feature points, and if so, marking the current image frame as a key image frame; obtaining map points in the local map and reprojecting according to the map points, projecting the coordinates of the visual feature points of the current image frame in the world coordinate system through the estimated camera position and the map point position in the local map, and then calculating the difference between the coordinates of the visual feature points of the current image frame in the world coordinate system and the three-dimensional coordinates projected into the local map, and expressing it using an error function; S706, using the Gauss-Newton method to calculate the Jacobian matrix and the Hessian matrix of the error function, and iteratively updating the variable optimization objective function; S707, monitor the local BA, loop detection and cycle counter of the local map. If the robot detects that it has returned to the previously visited area after the local map is built, the loop detection is triggered. When the cycle counter timing meets the condition, it will enter step S708; S708, perform global BA operation on all existing key frames and map point errors; then construct the key frames and map points into a graph structure, and use the g2o graph optimization method to further optimize the global map; S709, if it is detected that the robot returns to an area that it has visited before, add new constraints to further optimize the global map and eliminate the accumulated error; S710: Compare the key frame list, merge the newly acquired data with the existing map, and update the map information.

10. A multimodal SLAM system, characterized in that: The system includes a data acquisition unit, a semantic segmentation unit, a saliency prediction unit, a feature point extraction unit, a depth value calculation unit, a tracking unit and a global construction unit; The data acquisition unit is used to obtain a current image frame and a current point cloud frame of the surrounding environment using a monocular camera and a multi-line radar mounted on a mobile platform; The semantic segmentation unit is used to segment the current image frame using a lightweight semantic segmentation network to obtain a semantic mask of the current image frame; at the same time, the camera pinhole model and the transformation matrix are used to correct the semantic mask of the current image frame and project it onto the three-dimensional point cloud to obtain a point cloud map of the current image frame in the radar coordinate system, and then the lightweight semantic segmentation network is used to segment the point cloud map of the current image frame in the radar coordinate system to obtain multiple point cloud sets with different semantic classification labels; the lightweight semantic segmentation network is a semantic segmentation model based on lightweight Transformer multi-attention fusion; The saliency prediction unit is used to perform object masking on the current image frame and predict the saliency mask of the current image frame in combination with the semantic mask of the current image frame; The feature point extraction unit is used to dynamically adjust the extraction threshold of the visual ORB feature extraction algorithm according to the saliency mask of the current image frame to extract the visual feature points of the current image frame; The depth value calculation unit is used to use a fast depth registration algorithm based on three-axis projection clustering to associate the visual feature points of the current image frame with the radar points of the current point cloud frame to obtain the depth value of the visual feature points; The tracking unit is used to track the visual feature points and construct a local map; The global construction unit is used to perform loop detection and global BA on the point cloud set using the Scan-Context descriptor and the similarity distance and to construct a global map.

11. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory in communication with the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor, and the computer program instructions are executed by the at least one processor so that the at least one processor can perform the multimodal SLAM method as described in any one of claims 1-9.

12. A computer-readable storage medium storing a program, characterized in that: When the program is executed by a processor, the multimodal SLAM method described in any one of claims 1 to 9 is implemented.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the multimodal SLAM method according to any one of claims 1 to 9 is implemented.

Images