Multi-modal location identification method based on multi-scale self-attention mechanism

Abstract

The invention discloses a multi-modal site identification method based on a multi-scale self-attention mechanism, and belongs to the field of site identification. The implementation method comprises the steps of performing multi-modal feature fusion on multiple feature scales through a self-attention mechanism by simultaneously taking a look-around multi-view RGB image, a laser radar environment point cloud and a millimeter wave radar environment point cloud as input, generating a multi-modal global LCGD descriptor, and screening k candidate samples by using the LCGD descriptor according to an Euclidean distance; a point cloud scattering sectional area probability density function is extracted from millimeter wave radar environment point clouds, the total reordering distance is calculated, k candidate samples are reordered, and the coordinates of the candidate sample with the minimum reordering distance serve as the coordinates of the vehicle at the current moment. The method can be used for solving the problem of poor robustness for changes of environmental factors such as illumination and weather and changes of the visual angle of the vehicle in the existing site identification technology, and the identification precision of the site identification method and the generalization under different environmental conditions are enhanced.

Description

Technical Field

[0001] The present invention belongs to the technical field of place recognition, and in particular relates to a multimodal place recognition method based on a multi-scale self-attention mechanism. Background Art

[0002] Location recognition is a key technology in the field of intelligent vehicle navigation and positioning. It matches and locates the environment through the detection of the vehicle's own sensor. During the movement of intelligent vehicles, location recognition technology extracts a global location descriptor expressed in vector or matrix form for each location. When the vehicle needs to obtain its own location information, it compares the current location descriptor with the descriptor in the database to determine the location of the current vehicle in the global map. It can also determine whether the vehicle has reached the vicinity of the historical location in the database. Location recognition technology can solve the problem of rapid global positioning of intelligent vehicles in long-term repeated driving environments. It can also be applied to simultaneous localization and mapping (SLAM) technology to solve the loop detection problem and eliminate the cumulative positioning error of SLAM.

[0003] Place recognition technology relies on sensor observations, but each sensor has its inherent shortcomings and degradation scenarios. For example, although cameras are rich in semantic information and inexpensive, they are easily affected by lighting. LiDAR is not affected by lighting and can capture environmental three-dimensional structural information, but its data is sparse compared to camera data and is still affected by extreme weather. Millimeter-wave radar is not affected by extreme weather and lighting, but it is even sparser than LiDAR. Therefore, place recognition algorithms based on single-modality sensor data are limited by the inherent shortcomings of the sensor and cannot achieve stable recognition effects under various environmental conditions. Multimodal fusion technology is an effective means to overcome the shortcomings of a single sensor. Therefore, introducing multi-modal environmental observation data into place recognition tasks can enhance the generalization ability of place recognition technology under different environmental conditions. Although the existing technology has made progress in research based on multi-sensor fusion, the following problems still exist: (1) Only considering the fusion of camera and LiDAR, it is difficult to apply in rainy and foggy weather; (2) Only using single-view camera data, different modal data cannot be aligned, which is not conducive to multimodal feature fusion; (3) The impact of vehicle heading angle factors on place recognition has not been fully taken into account, and it cannot cope with the change of perspective. Summary of the invention

[0004] In order to solve the above problems, the purpose of the present invention is to provide a multimodal place recognition method based on a multi-scale self-attention mechanism. The method simultaneously takes a surround multi-view RGB image, a lidar environment point cloud, and a millimeter-wave radar environment point cloud as input, and performs multimodal feature fusion through a self-attention mechanism at multiple feature scales to generate a multimodal global descriptor (Lidar-Camera Global Descriptor, LCGD), and screens k candidate samples with the LCGD descriptor according to the Euclidean distance; then, the point cloud scattering cross-sectional area probability density function is extracted from the millimeter-wave radar environment point cloud, the total reordering distance is calculated, the k candidate samples are reordered, and the coordinates of the candidate sample with the smallest reordering distance are used as the coordinates of the intelligent vehicle at the current moment; the method is used to solve the problems of existing place recognition technology in the face of changes in environmental factors such as lighting and weather, and poor robustness to changes in the vehicle's own perspective, thereby enhancing the recognition accuracy of place recognition technology and its generalization under different environmental conditions.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] A multi-modal location recognition method based on a multi-scale self-attention mechanism uses a surround multi-view camera, a multi-line laser radar, and a millimeter wave radar as perception sensors. The multi-modal location recognition method includes the following steps:

[0007] Step S1, using an intelligent vehicle equipped with a surround multi-view camera, a multi-line laser radar and a millimeter-wave radar, to collect multiple groups of environmental observation samples in the environment to form a data set; the environmental observation samples include the coordinates of the intelligent vehicle at the time of collection, the odometer information of the intelligent vehicle, the surround multi-view RGB image, the laser radar environment point cloud and the multi-view millimeter-wave radar environment point cloud; the surround multi-view RGB image includes front view, left front view, left rear view, rear view, right rear view, right front view, 6-view RGB images; the multi-view millimeter-wave radar environment point cloud includes front view, left front view, left rear view, right rear view, right front view;

[0008] Step S2, obtaining a plurality of laser radar depth images by spherical projection of the laser radar environment point cloud of S1;

[0009] Step S3, back-project the surround multi-view RGB image of S1 to the laser radar coordinate system to obtain an RGB point cloud containing both RGB values ​​and three-dimensional coordinates; then stitch the RGB point clouds obtained from multiple viewpoints and obtain a panoramic RGB image from the laser radar perspective through spherical projection;

[0010] Step S4, grouping the multiple laser radar depth images obtained in S2 and the panoramic RGB images obtained in S3 to construct a triple data group; the triple data group includes multiple sample groups, each sample group includes a query sample, a positive sample and multiple negative samples; the triple data group is used as a training data set;

[0011] Step S5, using a multimodal descriptor extraction network to extract and aggregate features of the panoramic RGB image obtained in S3 and the lidar depth map obtained in S2 to obtain an LCGD descriptor; the multimodal descriptor extraction network is composed of a multimodal feature fusion module and a global feature aggregation module;

[0012] The panoramic RGB image obtained by S3 and the lidar depth map obtained by S2 are respectively extracted through the multimodal feature fusion module to obtain visual feature maps and lidar feature maps; the visual feature maps and lidar feature maps are respectively obtained through the global feature aggregation module to obtain visual sub-descriptors and lidar sub-descriptors, and then the two are spliced ​​to obtain the LCGD descriptor;

[0013] Step S6, using the triplet loss Triplet Loss, using the training data set obtained in step S4, to train the multimodal descriptor extraction network in S5, so that the descriptor of the query sample is close to the positive sample descriptor and the negative sample descriptor is not, and a trained multimodal descriptor extraction network is obtained;

[0014] Step S7, using the multimodal descriptor extraction network trained in step S6, and processing each sample in the data subset db in step S4 according to the method in step S5 to obtain a descriptor database;

[0015] Step S8: In the intelligent vehicle positioning stage, the surround multi-view RGB image and the laser radar environment point cloud collected at the current moment are subjected to the operations of steps S2, S3, and S5 to obtain the LCGD descriptor at the current moment; the LCGD descriptor at the current moment is compared with all the descriptors in the descriptor database of S7 according to the Euclidean distance, and the k descriptors with the smallest distance are selected from the descriptor database as candidate descriptors, and the corresponding samples are used as candidate samples;

[0016] Step S9, performing the following operations on the samples in the multi-view millimeter-wave radar environment point cloud collected at the current moment: uniformly projecting the millimeter-wave radar environment point clouds of all view angles of the current frame sample to the forward-looking millimeter-wave radar coordinate system to obtain the millimeter-wave radar environment point cloud in the forward-looking coordinate system; using the vehicle odometer information, projecting the millimeter-wave radar environment point cloud in the forward-looking coordinate system of the three frames before the current frame sample to the current sample coordinate system to obtain the dense millimeter-wave radar environment point cloud at the current moment;

[0017] Step S10, removing the dynamic points in the dense millimeter-wave radar environment point cloud at the current moment obtained in S9 to obtain a static point cloud; calculating the scattering cross-sectional area RCS probability density function of the static point cloud at the current moment;

[0018] Step S11, calculating the k candidate samples obtained in S8 by the method of step S9 and step S10 to obtain the point cloud RCS probability density function of the k candidate samples;

[0019] Step S12, using the RCS probability density function of the point cloud at the current moment obtained in step S10 and the RCS probability density function of the point cloud of the k candidate samples obtained in step S11, calculate the RCS probability distribution distance; combine the RCS probability distribution distance and the Euclidean distance to obtain the total reordering distance; the Euclidean distance is the Euclidean distance between the LCGD descriptor at the current moment obtained in S8 and the k candidate descriptors; reorder the k candidate samples using the total reordering distance;

[0020] Step S13: using the reordering result of step S12, selecting the sample with the smallest reordering distance as the matching sample, and taking the coordinates of the sample as the coordinates of the intelligent vehicle at the current moment, so as to complete the location recognition with higher accuracy;

[0021] Furthermore, the surround multi-view RGB image in step S1 can cover a 360° viewing angle, the lidar point cloud includes the spatial position of the point cloud, and each frame of data in the millimeter wave radar observation sequence includes the point cloud spatial position, radial velocity and RCS value.

[0022] Furthermore, the process of spherical projection of the laser radar point cloud into a depth map in step S2 is given by the following formula:

[0023]

[0024] in is the pixel coordinate of the depth map, r = ‖p‖2 is the distance value of the laser point, p = (xyz) T is the coordinate of the laser point, h and w represent the width and height of the target depth map respectively, and f = f up +f down Indicates the field of view of the laser radar;

[0025] Furthermore, the specific steps of obtaining the panoramic RGB image under the perspective of the laser radar in step S3 are:

[0026] Step S31, performing depth estimation on the surround multi-view RGB images so that each pixel of each image obtains an absolute depth value at a real scale;

[0027] Step S32: Using the absolute depth value obtained in step S31 and the intrinsic and extrinsic parameter matrices of each camera, project the RGB value of each pixel in a certain viewing angle image from the pixel coordinate system to the laser radar coordinate system to obtain an RGB point cloud;

[0028] Step S33: Project each view image using step S32, and stitch all the obtained results to obtain an RGB point cloud containing a 360° view in the laser radar coordinate system;

[0029] Step S34, spherically projecting the RGB point cloud with a 360° viewing angle obtained in step S33 to obtain a panoramic RGB image under the viewing angle of the laser radar;

[0030] Furthermore, the specific steps of constructing the training data set in step S4 are:

[0031] Step S41, in the data set of step S1, based on the distance threshold dis_thres, according to the coordinates in the environmental observation samples, the selected environmental observation samples are called data subset db, and the unselected environmental observation samples are called data subset query, that is, a query sample group is obtained;

[0032] Step S42, in the data subset query of S41, select the environmental observation samples whose sample timestamp is less than the threshold date_thres as the data subset train_query, and the remaining samples as the data subset val_query;

[0033] Step S43: for each query sample in the data subset query, find a negative sample group in the db subset whose distance to the sample is greater than the threshold neg_thres as the negative sample group corresponding to the query sample; find a sample in the db subset whose distance to the sample is less than the threshold pos_thres as the positive sample group corresponding to the query sample, and randomly select a positive sample from the sample as the positive sample corresponding to the query sample;

[0034] Combine each query sample in the data subset train_query and its corresponding positive sample and multiple negative samples to obtain a triple training set; combine each query sample in the data subset val_query and its corresponding positive sample and multiple negative samples to obtain a triple verification set; the triple training set and the triple verification set together constitute the training data set;

[0035] Furthermore, the specific implementation steps of step S5 are:

[0036] Step S51: record the panoramic RGB image obtained in S3 as a image with a size of B×C I ×H I ×W Itensor, B represents the size of the computation batch, H I and W I Represents the height and width of the RGB image, C I represents the number of RGB image channels; the lidar depth map obtained by S2 is recorded as a size of B×C L ×H L ×W L The tensor, H L and W L Respectively represent the height and width of the lidar depth map, C L Indicates the number of channels of the lidar depth map;

[0037] The i-th layer features extracted from the panoramic RGB image and the lidar depth map are respectively recorded as and Will and The fused visual features and lidar features are input into the multimodal feature fusion module to fuse the features of different modalities. and Both have the same shape as the input feature tensor; and and Add them separately to get the modal fusion features and and

[0038] Step S52: modal fusion feature I F and L F The global feature aggregation module is used to perform feature aggregation respectively to obtain visual sub-descriptors. and lidar subdescriptors

[0039] Step S53: I obtained in step S52 D and L D After concatenation in the channel dimension and L2 normalization, we get the LCGD descriptor.

[0040] Furthermore, the formula for global feature aggregation in step S52 is described as follows:

[0041]

[0042] Where N is the number of feature vectors, K is the number of clusters, and w k , b k 、c kare parameters to be learned, x i (j) represents the eigenvector x i The jth component of

[0043] Furthermore, in step S51, and The specific steps of inputting the multimodal feature fusion module together to fuse the features of different modalities are as follows:

[0044] Step S511: The input feature map of the multimodal feature fusion module is recorded as and Corresponding to visual features and lidar features respectively; through two vertical compression VC layers, and Compressed to size and Horizontal characteristics;

[0045] The VC layer includes a 1×1Conv layer, a Sigmoid activation layer, a 1×1Conv layer, a MaxPooling layer, and four 1-DConv layers in sequence; the MaxPooling layer is used for feature downsampling; a skip connection is used between the first and third 1-D Conv layers to accelerate convergence;

[0046] Step S512: splice the output of the VC layer in the width direction to obtain a multimodal feature embedding Then use multi-head self-attention to fuse multimodal features to get F′ e , the feature map size does not change before and after this step;

[0047] Step S513: F′ e Divided into two sizes and Then copy the feature map in the height dimension to obtain the fused features output by the multimodal feature fuser and

[0048] Furthermore, the triple loss function in step S6 is calculated as follows:

[0049]

[0050] where q and p q They represent the query sample and the positive sample corresponding to the query sample, respectively. represents the i-th negative sample corresponding to the query sample, […] + represents the hinge loss function […] + = max(0,…), d(·) represents the Euclidean distance, m is a constant representing the descriptor distance threshold, Nneg is the number of negative samples;

[0051] Furthermore, the specific process of removing dynamic points and calculating the probability density function of the scattering cross-sectional area RCS in step S10 is as follows:

[0052] Step S101: For all the measurement points of the dense millimeter-wave radar in the current frame, the radial velocity of each point relative to the millimeter-wave radar is calculated:

[0053]

[0054] in represents the radial velocity of the ith point, v i represents the velocity of the i-th point in the millimeter-wave radar coordinate system, θ i represents the azimuth of the i-th point, θ vi represents the velocity measurement angle of the i-th point;

[0055] Step S102: Since the radial velocity of most measurement points is 0 when the vehicle is stationary, when the radial velocity of the measurement points of the millimeter-wave radar point cloud frame is less than 1 m / s, it is determined that the vehicle in the frame is stationary; otherwise, it is determined that the vehicle in the frame is in a moving state;

[0056] Step S103: If the vehicle in the current frame is stationary, the measurement point with a radial velocity greater than 1 m / s is determined as a dynamic point; if the vehicle in the current frame is moving, the stationary measurement point satisfies the following equation:

[0057]

[0058] in represents the radial velocity of the ith point, v s is the speed of the millimeter wave radar, α is the vehicle heading angle, θ i is the azimuth of the i-th point; the dynamic measurement point is a discrete point for the above formula, so based on the above formula, α and v s As solution parameters, the random sampling consensus algorithm (RANSAC) algorithm is used to calculate outliers, and outliers are determined as dynamic measurement points, and dynamic points are filtered out;

[0059] Step S104, normalizing the RCS value of the millimeter wave radar measurement point to the range of (0, 1) using the maximum and minimum values ​​of the RCS measurement value of the frame;

[0060] Step S105: bin RCS is the interval size, the RCS histogram is calculated, and normalized on all intervals to eliminate the influence of different number of point clouds in different frames on the histogram, and the RCS probability density function is obtained;

[0061] Furthermore, the specific steps of calculating the total reordering distance in step S12 are as follows:

[0062] Step S121: Calculate the probability distribution distance between the query RCS probability density function and the candidate RCS probability density function based on the KL divergence:

[0063]

[0064] Wherein H(R1) and H(R2) represent the RCS probability density function obtained in step S10, and k is the discrete interval index of the probability density function;

[0065] Step S122: Calculate the total reordering distance using the RCS probability distribution distance and the multimodal descriptor LCGD Euclidean distance:

[0066] d rerank (q,c)=αd R (H(q),H(c))+(1-α)d E (LCGD(q),LCGD(c))

[0067] Where q and c represent query samples and candidate samples respectively, f(·) represents the LCGD descriptor extracted for the input sample according to step S5, and d E (·) represents the Euclidean distance, and α is the set weight used to adjust the proportion of the two distance measures;

[0068] Step S123: Calculate the total re-ranking distance d between the query sample and each candidate sample rerank , and d rerank Arrange the k candidate samples in descending order.

[0069] Beneficial effects:

[0070] 1. The present invention discloses a multimodal place recognition method based on a multi-scale self-attention mechanism. By taking surround multi-view RGB images, laser radar environment point clouds, and millimeter-wave radar environment point clouds as inputs at the same time, and performing multimodal feature fusion at multiple feature scales through a self-attention mechanism, the advantages of different sensors can be brought into play, and the influence of failure of a single sensor on the place recognition effect can be reduced. The problems existing in the existing place recognition technology that the recognition effect decreases when facing changes in environmental factors such as lighting and weather, and the poor robustness of the existing technology to changes in the vehicle's own perspective are solved, and the recognition accuracy of the place recognition technology and its generalization under different environmental conditions are enhanced.

[0071] 2. The present invention discloses a multimodal place recognition method based on a multi-scale self-attention mechanism, which realizes multimodal data collection and data space alignment by simultaneously taking surround multi-view RGB images, laser radar environment point clouds, and millimeter-wave radar environment point clouds as inputs; obtains triplet data sets to construct a training set, and performs multimodal feature fusion through a self-attention mechanism at multiple feature scales to generate a multimodal global descriptor LCGD, and extracts the LCGD descriptors of samples in all databases according to the Euclidean distance, and uses the triplet loss Triplet Loss to train a multimodal multimodal descriptor extraction network; obtains the current moment observation to extract the LCGD descriptor, and queries the k nearest neighbor candidate samples in the database; the present invention realizes more robust place recognition through the fusion of multiple sensors, can give full play to the respective advantages of different sensors, reduce the impact of the failure of a single sensor on the place recognition effect, and improve the robustness of multimodal place recognition.

[0072] 3. The present invention discloses a multimodal place recognition method based on a multi-scale self-attention mechanism. Based on beneficial effect 2, dynamic points are removed from the millimeter-wave radar point cloud, and the probability density function of the scattering cross-section of the millimeter-wave radar point cloud is calculated. The total reordering distance is calculated in combination with the Euclidean distance of the LCGD descriptor, and k candidate LCGD descriptors are reordered. The coordinates of the sample with the closest reordered total distance are used as the coordinates of the current intelligent vehicle, thereby achieving higher-precision place recognition.

[0073] 4. The present invention discloses a multi-modal location recognition method based on a multi-scale self-attention mechanism. In the process of removing dynamic points and calculating the scattering cross-sectional area RCS probability density function, the random sampling consensus algorithm RANSAC algorithm is used to calculate outliers, and the outliers are determined as dynamic measurement points, and the dynamic points are filtered out; the RCS value of the millimeter-wave radar measurement point is normalized to the range of (0,1) using the maximum and minimum values ​​of the RCS measurement value of the frame; and the bin RCS For the interval size, the RCS histogram is calculated and normalized on all intervals to eliminate the impact of different numbers of point clouds in different frames on the histogram, and the RCS probability density function is obtained. The probability distribution distance between the query RCS probability density function and the candidate RCS probability density function is calculated based on the KL divergence; the re-ranking total distance is calculated using the RCS probability distribution distance and the multimodal descriptor LCGD Euclidean distance, which can improve the accuracy of candidate sample retrieval, thereby enhancing the recognition accuracy of place recognition technology and its generalization under different environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The drawings are only for the purpose of illustrating particular embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0075] Figure 1 The flowchart of multimodal place recognition based on multi-scale self-attention mechanism is schematically shown.

[0076] Figure 2 The schematic diagram shows the principle of multimodal place recognition based on multi-scale self-attention mechanism.

[0077] Figure 3 A flowchart of multimodal data spatial alignment is schematically shown.

[0078] Figure 4 The flowchart of constructing the database sample set and the query sample set is schematically shown.

[0079] Figure 5 The flowchart of constructing a triplet training dataset is schematically shown.

[0080] Figure 6 The multimodal descriptor extraction network structure diagram is schematically shown.

[0081] Figure 7 The structure diagram of the multimodal feature fusion module is schematically shown. DETAILED DESCRIPTION

[0082] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0083] A specific embodiment of the present invention discloses a multi-modal place recognition method based on a multi-scale self-attention mechanism, which fuses multi-modal information through a multi-head self-attention mechanism, generates LCGD descriptors, generates candidate samples, and reorders the candidate samples through millimeter-wave radar post-fusion, thereby achieving robust place recognition. The principle of the method is shown in Figure 2 .

[0084] The present invention provides a multimodal place recognition method based on a multi-scale self-attention mechanism, which mainly realizes place recognition by using a deep neural network to extract LCGD descriptors and millimeter-wave radar post-fusion reordering from the input multimodal information. The LCGD descriptor extraction module consists of three modules: multimodal data space alignment, multimodal feature extraction and fusion, and multimodal feature aggregation. The millimeter-wave radar post-fusion reordering includes dynamic point filtering, RCS probability density function calculation, and a module for reordering candidate descriptors.

[0085] like Figure 1 As shown, the present embodiment discloses a multimodal place recognition method based on a multi-scale self-attention mechanism, and the specific implementation steps are as follows:

[0086] Step S1: using an intelligent vehicle equipped with a surround multi-view camera, a multi-line laser radar, and a millimeter-wave radar to collect multiple sets of environmental observation samples in the environment to form a data set;

[0087] Specifically, when the smart vehicle collects environmental data, preferably, the laser radar beam carried should be greater than or equal to 32 to obtain a better recognition effect, and the millimeter-wave radar carried is a frequency modulated continuous wave (Frequency Modulated Continuous Wave, FMCW) radar, which can measure distance and speed at the same time. When collecting environmental data, it is necessary to ensure that the recorded environmental roads and building structures have not changed on a large scale; the smart vehicle is equipped with an inertial navigation device to remove invalid points, points outside the area of ​​interest, and correct motion distortion from the collected original point cloud data. The surround multi-view RGB image includes: front view, left front view, left rear view, rear view, right rear view, right front view, 6-view RGB images; the multi-view millimeter-wave radar environmental point cloud includes: front view, left front view, left rear view, right rear view, right front view;

[0088] Step S2, obtaining a plurality of laser radar depth images by spherical projection of the laser radar environment point cloud of S1;

[0089] Specifically, when the LiDAR point cloud is calculated through spherical projection to obtain the LiDAR depth map, the coordinate conversion formula from the three-dimensional coordinates of the laser point to is:

[0090]

[0091] in is the pixel coordinate of the depth map, r = ‖p‖2 is the distance value of the laser point, p = (xyz) T is the coordinate of the laser point, h and w represent the width and height of the target depth map respectively, and f = f up +f down Represents the field of view of the laser radar; if multiple point clouds are projected onto the same depth map coordinate through the above formula, the minimum depth of these point clouds is selected as the depth value at that coordinate of the depth map.

[0092] Through spherical projection, the originally disordered and huge sparse point cloud data is transformed into ordered deep data with reduced data volume and more suitable for deep neural network processing.

[0093] Step S2, obtaining a plurality of laser radar depth images by spherical projection of the laser radar environment point cloud of S1;

[0094] Step S3, back-project the surround multi-view RGB image of S1 to the laser radar coordinate system to obtain an RGB point cloud containing both RGB values ​​and three-dimensional coordinates; then stitch the RGB point clouds obtained from multiple viewpoints and obtain a panoramic RGB image from the laser radar perspective through spherical projection;

[0095] like Figure 3 As shown, firstly, the depth of the multi-view RGB image is estimated to obtain the absolute depth value of each pixel in each image at the real scale. Afterwards, the estimated absolute depth value d and the intrinsic and extrinsic parameters of each camera are used to project the RGB value of each pixel in a certain view image from the pixel coordinate system to the lidar coordinate system to obtain an RGB point cloud. The RGB point cloud data contains both the three-dimensional coordinates and RGB values ​​of the point cloud. For each view image, the process described in step S32 is used to calculate and splice the results to obtain an RGB point cloud containing a 360° view in the lidar coordinate system.

[0096] Afterwards, the RGB point cloud containing a 360° viewing angle is projected on the spherical surface as described in step S2 to obtain a panoramic RGB image from the perspective of the lidar, thereby completing the cross-modal data spatial alignment.

[0097] Step S4, grouping the multiple laser radar depth images obtained in S2 and the panoramic RGB images obtained in S3 to construct a triple data group; the triple data group includes multiple sample groups, each sample group includes a query sample, a positive sample and multiple negative samples; the triple data group is used as a training data set; the construction process is as follows Figure 4 and Figure 5 As shown;

[0098] First, from the data set collected in step S1, the database sample set and the query sample set are divided. Define the data sets db and query as the database sample set and query sample set respectively. For each sample, calculate the distance between the sample and all samples in the db set. If the minimum distance is greater than the threshold dis_thres or the db set is empty, then add the sample to the db set, otherwise add the sample to the query set. After the above operations, the database set is actually obtained based on coordinate interval sampling with dis_thres as the distance threshold.

[0099] After obtaining the database sample set db and the query sample set query, according to the sample timestamp, add the samples in the query set whose timestamp is less than a threshold date_thres to the train_query set for training, and add the remaining samples to the val_query set for verification;

[0100] For each query sample in the train_query subset and the val_query subset, find a sample in the db subset whose distance to the sample is less than the threshold pos_thres as the positive sample group corresponding to the query sample, and randomly select a positive sample from it; for each sample in the query subset, find a negative sample group in the db subset whose distance to the sample is greater than the threshold neg_thres as the negative sample group corresponding to the query sample; combine each query sample in the data subset train_query and its corresponding positive sample and multiple negative samples to obtain a triple training set; combine each query sample in the data subset val_query and its corresponding positive sample and multiple negative samples to obtain a triple verification set; the triple training set and the triple verification set together constitute the training data set;

[0101] Step S5: adopt Figure 6 The multimodal descriptor extraction network shown extracts and aggregates features of the panoramic RGB image obtained by S3 and the lidar depth map obtained by S2 to obtain an LCGD descriptor; the multimodal descriptor extraction network is composed of a multimodal feature fusion module and a global feature aggregation module;

[0102] First, the input panoramic RGB image is recorded as a B×C I ×H I ×W I tensor, B represents the data batch size, H I and W I Represents the height and width of the RGB image, C I Indicates the number of RGB image channels; the input lidar depth map is recorded as a size of B×C L ×H L ×W L The tensor, H L and W L Respectively represent the height and width of the lidar depth map, C L Represents the number of channels of the LiDAR depth map; ResNet, a residual feature extraction network loaded with pre-trained weights, is used to extract features from the multi-view RGB image and the LiDAR depth map. The extracted i-th layer features are recorded as and Will and They are input into the multimodal feature fusion module to fuse the features of different modalities. The fused visual features and lidar features are and Both have the same shape as the input feature tensor; and and Add them separately to get the modal fusion features and Afterwards and The model is sent to the next layer of the ResNet network and the above process is repeated for feature extraction and fusion. and The global feature aggregation module is used to perform feature aggregation respectively to obtain visual sub-descriptors. and lidar subdescriptors Finally, the sub-descriptor I obtained in the above steps is D and L D Splicing in the channel dimension and then normalizing by L2 to obtain a multimodal global descriptor

[0103] Furthermore, the structure of the multimodal feature fusion module in step S5 is as follows: Figure 7 As shown. The input feature map of this module is recorded as and Corresponding to the visual branch and the lidar branch respectively, i represents the number of layers of the feature extractor. For the convenience of writing, the module description no longer uses the subscript i to specify the number of layers. First, two vertically compressed (VC) layers are used to compress and Compressed to size and The VC layer includes a 1×1Conv layer, a Sigmoid activation layer, a 1×1Conv layer, a MaxPooling layer, and four 1-D Conv layers in sequence; the MaxPooling layer is used for feature downsampling; a jump connection is used between the first and third 1-DConv layers to accelerate convergence. The output of the VC layer is then spliced ​​in the width direction to obtain a multimodal feature embedding Then use multi-head self-attention to fuse multimodal features to get F′ e , the feature map size does not change before and after this step. e Divided into two sizes and Then copy the feature map in the height dimension to restore it to the same size as the input and The same size, finally get the fusion feature of the i-th layer multimodal feature fusion device and

[0104] Furthermore, the formula for performing feature aggregation using the differentiable feature aggregation module in step S5 is described as follows:

[0105]

[0106] Where N is the number of feature vectors, K is the number of clusters, and w k , b k 、c k are parameters to be learned, x i (j) represents the eigenvector x i The jth component of

[0107] Step S6, using the triplet loss Triplet Loss, using the training data set obtained in step S4, to train the multimodal descriptor extraction network in S5, so that the descriptor of the query sample is close to the positive sample descriptor and the negative sample descriptor is not, and a trained multimodal descriptor extraction network is obtained;

[0108] Furthermore, the triplet loss used in deep neural network training is calculated as follows:

[0109]

[0110] where q and p q They represent the query sample and the positive sample corresponding to the query sample, respectively. represents the i-th negative sample corresponding to the query sample, […] + represents the hinge loss function […] + = max(0,…), d(·) represents the Euclidean distance, m is a constant representing the descriptor distance threshold, N neg is the number of negative samples. The purpose of this loss function is to reduce the distance between the LCGD descriptor of the query sample and the LCGD descriptor of the positive sample through learning, while increasing the distance between the LCGD descriptor of the query sample and the negative sample.

[0111] Step S7, using the multimodal descriptor extraction network trained in step S6, and processing each sample in the data subset db in step S4 according to the method in step S5 to obtain a descriptor database;

[0112] Step S8: In the intelligent vehicle positioning stage, the surround multi-view RGB image and the laser radar environment point cloud collected at the current moment are subjected to the operations of steps 2, 3, and 5 to obtain the LCGD descriptor at the current moment; the LCGD descriptor at the current moment is compared with all the descriptors in the descriptor database of S7 according to the Euclidean distance, and the k descriptors with the smallest distance are selected from the descriptor database as candidate descriptors, and the corresponding samples are used as candidate samples;

[0113] Step S9, performing the following operations on the samples in the multi-view millimeter-wave radar environment point cloud collected at the current moment: uniformly projecting the millimeter-wave radar environment point clouds of all view angles of the current frame sample to the forward-looking millimeter-wave radar coordinate system to obtain the millimeter-wave radar environment point cloud in the forward-looking coordinate system; using the vehicle odometer information, projecting the millimeter-wave radar environment point cloud in the forward-looking coordinate system of the three frames before the current frame sample to the current sample coordinate system to obtain the dense millimeter-wave radar environment point cloud at the current moment;

[0114] Step S9, removing the dynamic points in the dense millimeter-wave radar environment point cloud at the current moment obtained in S9 to obtain a static point cloud; and calculating the scattering cross-sectional area RCS probability density function of the static point cloud at the current moment.

[0115] Different dynamic point filtering strategies are required for the two states of vehicle movement and stillness. First, for all the dense millimeter-wave radar measurement points of the current frame, calculate the radial velocity of each point relative to the millimeter-wave radar:

[0116]

[0117] in represents the radial velocity of the ith point, v i represents the velocity of the i-th point in the millimeter-wave radar coordinate system, θ i represents the azimuth of the i-th point, θ vi Represents the velocity measurement angle of the i-th point.

[0118] Since when the vehicle is stationary, the radial velocity of most measurement points is 0, when the radial velocity of the millimeter-wave radar point cloud of this frame is less than the threshold value v_thres, the vehicle of this frame is judged to be stationary; otherwise, the vehicle of this frame is judged to be in a moving state. If the vehicle of the current frame is stationary, the measurement points with radial velocity greater than 1m / s are judged as dynamic points; it should be noted that when collecting data, the real-time moving speed of the vehicle body can also be recorded, and the real-time speed record data can also be used to directly determine whether the vehicle is in a moving state.

[0119] If the vehicle is in a moving state in the current frame, the stationary measurement points in the environment satisfy the following equation:

[0120]

[0121] in represents the radial velocity of the ith point, v s is the speed of the millimeter wave radar, α is the vehicle heading angle, θ iis the azimuth of the i-th point. The static measurement points in the environment should conform to the above formula, while the dynamic measurement points will not strictly conform to the above formula because their own speed is arbitrary. Therefore, the dynamic measurement points in the environment are discrete points for the above formula. Therefore, based on the above formula, α and v s As a solution parameter, the random sampling consensus algorithm (RANSAC) algorithm is used to calculate outliers, and the outliers are determined as dynamic measurement points, and the dynamic points are filtered out based on the determination result.

[0122] Then, the RCS value of the millimeter-wave radar measurement point is normalized to the range of (0,1) using the maximum and minimum values ​​of the RCS measurement value of the frame. RCS The RCS histogram is calculated and normalized over all intervals to eliminate the influence of different numbers of point clouds in different frames on the histogram, and the RCS probability density function is obtained.

[0123] Step S11, calculating the k candidate samples obtained in S8 by the method of step S9 and step S10 to obtain the point cloud RCS probability density function of the k candidate samples;

[0124] Step S12, using the RCS probability density function of the point cloud at the current moment obtained in step S10 and the RCS probability density function of the point cloud of the k candidate samples obtained in step S11, calculate the RCS probability distribution distance; combine the RCS probability distribution distance and the Euclidean distance to obtain the total reordering distance; the Euclidean distance is the Euclidean distance between the LCGD descriptor at the current moment obtained in S8 and the k candidate descriptors; reorder the k candidate samples using the total reordering distance;

[0125] First, the probability distribution distance between the query RCS probability density function and the candidate RCS probability density function is calculated based on the KL divergence:

[0126]

[0127] Where P(R q ) and P(R candidate ) represents the RCS probability density function obtained in step S9, corresponding to the query sample and the candidate sample respectively, and k is the discrete interval index of the probability density function.

[0128] The total re-ranking distance is calculated using the RCS probability distribution distance and the multimodal descriptor LCGD Euclidean distance:

[0129] d rerank (q,c)=αd R (H(q),H(c))+(1-α)d E (LCGD(q),LCGD(c))

[0130] Where q and c represent query samples and candidate samples respectively, LCGD(·) represents the multimodal descriptor extracted for the input sample according to step S5, and d E (·) represents the Euclidean distance, and α is the set weight used to adjust the proportion of the two distance measures.

[0131] Step S13, using the reordering result of step S12, select the sample with the smallest reordering distance as the matching sample, and take the coordinates of the sample as the coordinates of the intelligent vehicle at the current moment, so as to complete the location recognition with higher accuracy;

[0132] Through the above steps, the measurement information of three sensors: surround multi-view camera, lidar, and millimeter-wave radar are used as data sources at the same time, multimodal feature fusion is performed on multiple scales based on the multi-head self-attention mechanism, and the candidate locations are re-ranked by millimeter-wave radar post-fusion. This can solve the problems existing in existing location recognition technologies in the face of changes in environmental factors such as changes in lighting conditions and weather conditions, as well as poor robustness to changes in the vehicle's own perspective. A novel location recognition solution is provided to enhance the recognition accuracy of location recognition technology and its generalization under different environmental conditions.

[0133] Those skilled in the art will appreciate that all or part of the processes of the above-mentioned embodiments can be implemented by hardware related to computer program instructions, and the program can be stored in a computer-readable storage medium, wherein the computer-readable storage medium is a disk, an optical disk, a read-only storage memory, or a random access memory, etc.

[0134] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A multimodal place recognition method based on a multi-scale self-attention mechanism, characterized by: The following steps are included: Step S1, using an intelligent vehicle equipped with a surround multi-view camera, a multi-line laser radar and a millimeter-wave radar, to collect multiple groups of environmental observation samples in the environment to form a data set; the environmental observation samples include the coordinates of the intelligent vehicle at the time of collection, the odometer information of the intelligent vehicle, the surround multi-view RGB image, the laser radar environment point cloud and the multi-view millimeter-wave radar environment point cloud; the surround multi-view RGB image includes front view, left front view, left rear view, rear view, right rear view, right front view, 6-view RGB images; the multi-view millimeter-wave radar environment point cloud includes front view, left front view, left rear view, right rear view, right front view; Step S2, obtaining a plurality of laser radar depth images by spherical projection of the laser radar environment point cloud of S1; Step S3, back-project the surround multi-view RGB image of S1 to the laser radar coordinate system to obtain an RGB point cloud containing both RGB values ​​and three-dimensional coordinates; then stitch the RGB point clouds obtained from multiple viewpoints and obtain a panoramic RGB image from the laser radar perspective through spherical projection; Step S4, grouping the multiple laser radar depth images obtained in S2 and the panoramic RGB images obtained in S3 to construct a triple data group; the triple data group includes multiple sample groups, each sample group includes a query sample, a positive sample and multiple negative samples; the triple data group is used as a training data set; Step S5, using a multimodal descriptor extraction network to extract and aggregate features of the panoramic RGB image obtained in S3 and the lidar depth map obtained in S2 to obtain an LCGD descriptor; the multimodal descriptor extraction network is composed of a multimodal feature fusion module and a global feature aggregation module; The panoramic RGB image obtained by S3 and the lidar depth map obtained by S2 are respectively extracted with the multimodal feature fusion module to obtain the visual feature map and the lidar feature map; the visual feature map and the lidar feature map are respectively obtained by the global feature aggregation module to obtain the visual sub-descriptor and the lidar sub-descriptor, and the visual sub-descriptor and the lidar sub-descriptor are spliced ​​to obtain the LCGD descriptor; Step S6, using the triplet loss Triplet Loss, using the training data set obtained in step S4, to train the multimodal descriptor extraction network in S5, so that the descriptor of the query sample is close to the positive sample descriptor and the negative sample descriptor is not, and a trained multimodal descriptor extraction network is obtained; Step S7, using the multimodal descriptor extraction network trained in step S6, and processing each sample in the data subset db in step S4 according to the method in step S5 to obtain a descriptor database; Step S8: In the intelligent vehicle positioning stage, the surround multi-view RGB image and the laser radar environment point cloud collected at the current moment are subjected to the operations of steps S2, S3, and S5 to obtain the LCGD descriptor at the current moment; the LCGD descriptor at the current moment is compared with all the descriptors in the descriptor database of S7 according to the Euclidean distance, and the k descriptors with the smallest distance are selected from the descriptor database as candidate descriptors, and the corresponding samples are used as candidate samples; Step S9, performing the following operations on the samples in the multi-view millimeter-wave radar environment point cloud collected at the current moment: uniformly projecting the millimeter-wave radar environment point clouds of all view angles of the current frame sample to the forward-looking millimeter-wave radar coordinate system to obtain the millimeter-wave radar environment point cloud in the forward-looking coordinate system; using the vehicle odometer information, projecting the millimeter-wave radar environment point cloud in the forward-looking coordinate system of the three frames before the current frame sample to the current sample coordinate system to obtain the dense millimeter-wave radar environment point cloud at the current moment; Step S10, removing the dynamic points in the dense millimeter-wave radar environment point cloud at the current moment obtained in S9 to obtain a static point cloud; calculating the scattering cross-sectional area RCS probability density function of the static point cloud at the current moment; Step S11, calculating the k candidate samples obtained in S8 by the method of step S9 and step S10 to obtain the point cloud RCS probability density function of the k candidate samples; Step S12, using the RCS probability density function of the point cloud at the current moment obtained in step S10 and the RCS probability density function of the point cloud of the k candidate samples obtained in step S11, calculate the RCS probability distribution distance; combine the RCS probability distribution distance and the Euclidean distance to obtain the total reordering distance; the Euclidean distance is the Euclidean distance between the LCGD descriptor at the current moment obtained in S8 and the k candidate descriptors; reorder the k candidate samples using the total reordering distance; Step S13: Using the reordering result of step S12, select the sample with the smallest reordering distance as the matching sample, and take the coordinates of the sample as the coordinates of the smart vehicle at the current moment, so as to complete higher-precision location recognition.

2. The multimodal place recognition method based on a multi-scale self-attention mechanism according to claim 1, characterized in that: The specific implementation method of step S3 is: Step S31, performing depth estimation on the surround multi-view RGB images so that each pixel of each image obtains an absolute depth value at a real scale; Step S32: Using the absolute depth value obtained in step S31 and the intrinsic and extrinsic parameter matrices of each camera, project the RGB value of each pixel in a certain viewing angle image from the pixel coordinate system to the laser radar coordinate system to obtain an RGB point cloud; Step S33: Project each view image using step S32, and stitch all the obtained results to obtain an RGB point cloud containing a 360° view in the laser radar coordinate system; Step S34: Spherically project the RGB point cloud with a 360° viewing angle obtained in step S33 to obtain a panoramic RGB image from the perspective of the laser radar.

3. The multimodal place recognition method based on a multi-scale self-attention mechanism as claimed in claim 2, characterized in that: The specific implementation method of step S4 is: Step S41, in the data set of step S1, based on the distance threshold dis_thres, according to the coordinates in the environmental observation samples, the selected environmental observation samples are called data subset db, and the unselected environmental observation samples are called data subset query, that is, the query sample set is obtained; Step S42, in the data subset query of S41, select the environmental observation samples whose sample timestamp is less than the threshold date_thres as the data subset train_query, and the remaining samples as the data subset val_query; Step S43: for each query sample in the data subset query, find a negative sample group in the db subset whose distance to the sample is greater than the threshold neg_thres as the negative sample group corresponding to the query sample; find a sample in the db subset whose distance to the sample is less than the threshold pos_thres as the positive sample group corresponding to the query sample, and randomly select a positive sample from the sample as the positive sample corresponding to the query sample; Each query sample in the data subset train_query and its corresponding positive sample and multiple negative samples are combined to obtain a triplet training set; each query sample in the data subset val_query and its corresponding positive sample and multiple negative samples are combined to obtain a triplet verification set; the triplet training set and the triplet verification set together constitute the training data set.

4. The multimodal place recognition method based on a multi-scale self-attention mechanism as claimed in claim 3, characterized in that: The specific implementation method of step S5 is: Step S51: record the panoramic RGB image obtained in S3 as a image with a size of B×C I ×H I ×W I tensor, B represents the size of the computation batch, H I and W I Represents the height and width of the RGB image, C I represents the number of RGB image channels; the lidar depth map obtained by S2 is recorded as a size of B×C L ×H L ×W L Tensors, HL and W L Respectively represent the height and width of the lidar depth map, C L Indicates the number of channels of the lidar depth map; The i-th layer features extracted from the panoramic RGB image and the LiDAR depth map are respectively recorded as and Will and The fused visual features and lidar features are input into the multimodal feature fusion module to fuse the features of different modalities. and Both have the same shape as the input feature tensor; and and Add them separately to get the modal fusion features and and Step S52: modal fusion feature I F and L F The global feature aggregation module is used to perform feature aggregation respectively to obtain visual sub-descriptors. and lidar subdescriptors Step S53: I obtained in step S52 D and L D After concatenation in the channel dimension and L2 normalization, we get the LCGD descriptor.

5. The multimodal place recognition method based on multi-scale self-attention mechanism according to claim 4, characterized in that: Step S51 and The specific method of inputting the multimodal feature fusion module together to fuse the features of different modalities is as follows: Step S511: The input feature map of the multimodal feature fusion module is recorded as and They correspond to visual features and lidar features respectively; Through two vertical compression VC layers, and Compressed to size and Horizontal characteristics; The VC layer includes a 1×1 Conv layer, a Sigmoid activation layer, a 1×1 Conv layer, a MaxPooling layer, and four 1-D Conv layers in sequence; the MaxPooling layer is used for feature downsampling; a skip connection is used between the first and third 1-D Conv layers to accelerate convergence; Step S512: splice the output of the VC layer in the width direction to obtain a multimodal feature embedding Then use multi-head self-attention to fuse multimodal features to get F′e ， The size of the feature map does not change before and after this step; Step S513: F′ e Divided into two sizes and Then copy the feature map in the height dimension to obtain the fused features output by the multimodal feature fuser and 6. The multimodal place recognition method based on multi-scale self-attention mechanism according to claim 5, characterized in that: The process of spherical projection of the laser radar point cloud into a depth map in step S2 is given by the following formula: in is the pixel coordinate of the depth map, r = ||p||2 is the distance value of the laser point, p = (xyz) T is the coordinate of the laser point, h and w represent the width and height of the target depth map respectively, and f = f up +f down Represents the field of view of the lidar. The triplet loss function in step S6 is calculated as follows: where q and p q They represent the query sample and the positive sample corresponding to the query sample, respectively. represents the i-th negative sample corresponding to the query sample, […] + represents the hinge loss function […] + =max(0, ...), d(·) represents the Euclidean distance, m is a constant representing the descriptor distance threshold, N neg is the number of negative samples.

7. The multimodal place recognition method based on multi-scale self-attention mechanism according to claim 6, characterized in that: The specific method of removing dynamic points and calculating the probability density function of the scattering cross-sectional area RCS in step S10 is: Step S101: For all the measurement points of the dense millimeter-wave radar in the current frame, the radial velocity of each point relative to the millimeter-wave radar is calculated: in represents the radial velocity of the ith point, v i represents the velocity of the i-th point in the millimeter-wave radar coordinate system, θ i represents the azimuth of the i-th point, θ vi represents the velocity measurement angle of the i-th point; Step S102: Since the radial velocity of most measurement points is 0 when the vehicle is stationary, when the radial velocity of the measurement points of the millimeter-wave radar point cloud frame is less than 1 m / s, it is determined that the vehicle in the frame is stationary; Otherwise, it is determined that the vehicle in this frame is in a moving state; Step S103: If the vehicle in the current frame is stationary, the measurement point with a radial velocity greater than 1 m / s is determined as a dynamic point; if the vehicle in the current frame is moving, the stationary measurement point satisfies the following equation: in represents the radial velocity of the ith point, v s is the speed of the millimeter wave radar, α is the vehicle heading angle, θ i is the azimuth of the i-th point; the dynamic measurement point is a discrete point for the above formula, so based on the above formula, α and v s As solution parameters, the random sampling consensus algorithm RANSAC algorithm is used to calculate outliers, and outliers are determined as dynamic measurement points, and dynamic points are filtered out; Step S104: normalize the RCS value of the millimeter wave radar measurement point to the range of (0, 1) using the maximum and minimum values ​​of the RCS measurement value of the frame; Step S105: bin RCS The RCS histogram is calculated and normalized over all intervals to eliminate the influence of different numbers of point clouds in different frames on the histogram, and the RCS probability density function is obtained.

8. The multimodal place recognition method based on multi-scale self-attention mechanism according to claim 7, characterized in that: The specific steps of calculating the total reordering distance in step S12 are as follows: Step S121: Calculate the probability distribution distance between the query RCS probability density function and the candidate RCS probability density function based on the KL divergence: Wherein H(R1) and H(R2) represent the RCS probability density function obtained in step S10, and k is the discrete interval index of the probability density function; Step S122: Calculate the total reordering distance using the RCS probability distribution distance and the multimodal descriptor LCGD Euclidean distance: d rerank (q, c) = ad R (H(q), H(c))+(1-α)d E (LCGD(q),LCGD(c)) Where q and c represent query samples and candidate samples respectively, f(·) represents the LCGD descriptor extracted for the input sample according to step S5, and d E (·) represents the Euclidean distance, and α is the set weight used to adjust the proportion of the two distance measures; Step S123: Calculate the total re-ranking distance d between the query sample and each candidate sample rerank , and d rerank Arrange the k candidate samples in descending order.

Images