A multi-modal descriptor place recognition method based on mamba mechanism

Abstract

The application discloses a kind of multi-modal descriptor place recognition methods based on Mamba mechanism, comprising: by several levels of fusion module, feature extraction is carried out to multi-view image and distance image, and image branch feature and laser radar branch feature are obtained;Wherein, the fusion module includes Mamba Fusion module, Mamba Fusion module is handled to the input feature after convolution processing, and split feature is obtained, the input feature and split feature are spliced, and the output feature of current fusion module is obtained as the input feature of next level fusion module, image branch feature and laser radar branch feature are compressed respectively, and image branch panoramic feature and laser radar branch panoramic feature are obtained;Image branch panoramic feature and laser radar branch panoramic feature are extracted and fused, and global multi-modal descriptor is obtained, according to place global multi-modal descriptor search, and place recognition result is obtained, to reach positioning function.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer vision, and particularly relates to a multi-modal descriptor place recognition method based on a Mamba mechanism. BACKGROUND

[0002] Place recognition is an important task in the field of computer vision, and has a wide range of practical applications, such as the closed loop of SLAM or autonomous driving. Place recognition is usually a retrieval problem, which compares the current sensor observation information with the map or the database of previous observations, retrieves the closest observation, and determines whether the sensor is currently located at a previously visited location. It is an important branch of simultaneous localization and mapping (SLAM), which can find the re-visited location and thus reduce the cumulative error.

[0003] Multi-modal fusion is essentially to mine the complementary information between different sensors and different modalities. Using a single sensor for place recognition will be affected by different environments. For example, using an RGB camera alone will be affected by different weather and light changes; using a laser radar alone cannot capture the fine details of the observed scene. In order to improve the place recognition problem and improve the robustness, the respective advantages of radar and camera are used to enhance the ability of place recognition in a fusion manner.

[0004] The traditional method of fusing camera and laser radar is to use Tansformer and CNN to splice and weighted average the camera and laser radar data, but this will increase the computational complexity, especially when dealing with long sequences and high-dimensional features, the computational and memory requirements increase significantly; there is also the problem of real-time, the calculation delay may affect the response time of the system, especially in dynamic environments, the above problems are more prominent. SUMMARY

[0005] To solve the above technical problems, the application provides a multi-modal descriptor place recognition method based on a Mamba mechanism to solve the problems existing in the prior art.

[0006] To achieve the above purpose, the application provides a multi-modal descriptor place recognition method based on a Mamba mechanism, comprising:

[0007] obtaining a multi-view image and a distance image;

[0008] Feature extraction is performed on the multi-view image and the range image through several levels of fusion modules to obtain image branch features and lidar branch features; wherein the fusion module includes a Mamba Fusion module. In the fusion module, the input end of the MambaFusion module is connected to two convolution layers for processing the input features and then inputting them into the MambaFusion module. The Mamba Fusion module processes the input features after the convolution processing to obtain multi-view image splitting features and lidar splitting features. The input features and the multi-view image splitting features and the lidar splitting features are correspondingly spliced ​​to obtain the output features of the current level fusion module as the input features of the next level fusion module, wherein the input features of the first level fusion module are the multi-view image and the range image, and the output of the last level fusion module is the image branch features and the lidar branch features.

[0009] The image branch features and the lidar branch features are compressed respectively to obtain the image branch panoramic features and the lidar branch panoramic features;

[0010] The image branch panoramic features and lidar branch panoramic features are perceptually extracted and fused to obtain a global multimodal descriptor.

[0011] Optionally, the process of acquiring the range image includes:

[0012] Point cloud data is acquired, and spherical projection is performed on the point cloud data to obtain a range image.

[0013] Optionally, in the Mamba Fusion module, the input features are reshaped and spliced, and the spliced ​​features are input into the Mamba module. The spliced ​​features are processed by the Mamba module, and the features processed by the Mamba module are fused and split and reshaped so that the split and reshaped features have the same size as the input features, thereby obtaining output features, wherein the output features include image coding features and lidar coding features, and the image coding features and lidar coding features are used as input features of the next-level fusion module.

[0014] Optionally, in the Mamba module, the Mamba module includes a first linear layer and a second linear layer connected in parallel, wherein the first linear layer and the second linear layer are used to perform linear processing on the spliced ​​features, and the output of the first linear layer is directly used as the A matrix of the SSM. The X matrix, B matrix, and C matrix of the SSM are obtained by calculating the output of the first linear layer after the convolution layer processing through different activation functions. The output of the SMM is spliced ​​and normalized by weighted union and the calculation result of the output of the second linear layer after the convolution layer processing through the activation function. The normalized result is processed by the third linear layer and normalized again to obtain the features processed by the Mamba module.

[0015] Optionally, the image branch feature and the lidar branch feature are compressed respectively by a VC layer.

[0016] Optionally, the image branch panoramic feature and the lidar branch panoramic feature are perceived and extracted by a perception module, wherein the perception module comprises an MLP layer, a NetVLAD and an MLP layer connected in sequence.

[0017] Optionally, the image branch panoramic feature and the lidar branch panoramic feature are perceived and extracted to obtain image branch sub-descriptors and lidar sub-descriptors, and the image branch sub-descriptors and the lidar sub-descriptors are fused by a splicing operation to obtain global multi-modal descriptors.

[0018] Optionally, the process of obtaining the location recognition result comprises: comparing and querying the location global multi-modal descriptor with data stored in a database to obtain the most similar descriptor to the location global multi-modal descriptor, and taking the location data corresponding to the most similar descriptor as the location recognition result.

[0019] Compared with the prior art, the present application has the following advantages and technical effects:

[0020] The present application provides a camera and radar fusion multi-modal descriptor location recognition method and system based on Mamba mechanism, which is used for intelligent automobile automatic driving global positioning and SLAM closed loop location recognition. The present application considers that the Transformer model will produce quadratic complexity when performing self-attention mechanism, causing slow fusion speed, low efficiency and other problems. In order to solve the above-mentioned defects, the fusion module adopts Mamba mechanism, which is based on state space model (SSM), which makes it able to capture the global context information of the input data, not just local features. Compared with the traditional Transformer-based method, the Mamba Fusion module maintains linear computational complexity. This global perception ability helps to retain and integrate more extensive spatial information in the location recognition process, thereby improving the accuracy and robustness of recognition. Through multi-scale extraction and fusion, a multi-modal descriptor is finally formed. BRIEF DESCRIPTION OF DRAWINGS

[0021] The accompanying drawings, which form a part of this application, are included to provide a further understanding of the application and are incorporated in and constitute a part of this application. The embodiments of this application and their explanations are used to explain this application, and do not constitute an improper limitation on this application. In the drawings:

[0022] Figure 1 The flowchart of the multi-modal descriptor location recognition method based on Mamba mechanism of the embodiments of the present application;

[0023] Figure 2 This is a Mamba Fusion module fusion flow chart of an embodiment of the present invention. DETAILED DESCRIPTION

[0024] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0025] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0026] In response to the shortcomings of the existing technology, the present invention proposes a multimodal descriptor location recognition method based on the Mamba mechanism. It adopts the structure of a deep learning network and the Mamba mechanism model to fuse the camera and lidar to generate a multimodal descriptor, thereby improving the accuracy, robustness and recognition speed of location recognition.

[0027] The Mamba model introduces the concept of a selective state space to achieve linear-time sequence modeling. When processing long sequences, its computational complexity grows linearly with sequence length, rather than quadratically as in the Transformer, significantly improving computational efficiency. Furthermore, the Mamba model employs a hardware-aware parallel scanning algorithm. This algorithm optimizes the computational process through a recursive pattern, reducing I / O accesses between GPU memory levels. This results in up to a 3x improvement in computational speed on the RTX3090Ti GPU. The Mamba model introduces a novel selectivity mechanism that allows the model to dynamically adjust its behavior based on the input. This mechanism effectively filters out irrelevant information and enhances task-relevant information, similar to the gating mechanism introduced in RNNs, but with greater flexibility within the framework of the SSM. Furthermore, the Mamba model's linear-time complexity makes it more efficient when processing long sequences. When processing multimodal data, Mamba leverages its ability to capture long-range dependencies, leveraging its efficiency advantages in learning and inference, making it even more efficient in processing multimodal data. Therefore, we propose to apply the Mamba mechanism to fuse camera and lidar data to overcome the computational complexity and real-time issues, generate multimodal descriptors, and improve the robustness, accuracy and speed of recognition in location recognition.

[0028] like Figure 1As shown, this embodiment provides a multimodal descriptor location recognition method based on the Mamba mechanism, including the following steps:

[0029] First, point cloud data is acquired through LIDAR radar, and multi-view images are obtained through cameras with different perspectives. In order to solve the disordered arrangement of point cloud data, the point cloud data is converted into 2D range images after relevant preprocessing, and the range images are used as the input of the point cloud feature extraction descriptor.

[0030] After that, the input convolution layer Input conv performs preliminary feature extraction on the multi-view image (RGB image) and the distance image corresponding to the point cloud data obtained by LIDAR. (The main information extracted from the multi-view image is the color and texture information of the environment, which helps to identify the semantic content in the environment; the LiDAR point cloud is projected into a distance image, which contains the distance information from each point in the point cloud data to the sensor. This information is used as the input for feature extraction). After extracting the information contained in the multi-view image and the LIDAR distance image, the features of the multi-view image and the distance image at different resolutions are obtained respectively.

[0031] The collected multi-view image features and range image features are then fed into the Mamba Fusion module for multi-scale fusion. Because multi-view images carry color and texture information and primarily lack depth information, while range images carry depth information, the Mamba Fusion module processes the features from the image branch and the lidar branch, combining the multi-view image features with the range image features. This creates a fused feature that carries both texture and depth information. The fused feature is then segmented and reshaped to its original size, then fed back into the original branch and fused with the corresponding original features to extract feature information from different scales.

[0032] After four Mamba module processings, multi-view images and range image information features with different extraction levels are fused and extracted, making full use of the panoramic views of the environment from different sensor modalities and their mutual relationships to generate discriminative place identification descriptors, including image branch features and lidar branch features.

[0033] After that, the outputs of the two encoding branches (image branch features and lidar branch features) are respectively input into the VC layer for compression processing to obtain more dense information. The VC layer can effectively reduce the spatial dimension of the features while maintaining the expression ability of the features. Then, the compressed features are sequentially aggregated through the MLP layer, the NetVLAD, and the MLP layer to generate two sub-descriptors. Finally, the sub-descriptors from the two encoding branches are connected through the descriptor aggregation module to generate a multi-modal global descriptor. The multi-modal global descriptor is then used for database creation and fast location retrieval. By comparing the multi-modal global descriptor with the data in the database and querying the features, the most similar descriptor is found to achieve location recognition and achieve positioning function.

[0034] The above technical solutions are described in detail as follows:

[0035] S1, first, point cloud data is obtained by LIDAR radar, and multi-view images are obtained by cameras with different viewing angles, wherein the multi-view images are RGB images; then, the point cloud data is preprocessed to generate a distance image. Specifically, a distance image is generated using spherical projection on the point cloud data obtained by the LIDAR laser radar. Each three-dimensional point P = (x, y, z) in the point cloud data is converted into a corresponding image two-dimensional R = (u, v) through formula (1), wherein x, y, z represent the numerical values of different dimensions under the coordinates, and u, v represent the numerical values of the horizontal and vertical coordinates of the corresponding pixel of the point cloud in the distance image.

[0036]

[0037] wherein r is the distance, f = f up +f down is the vertical field of view of the sensor, f up represents the maximum field of view, f down represents the minimum field of view, and w, h are the width and height of the distance image R. The point cloud binary file of the data set is converted into a depth image, i.e., a distance image. The size of the distance image is 1 x 32 x 1056, 1 is the depth of the distance image, 32 is the height of the distance image, and 1056 is the width of the distance image.

[0038] S2, the multi-view images and the distance images are respectively sent into the camera and the laser radar encoding branch, wherein the size of the input RGB image is 3 x 704 x 256.

[0039] S3 uses the InputConv input convolutional layer extraction block to extract intermediate features from the multi-view image and range image, obtaining multi-view image features and range image features. These features are then fed into the MambaFusion module, which reshapes and fuses the multi-view image and LIDAR range image features. MambaFusion then segments and reshapes the fused intermediate features to meet the corresponding branch size and returns them to the original branch. This process is repeated four times.

[0040] Among them, such as Figure 2 As shown, in the MambaFusion module, the multi-view image features and the range image features are first reshaped and spliced, and the spliced ​​features are input into the Mamba module. The spliced ​​features are processed by the Mamba module, and the processed features are fused and split and reshaped to the original size to obtain multi-view image split features and lidar split features, and the multi-view image split features and the multi-view image features are spliced, and the range image split features and the range image features are spliced ​​to obtain the coding features of each branch, including image coding features and lidar coding features. The coding features of each branch are processed by the convolution layer respectively and input into the MambaFusion module again. The above process is repeated until the maximum number of processing times of the MambaFusion module is reached, and the output features of the last Mamba Fusion module are output to obtain image branch features and lidar branch features.

[0041] The Mamba module includes a first linear layer and a second linear layer for linearly processing the features after splicing the image coding features and the lidar coding features, respectively. The output of the first linear layer is directly used as the A matrix of the SSM. The X matrix, B matrix and C matrix of the SSM are calculated by using different activation functions on the output of the first linear layer after the convolution layer processing. The output of the SMM is spliced ​​with the output of the second linear layer after the convolution layer processing through the weighted union and the activation function and expanded by the state expansion factor. The normalized result is processed by the third linear layer and normalized again as the final output.

[0042] In the Mamba module, the data operation process of the SSM model is as follows: first, the data is rearranged, and the input sequence x(t) and matrices A, B, and C are divided into several blocks according to chunk_size;

[0043] In different blocks, the block calculation is then performed to calculate the output Y_diag in each block, which is obtained by matrix multiplication of C, B, L (the cumulative effect of the state transfer matrix calculated by the segsum function) and the input x, that is, Y_diag = C*B*L*x. The final state states in each block is calculated by matrix multiplication of B, decay_states (calculating the decay of the state) and x, that is,

[0044] states = B*decay_states*x, representing the state within the block;

[0045] After that, the block loop is performed, and the state is initialized using initial_states, or the default zero vector is used. Decay_chunk calculates the attenuation between blocks, and new_states is obtained by matrix multiplication of decay_chunk and states, that is, new_states = decay_chunk * states, which represents the updated state; then the state output conversion is performed, state_decay_out calculates the impact of state decay on the output, and Y_off is obtained by matrix multiplication of C, states and state_decay_out, that is, Y_off = C * states * state_decay_out, which represents the contribution of inter-block information transfer to the output; finally, the output is merged, and the intra-block output Y_diag and the inter-block output Y_off are added to obtain the final output Y.

[0046] In the MambaFusion fusion module proposed this time, the Mamab model can replace the SSM model by introducing the structured state space sequence model (S4), preprocessing the input data with the input selection mechanism, and setting the model's operating environment with the hardware-aware parallel algorithm. It also introduces state space duality (SSD) and integrates multi-head attention (MHA) into the SSM model or S4 model to optimize the framework. When data enters the Mamab model, it first undergoes data preprocessing. Multi-view image features and LiDAR range image features are normalized and scaled, mapped to the same dimension to ensure data range consistency. Under the input selection mechanism, these features are filtered using linear transformations, attention mechanisms, gating units, or state-dependent functions to select distinct features. This filtering method can be random, fixed, or dynamic. Finally, the task-specific filtering result s[k] is generated. The HiPPO algorithm implicitly updates the state, constructs local features, and outputs them. Under the multi-head attention mechanism (MHA), attention weights are calculated using the Q, K, V mechanism to enhance fine-grained global features. The enhanced local features are then output, and the two features are then fused and split. State-space duality (SSD) is primarily responsible for controlling feature evolution and updating, accelerating filtering and processing efficiency. It is the underlying implementation logic of the Mamba module.

[0047] S4 is an extension of the SSM (state-space model). S4 is a structured state-space sequence model. Inspired by specific continuous systems, it maps one-dimensional sequences through implicit potential states, which is equivalent to simply writing the SSM in the form of matrix multiplication. The S4 model constructs the matrix A by introducing HiPPO (High-order Polynomial Projection Operators). This matrix A can capture the most recent tokens and decay old tokens, thereby better memorizing historical information.

[0048] In the Mamba module, SSM (state space model) and structured state space model (S4) are inspired by continuous systems and mathematically represented by linear differential equations:

[0049]

[0050] A represents the state transfer matrix; B represents the input matrix; C represents the output matrix; D represents the direct transfer matrix; x(t), i.e., X in the attached figure, represents the input sequence; y(t) represents the output sequence; h(t) represents the hidden state, and h'(t) represents the derivative of the hidden state h(t) with respect to time t. and If the system is affected by the past state and introduces it, A i (t) is the state matrix associated with the time delay, i is the time step; τ is the unit time delay; C i (t) Output matrix associated with time delay, N is the hidden state size, and M is the output sequence size.

[0051] Since continuous-time models, SSM and S4 face great challenges in integrating into deep learning algorithms, they are discretized using zero-order hold:

[0052]

[0053] y t =h t

[0054] By simplifying the structures of A, B, and C, for example, the structure of matrix A is further simplified from the diagonal to a scalar multiplied by the unit matrix structure, It can be represented by a scalar, where the superscript - indicates the scalar representation of the matrix, state space duality SSD, and improves the original Mamba performance.

[0055] S4 sends the final image branch features and lidar branch features to the corresponding VC layer respectively, and compresses the image branch features and lidar branch features through the VC layer to obtain denser image branch panoramic features and lidar branch panoramic features.

[0056] S5 inputs the panoramic features processed by the VC layer into the perception module respectively, where the perception module includes the connected MLP layer, NetVLAD, and MLP layer in sequence, and generates image branch sub-descriptors and lidar sub-descriptors through the perception module.

[0057] S6 concatenates the image branch sub-descriptors and the lidar sub-descriptors through a descriptor aggregation module to generate a global multimodal descriptor for use as a query or reference in the database.

[0058] For the above-mentioned camera and lidar encoding branches, under different branches, the input convolution layer is improved through different adaptive structures as an improvement, so as to adaptively extract features from the initial data of the above-mentioned camera and lidar through different structures, provide relevant valid data for the subsequent fusion model, and ensure the accuracy of subsequent fusion and data recognition. Among them, the input convolution layer under the camera encoding branch can choose to use the YOLOv8n backbone network, and extract the structure of the image through a structure with better feature extraction effect, so as to realize the extraction of each tiny feature of the camera's multi-view image.

[0059] At the same time, the output end of the YOLOv8n backbone network is further optimized using the MCS module based on multi-channel attention and multi-dimensional weighting to extract features, so as to ensure that the features in the multi-view images are clearer.

[0060] Specifically, the MCS module combines the MEW multi-axis external weight module with the CBAM attention. The MCS module effectively extracts global and local information through multi-dimensional weighting, thereby better capturing the subtle features of different targets in multi-view images. The CBAM mechanism makes precise adjustments and optimizations on this basis, enabling the model to focus more on features that reflect the location, significantly reducing background and image object noise interference, and improving detection accuracy. After inserting the MCS module into the YOLOv8n backbone network, that is, setting the MCS module at the output end of the backbone network, and using the features processed by the MCS module as the final output, the algorithm can better focus on the key features that reflect the location. The formula for the specific process of MEW is as follows:

[0061] x1,x2,x3,x4=Split(X)

[0062] x i(I,J) =W (I,J) ⊙F (I,J) [x i ]

[0063]

[0064] x4'=DW(x4)

[0065] Y=Concat(x1',x2',x3',x4')+X

[0066] Where X represents the output feature of the corresponding backbone network, x represents the feature after the output feature is split, split represents the separation function, i=1, 2, 3 represent the first three branches of the MEW multi-axis external weight module, i=4 represents the last branch of the MEW multi-axis external weight module, and ' plus x represents the corresponding output. (I,J) and F (I,J) where (i, j) denotes the learnable external weights and the 2D discrete Fourier transform of the corresponding axis, respectively. ⊙ denotes element-wise multiplication. When i = 1, (i, j) denotes the Height-Width axis; when i = 2, (i, j) denotes the Channel-Width axis; and when i = 3, (i, j) denotes the Channel-Height axis. Denotes a two-dimensional inverse discrete Fourier transform. DW denotes a depthwise separable convolution. Y denotes the output features of MEW. The output features of MEW are input into the CBAM attention for correlation processing to obtain the initial features of the camera encoding branch for subsequent correlation fusion processing.

[0067] The input convolution layer under the lidar encoding branch can be optionally followed by a spatial attention mechanism to extract relevant features. The spatial attention mechanism weights the spatial positions containing the main location features in the range image, thereby highlighting the areas that contribute most to the task and suppressing irrelevant or redundant areas to ensure the effectiveness of feature extraction from the range image.

[0068] The spatial attention calculation formula is as follows:

[0069]

[0070] Where F represents the feature after convolution of the range image, σ represents the activation function, f represents the convolution process, AvgPool represents average pooling, MaxPool represents maximum pooling, Ms(F) represents the spatial attention feature, and the subscripts avg and max of the F feature represent the corresponding average pooling and maximum pooling processes. The features after spatial attention processing are used as the initial features of the lidar score for subsequent related fusion processing.

[0071] The specific contents introduced or improved for the above-mentioned MambaFusion fusion module are as follows:

[0072] In this paper, the fusion module adopts the Mamba mechanism. For this Mamba mechanism, a structured state-space sequence model (S4) can be introduced. This model modifies the A matrix in the SSM and transforms it into a low-rank modified conditional matrix. An input selection mechanism and a hardware-aware parallel algorithm are introduced to improve the performance of S4 and significantly increase computational efficiency. State-space duality (SSD) is introduced to enhance the original Mamba, and a multi-head attention (MHA) design is integrated into the SSM to optimize the framework, thereby promoting the fusion of sensor data.

[0073] S4 is an extension of SSM. S4 is a structured state space sequence model. It is inspired by specific continuous systems and maps one-dimensional sequences through implicit potential states, which is equivalent to simply writing SSM in the form of matrix multiplication. The S4 model constructs the matrix A by introducing HiPPO (High-order Polynomial Projection Operators). This matrix A can capture the most recent tokens and decay old tokens, thereby better memorizing historical information.

[0074] The HiPPO matrix A is derived by projecting the past input sequence and constructed based on the Legendre polynomial orthogonal basis.

[0075]

[0076] The input selection mechanism allows the model to selectively propagate or forget information based on the current token. This is achieved by making the parameters of the SSM (State Space Model) a function of the input, thus addressing the SSM's weakness in processing discrete modal data. The key components of the mechanism are projection and selection. The projection component is responsible for mapping the input sequence of the SSM model into a high-dimensional space suitable for state-space model processing for preprocessing, ensuring that the input data can effectively interact with the state-space model. Projection can be a linear or nonlinear transformation, typically performed by a neural network layer. Its goal is to transform the input data into the specific representation required by the model so that subsequent state updates and output generation steps can proceed effectively. The selection mechanism dynamically adjusts the parameters of the state-space model based on the current input sequence, primarily the input matrix B, which influences state transitions, and the output matrix C, which influences outputs. Dynamic parameter adjustment means that the selection mechanism allows the model to selectively propagate or forget information based on the characteristics of the specific input. For example, when processing different types of input data, the model can dynamically adjust B and C to optimize state updates and output generation. The selection mechanism can be based on attention mechanisms, gating mechanisms, or other dynamic parameter adjustment methods. In selective state-space models, this flexibility is particularly important because it enables the model to adapt to a variety of complex input data patterns.

[0077] The hardware-aware parallel algorithm is a hardware-aware parallel recursive pattern algorithm that takes advantage of the parallelism of modern accelerators (such as GPUs and TPUs) to run the above model and perform selective SSM calculations in a memory-efficient manner. In other words, the hardware-aware algorithm uses a parallel association scan (Parallel Scan Algorithm), an algorithm that uses linear association calculation properties to achieve efficient calculations by constructing a balanced binary tree and sweeping from leaf to root.

[0078] The SSD algorithm, based on a sequence processing method based on a state-space model, decomposes a sequence into several blocks and efficiently transfers information within and between blocks. SSD leverages the low-rank decomposition and exponential decay properties of matrices to transform complex sequence modeling problems into a series of efficient matrix multiplication operations, significantly reducing computational complexity.

[0079] The MHA mechanism is set at the output end of the SSM model. The input of MHA includes three vectors: query vector, key vector and value vector. For a given query vector, MHA performs weighted summation on the key vector, and the weight is calculated by the similarity between the query vector and the key vector, and then the weighted sum is multiplied by the value vector to output. The multi-head mechanism of MHA can effectively improve the expression ability of the model, and also enable the model to learn more diverse and complex features. Under the multi-head mechanism, the input sequence data is divided into multiple heads, each head performs independent calculation to obtain different outputs. These outputs are finally spliced together to form the final output.

[0080] The training module uses the training set to train the laser radar point cloud data and multi-view camera image data, and trains the deep neural network model, aiming to improve the accuracy and robustness of place recognition in low-texture and appearance-similar environments. During training, data preprocessing is performed, including resizing the input camera image data to a uniform size and applying standard normalization processing. Specifically, the color channels of the image are normalized, with the mean set to [0.485, 0.456, 0.406] and the standard deviation set to [0.229, 0.224, 0.225]. For LiDAR point cloud data, appropriate format conversion and normalization processing are applied to ensure that the point cloud data can be effectively fused with image data in subsequent networks. Data augmentation operations are applied to image data, including random cropping, rotation, flipping, etc., to improve the generalization ability of the model and avoid overfitting.

[0081] The training dataset contains three types of data: training query data, database query data and validation query data. The dataset is trained using a triplet loss function to ensure that the model can learn to distinguish the distance relationship between positive and negative samples. The training set uses TripletDataset, where each triplet consists of a query image, a positive sample image and a negative sample image. The training set is constructed according to the distance threshold (nonTrivPosDistThres, posDistThr) and the number of negative samples (nNeg). The full dataset includes the entire database dataset (DatabaseQueryDataset) for model evaluation, ensuring that the model can effectively perform place recognition on all database samples. Data loading is batched using PyTorch's DataLoader and multi-processing (num_workers) is used to accelerate the data loading process.

[0082] The present invention adopts deep convolutional neural network (CNN) and a customized multimodal fusion module (MambaFusion) to perform place recognition tasks by fusing the features of image and point cloud data.

[0083] During the training process, triplet loss is used to optimize the performance of the model. The triplet loss function minimizes the distance between the positive sample and the query sample, while maximizing the distance between the negative sample and the query sample, thereby achieving effective distinction in the place recognition task. The triplet loss L formula is:

[0084] L=max(d(query,positive)-d(query,negative)+margin,0)

[0085] Where d(query, positive) and d(query, negative) represent the Euclidean distances between the query and the positive and negative samples, respectively. margin is a hyperparameter that controls the distance difference between positive and negative samples. Positive samples are defined as those located within 9 meters of the query capture location. Negative samples are defined as those located at least 18 meters from the query capture location.

[0086] During training, the optimizer uses the Adam optimizer for model training, the decay factor of the learning rate scheduler is 0.8, and the adjustment interval is 5.

[0087] The training process consists of an initialization phase, a training phase, and a validation phase. The initialization phase initializes model parameters and loads pre-trained weights as the initial parameters of the image feature extraction network. During each training cycle, the model is fed with triplets from the training set for forward propagation, loss is calculated, and the model parameters are updated through backpropagation. The model is optimized using the Adam optimizer and a learning rate decay strategy. The validation phase evaluates the model using data from the entire database and calculates recognition accuracy and recall. The training strategy is dynamically adjusted based on the performance of the validation set.

[0088] The above are merely preferred embodiments of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A multimodal descriptor location recognition method based on the Mamba mechanism, characterized in that: include: Acquire multi-view images and range images; Feature extraction is performed on the multi-view image and the range image through several levels of fusion modules to obtain image branch features and lidar branch features; wherein the fusion module includes a Mamba Fusion module. In the fusion module, the input end of the Mamba Fusion module is connected to two convolution layers for processing the input features and then inputting them into the Mamba Fusion module. The Mamba Fusion module processes the input features after the convolution processing to obtain multi-view image splitting features and lidar splitting features. The input features and the multi-view image splitting features and the lidar splitting features are correspondingly spliced ​​to obtain the output features of the current level fusion module as the input features of the next level fusion module, wherein the input features of the first level fusion module are the multi-view image and the range image, and the output of the last level fusion module is the image branch features and the lidar branch features. The image branch features and the lidar branch features are compressed respectively to obtain the image branch panoramic features and the lidar branch panoramic features; The image branch panoramic features and the lidar branch panoramic features are sensed, extracted, and fused to obtain a global multimodal descriptor of the location. The location recognition result is obtained by searching based on the global multimodal descriptor of the location. In the Mamba Fusion module, the input features are reshaped and spliced, and the spliced ​​features are input into the Mamba module. The spliced ​​features are processed by the Mamba module, and the features processed by the Mamba module are fused and split and reshaped so that the split and reshaped features have the same size as the input features. Output features are obtained, where the output features include image coding features and lidar coding features. The image coding features and lidar coding features are used as input features of the next-level fusion module. In the Mamba module, the Mamba module includes a first linear layer and a second linear layer connected in parallel, wherein the first linear layer and the second linear layer are used to perform linear processing on the spliced ​​features, and the output of the first linear layer is directly used as the A matrix of the SSM. The X matrix, B matrix, and C matrix of the SSM are obtained by calculating the output of the first linear layer after the convolution layer processing through different activation functions. The output of the SSM is spliced ​​and normalized by weighted union and the calculation result of the output of the second linear layer after the convolution layer processing through the activation function. The normalized result is processed by the third linear layer and normalized again to obtain the features processed by the Mamba module.

2. The method according to claim 1, characterized in that The process of acquiring the range image includes: Point cloud data is acquired, and spherical projection is performed on the point cloud data to obtain a range image.

3. The method according to claim 1, characterized in that The image branch features and lidar branch features are compressed separately through the VC layer.

4. The method according to claim 1, wherein The image branch panoramic features and the lidar branch panoramic features are perceived and extracted through the perception module, where the perception module includes an MLP layer, a NetVLAD and an MLP layer connected in sequence.

5. The method according to claim 1, wherein The image branch panoramic features and the lidar branch panoramic features are perceived and extracted to obtain the image branch sub-descriptors and the lidar sub-descriptors. The image branch sub-descriptors and the lidar sub-descriptors are fused through splicing operations to obtain a global multimodal descriptor of the location.

6. The method according to claim 1, characterized in that The process of obtaining location recognition results includes: The global multimodal descriptor of the location is compared and queried with the data stored in the database to obtain the descriptor that is most similar to the global multimodal descriptor of the location, and the location data corresponding to the most similar descriptor is used as the location recognition result.

Images