Robust position recognition method for unmanned vehicles based on surround view images

Abstract

The present invention belongs to the technical field of vehicle positioning, and specifically discloses a method for robust position recognition of an unmanned vehicle based on surround view images, in which a position recognition model based on an improved Mamba module is proposed. In terms of model input, a 360° surround view image is introduced; in terms of model structure, the surround view image feature encoding module in the model adopts an encoder branch with multi-channel image weight sharing to capture multi-level and multi-channel fine-grained features, and each encoder adopts a DenseNet network to process image features, and the association between multi-view images is captured through the improved Mamba module, and a final one-dimensional global descriptor for position recognition is generated. The method of the present invention is conducive to achieving comprehensive coverage of the target area, capturing spatial information without omission, so as to overcome the problem of viewpoint difference in visual recognition, thereby improving the robustness and accuracy of position recognition.

Description

Technical Field

[0001] The invention belongs to the field of vehicle positioning and relates to a robust position recognition method for an unmanned vehicle based on surround view images. Background Art

[0002] In recent years, Simultaneous Localization and Mapping (SLAM) technology has become a key technology in the field of navigation. SLAM technology can provide accurate positioning services in real time in unknown complex environments and create consistent environmental maps without relying on external signals. In environments where GNSS signals cannot be obtained, SLAM can still achieve high-precision and high-continuity positioning, thus providing a reliable alternative for vehicle positioning.

[0003] In addition, SLAM has also demonstrated extensive and important application value in the fields of autonomous driving, digital twins, augmented reality, search and rescue operations, and industrial inspection robots. Visual recognition plays an irreplaceable role in SLAM. Its main contribution is to enhance environmental understanding, achieve real-time data processing, and adapt to changing external environments. These factors together shape the core position of visual recognition in SLAM systems. Nevertheless, the challenges brought by viewpoint changes cannot be ignored, because the parallax caused by changes in the orientation and position of objects in the image will affect the accuracy of visual recognition, thereby affecting the accuracy of positioning. Summary of the invention

[0004] The purpose of the present invention is to propose a robust position recognition method for an unmanned vehicle based on surround view images. By combining 360° surround view image acquisition with an improved Mamba module, it is beneficial to achieve comprehensive coverage of the target area and capture spatial information without omission, so as to overcome the viewpoint difference problem in visual recognition, thereby improving the robustness and accuracy of position recognition.

[0005] In order to achieve the above object, the present invention adopts the following technical scheme:

[0006] The method for robust position recognition of an unmanned vehicle based on surround view images comprises the following steps:

[0007] Step 1. First, obtain multiple sets of 360-degree surround images and build a dataset;

[0008] The 360-degree surround view image is acquired by M cameras mounted on the vehicle, where M is a natural number;

[0009] Step 2. Build a robust location recognition model based on surround view images, which includes a surround view image feature encoding module, a vertical compression module, an improved Mamba module, a GeM module, and an ECA module;

[0010] The surround view image feature encoding module includes encoder branches with multiple image weights shared, which are used to capture multi-level, multi-channel fine-grained features; the number of encoder branches is equal to the number of cameras;

[0011] The input of the surround view image feature encoding module is the 360-degree surround view image collected by M cameras as input data, and each encoder branch extracts the visual image features of the image data collected by one camera;

[0012] The image encoder in each encoder branch uses a DenseNet network to process image features;

[0013] The output features of each encoder branch in the surround image feature encoding module are spliced ​​along the width dimension, and the spliced ​​feature quantities are input into the vertical compression module for feature compression operation to obtain compressed features;

[0014] Then, the compressed features are input into the improved Mamba module, wherein the improved Mamba module extracts global information from the convolutional coded surround image features and reduces channel redundancy and complexity;

[0015] The output results of the improved Mamba module are then sent to the GeM pooling module and the ECA module in sequence to capture the association between multi-view images and generate the final one-dimensional global descriptor for location recognition;

[0016] Step 3. Train the robust location recognition model built in step 2 based on the data set constructed in step 1, and further use the trained model to achieve perspective-invariant robust location recognition based on surround images.

[0017] The present invention has the following advantages:

[0018] As described above, the present invention proposes a method for robust position recognition of an unmanned vehicle based on surround view images, in which a robust position recognition model suitable for surround view images is constructed. Among them, in terms of model input, the present invention introduces 360° surround view images so that the robust position recognition model can fully cover the target area, effectively capture spatial information, and overcome the recognition challenges brought by perspective differences. In terms of model structure, the model includes a surround view image feature encoding module, a vertical compression module, an improved Mamba module, a GeM module, and an ECA module. The surround view image feature encoding module adopts an encoder branch with multi-channel image weight sharing, which can capture multi-level and multi-channel fine-grained features. Each encoder branch uses a DenseNet network to process image features to enhance information transfer, reduce redundancy, and improve the expressiveness of features, thereby better reflecting scene details. The improved Mamba module in the present invention combines a spatial attention mechanism, effectively compensates for perspective changes by introducing key structural design points, ensures accurate attention to the target, and enhances the spatial information capture capability of the model. Through the organic combination of model input and network structure, the present invention achieves comprehensive coverage of the target area, ensures the capture of spatial information without omission, significantly overcomes the problem of perspective change in visual recognition, and improves the robustness and accuracy of vehicle position recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is an overall flow chart of a method for robust position recognition of an unmanned vehicle based on surround view images in an embodiment of the present invention;

[0020] Figure 2 is a network structure diagram of a robust location recognition model based on surround view images in an embodiment of the present invention;

[0021] Figure 3 is a structural diagram of a vertical compression module in an embodiment of the present invention;

[0022] Figure 4 It is a structural diagram of an improved Mamba module in an embodiment of the present invention;

[0023] Figure 5 4 is a structural diagram of the look-around state model in an embodiment of the present invention. DETAILED DESCRIPTION

[0024] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0025] Example 1

[0026] The present embodiment describes a method for robust position recognition of an unmanned vehicle based on surround view images, in which a robust position recognition model based on surround view images is proposed, which mainly includes a surround view image feature encoding module, a vertical compression module, and an improved Mamba module, etc. The method of the present invention encodes the 360° surround view image by the surround view image feature encoding module, and then removes the redundant information of the height dimension by the vertical compression module to improve the efficiency of network processing, and simultaneously applies the improved Mamba module to extract the global information receptive field, thereby improving the accuracy of robust position recognition. The method of the present invention is conducive to achieving comprehensive coverage of the target area, capturing spatial information without omission, and is conducive to overcoming the viewpoint difference problem in visual recognition, which is of significant significance for improving the robustness and accuracy of vehicle position recognition in the SLAM system.

[0027] like Figure 1 As shown, the method for robust position recognition of an unmanned vehicle based on surround view images comprises the following steps:

[0028] Step 1. First, obtain multiple sets of 360-degree surround images and build a dataset;

[0029] The 360-degree surround view image is acquired by M cameras mounted on the vehicle, where M is a natural number.

[0030] In this embodiment, the number of cameras M mounted on the vehicle is, for example, 6, and the 6 cameras are respectively used to capture images of the front, rear, left, right, left front, and right rear of the vehicle.

[0031] The images taken by 6 cameras are combined to form a 360-degree surround view image of the vehicle.

[0032] In this way, N groups of 360-degree surround images can be collected, and the size of each image is 704×256.

[0033] These N sets of data are used as the data set for the following model training. The data set formed in step 1 is divided into a training set, a validation set, and a test set according to a certain ratio (for example, 8:1:1), which are used for model training, validation, and testing, respectively.

[0034] Step 2. Build a robust location recognition model based on surround view images.

[0035] like Figure 2 As shown, the robust location recognition model based on surround view images includes a surround view image feature encoding module, a vertical compression module, an improved Mamba module, a GeM module and an ECA module.

[0036] The surround view image feature encoding module includes encoder branches with multi-image weight sharing, which is used to capture multi-level and multi-channel fine-grained features; the number of encoder branches is equal to the number of cameras.

[0037] The input of the surround view image feature encoding module is the 360-degree surround view image collected by M cameras as input data, and each encoder branch extracts the visual image features of the image data collected by one camera.

[0038] Each encoder branch includes a convolution layer, a maximum pooling layer, and an image encoder. The image encoder on each encoder branch uses a DenseNet network to process image features.

[0039] Use multiple image inputs as the data source, and the input data is recorded as M×3×h1×w1.

[0040] The advantage of the DenseNet network in image feature processing is that it uses feature maps from all previous layers as input in each layer through dense connections, achieving efficient reuse of features. Compared with the residual connection of the traditional ResNet, the DenseNet network has significant advantages in computational efficiency, feature transfer and gradient propagation.

[0041] The output features of each encoder branch in the surround view image feature encoding module are spliced ​​along the width dimension, and the spliced ​​feature quantities are input into the vertical compression module for feature compression operation to obtain compressed features.

[0042] Through the vertical compression module, redundant information in the height dimension can be removed, thereby improving the efficiency of network processing.

[0043] Then, the compressed features are input into the improved Mamba module, wherein the improved Mamba module extracts global information from the convolutionally encoded surround view image features and reduces channel redundancy and complexity.

[0044] The output results of the improved Mamba module are then sent to the GeM pooling module and the ECA module in sequence to capture the association between multi-view images and generate the final one-dimensional global descriptor for location recognition.

[0045] Through this one-dimensional global descriptor, location recognition can be achieved by similarity matching with descriptors of known locations in a database (e.g., the nuScence database, etc.), and by calculating the distance between feature vectors (e.g., Euclidean distance or cosine similarity), the closest match can be identified to determine the current location of the vehicle, thereby achieving accurate location recognition.

[0046] The present invention can extract high-quality and high-efficiency global information from the surround view image features after convolution coding and reduce channel redundancy and complexity through the improved Mamba module, thereby improving the robustness and accuracy of position recognition.

[0047] The specific structure of each module in the robust location recognition model is described in detail below.

[0048] The basic unit of the DenseNet network is the Dense Block, each of which consists of several convolutional layers, and in each convolutional layer, all previous feature maps are used as inputs to the convolutional layer. The DenseNet network enhances the transfer of features through this connection method, making the network easier to train and effectively reducing the number of parameters.

[0049] The DenseNet network includes a first convolutional layer, a first pooling layer, a first Dense Block, a first transition layer, a second Dense Block, a second transition layer, a third Dense Block, a third transition layer, and an output layer.

[0050] The first Dense Block, the second Dense Block, and the third Dense Block are all composed of several convolutional layers, and in each layer of the corresponding Dense Block, all previous feature maps are used as inputs of the layer.

[0051] The first convolutional layer uses 64 3×3 convolution kernels to convolve with the visual image of input dimension 3×h1×w1, with a step size of 1 and a dilation rate of 1. After BN and ReLU activation, a feature map of dimension 64×h1×w1 is obtained.

[0052] Among them, the input dimension 3×h1×w1, 3, h1, and w1 represent the channel dimension, height, and width respectively.

[0053] The first pooling layer uses a 2×2 pooling kernel to perform maximum pooling on the feature map output by the first convolutional layer with a step size of 4, obtaining a feature map with a dimension of 64×h1 / 4×w1 / 4.

[0054] The first Dense Block is used to extract features from the feature map output by the first pooling layer through dense connections, and obtain a feature map with an output dimension of 64×h1 / 8×w1 / 8.

[0055] Specifically, the first Dense Block consists of three 3×3 convolutional layers, and through dense connections, each convolutional layer in the first Dense Block receives the feature maps of all previous convolutional layers as input.

[0056] The three convolutional layers in the first Dense Block are defined as convolutional layer 1-1, convolutional layer 1-2, and convolutional layer 1-3.

[0057] Convolutional layer 1-1 uses 64 3×3 convolution kernels to perform convolution with the feature map output by the first pooling layer, with a step size of 2 and a dilation rate of 1. After BN and ReLU activation, a feature map with a dimension of 64×h1 / 8×w1 / 8 is obtained.

[0058] The output of the convolutional layer 1-1 is used as the input of all subsequent convolutional layers in the first Dense Block.

[0059] Convolutional layer 1-2 uses 64 3×3 convolution kernels and the feature map output by convolutional layer 1-1 as the input of convolutional layer 1-2. After BN and ReLU activation, the output dimension is 64×h1 / 8×w1 / 8.

[0060] Convolutional layer 1-3 uses 64 3×3 convolution kernels and uses all feature maps of previous convolutional layers 1-1 and 1-2 as input of convolutional layer 1-3 to obtain a feature map with an output dimension of 64×h1 / 8×w1 / 8.

[0061] In order to reduce the size of the feature map and avoid excessive growth of the number of parameters, the DenseNet network introduces a transition layer between adjacent DenseBlocks and between the last Dense Block and the output layer.

[0062] The first transition layer uses a 1×1 convolution layer to reduce the dimension, followed by a 2×2 average pooling layer. The first transition layer first uses 128 1×1 convolution kernels to reduce the dimension of the feature map output by the first Dense Block, and the output dimension is 64×h1 / 8×w1 / 8; then uses a 2×2 average pooling kernel with a step size of 1, and the output dimension is 64×h1 / 8×w1 / 8.

[0063] The second Dense Block is used to extract features from the feature map output by the first transition layer through dense connection to obtain a feature map with an output dimension of 128×h1 / 16×w1 / 16.

[0064] The second Dense Block consists of three 3×3 convolutional layers, and through dense connections, each convolutional layer in the second Dense Block uses the features of all previous convolutional layers.

[0065] The three convolutional layers in the second Dense Block are defined as convolutional layer 2-1, convolutional layer 2-2, and convolutional layer 2-3.

[0066] Convolutional layer 2-1 uses 64 3×3 convolution kernels to perform convolution with the feature map output by the first transition layer, with a step size of 2 and a dilation rate of 1, and the output dimension is a feature map of 128×h1 / 16×w1 / 16.

[0067] The output of the convolution layer 2-1 is used as the input of all subsequent convolution layers in the second Dense Block.

[0068] Convolution layer 2-2 uses 64 3×3 convolution kernels and the feature map output by convolution layer 2-1 as the input of convolution layer 2-2. After BN and ReLU activation, a feature map with a dimension of 128×h1 / 16×w1 / 16 is obtained.

[0069] Convolution layer 2-3 uses 64 3×3 convolution kernels and uses all feature maps of previous convolution layer 2-1 and convolution layer 2-2 as the input of convolution layer 2-3. It continues to use dense connections and outputs feature maps with a dimension of 128×h1 / 16×w1 / 16.

[0070] The second transition layer further reduces the feature map size through 1×1 convolution and average pooling. The second transition layer first uses 128 1×1 convolution kernels to reduce the dimension of the feature map output by the second Dense Block, and the output dimension is 128×h1 / 16×w1 / 16; then uses a 2×2 average pooling kernel with a step size of 1, and the output dimension is 128×h1 / 16×w1 / 16.

[0071] The third Dense Block is used to extract features from the feature map output by the second transition layer through dense connections, and obtain a feature map with an output dimension of 256×h1 / 32×w1 / 32.

[0072] The third Dense Block is similar to the previous two Dense Blocks. The third Dense Block consists of two 3×3 convolutional layers and further extracts features through dense connections.

[0073] Define the two convolutional layers in the third Dense Block as convolutional layer 3-1 and convolutional layer 3-2 respectively;

[0074] Convolutional layer 3-1 uses 64 3×3 convolution kernels to convolve the feature map output by transition layer 2, with a step size of 2 and a dilation rate of 1, and the output dimension is a feature map of 256×h1 / 32×w1 / 32;

[0075] Convolutional layer 3-2 uses 64 3×3 convolution kernels to densely connect the feature map output by convolutional layer 3-1, and the output dimension is a feature map of 256×h1 / 32×w1 / 32.

[0076] The third transition layer further reduces the feature map size through 1×1 convolution and average pooling. The third transition layer first uses 256 1×1 convolution kernels to reduce the dimension of the feature map output by the third Dense Block, and the output dimension is 256×h1 / 32×w1 / 32; the pooling layer uses a 2×2 average pooling kernel with a step size of 1, and the output dimension is 256×h1 / 32×w1 / 32.

[0077] The features output by the third transition layer are output through the output layer of the DenseNet network.

[0078] In the vehicle position recognition task, different parts of the image (such as the sky, buildings or trees) may generate redundant information in the height dimension, which is not very relevant to the direct judgment of the vehicle's position and increases the computational cost and complexity.

[0079] The vertical compression module is introduced in the embodiment of the present invention to effectively compress the height dimension of the input image, remove redundant information, and retain the most useful features for vehicle position recognition. Through this compression operation, the model can focus more on the horizontal area where the vehicle is located, rather than processing background information unrelated to the vehicle.

[0080] The vertical compression module is used to perform vertical compression operations on the feature map output by the surround view image feature encoding module; for the visual feature map, the feature quantity dimension output by the surround view image feature encoding module is M×C×h×w.

[0081] The output features of each encoder branch in the surround image feature encoding module are spliced ​​along the width dimension, and the spliced ​​feature quantity with a dimension of C×h×Mw is input into the vertical compression module for feature compression operation to obtain the compressed feature. .

[0082] The vertical compression module includes six convolutional layers and one pooling layer. The six convolutional layers are defined as convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, convolutional layer 5 and convolutional layer 6, and the pooling layer is defined as pooling layer 1.

[0083] Convolutional layer 1 uses 256 The convolution kernel and input dimension are The visual image is convolved with a step size of 1 and a dilation rate of 1. After Sigmoid activation, the dimension is feature map.

[0084] Convolutional layer 2 uses 256 The convolution kernel and input dimensions are The visual image is convolved with a step size of 1 and a dilation rate of 1. After Sigmoid activation, the dimension is feature map.

[0085] Pooling layer 1 is the maximum pooling layer. Convolution layer 1 performs maximum pooling on the feature map output by convolution layer 2 along the height dimension, and the dimension is feature map.

[0086] Convolutional layer 3 uses 256 The convolution kernel is convolved with the feature map output by pooling layer 1, with a step size of 1 and a dilation rate of 1. After BN and ReLU activation, the dimension is feature map.

[0087] Convolutional layer 4 uses 256 The convolution kernel is convolved with the feature map output by convolution layer 3, with a step size of 1 and a dilation rate of 1. After BN and ReLU activation, the dimension is feature map.

[0088] Convolutional layer 5 uses 256 The convolution kernel is convolved with the feature map output by convolution layer 4, with a step size of 1 and a dilation rate of 1. After BN and ReLU activation, the dimension is feature map.

[0089] The output feature map of convolution layer 5 is residually connected to the input feature map of convolution layer 4.

[0090] Convolutional layer 6 uses 256 The convolution kernel is convolved with the residual connected feature map, with a step size of 1 and a dilation rate of 1. After BN and ReLU activation, the dimension is feature map.

[0091] The present invention removes redundant information in the height dimension through a vertical compression module, improves network processing efficiency and vehicle recognition accuracy, and enhances the generalization ability of the model.

[0092] Specifically, in terms of processing efficiency, the vertical compression module reduces the feature dimensions that need to be processed, thereby reducing the computational workload and storage requirements of the model, improving the inference speed and resource utilization efficiency. Especially when processing high-definition images, reducing unnecessary height information can significantly reduce computational overhead.

[0093] In terms of vehicle recognition accuracy, the vertical compression module removes height dimension information that is irrelevant to the vehicle's position, allowing the model to focus more on the effective area, reduce errors, and improve the accuracy of vehicle position recognition. The elimination of redundant information enables the model to more effectively utilize limited feature information and improve the reliability and accuracy of positioning.

[0094] In terms of generalization, for images in different scenes, such as highways, city streets, or parking lots, the redundant height information in the background varies greatly, but the area where the vehicle is located is relatively stable. The vertical compression module improves the generalization ability of the model in various scenes by uniformly processing these redundancies, making vehicle position recognition more robust.

[0095] In addition, in the task of vehicle position recognition, accurately locating the vehicle requires the model to have a wide receptive field, that is, it can capture global information (such as lanes, surrounding vehicles, buildings, etc.) and local details (such as wheels, lights, etc.).

[0096] The present invention uses an improved Mamba module to enable the model to expand the receptive field in a more efficient manner, thereby better extracting fine-grained features related to the vehicle position while retaining global information.

[0097] Improved Mamba module input is (w1 is 256) feature encoding, the output is The feature encoding is used to more conveniently enhance the contextual relationship of long sequence images.

[0098] like Figure 4 As shown in FIG. 1 , the improved Mamba module includes a LayerNorm layer, a surround state model PSM, a local convolution layer, a hybrid attention mechanism layer, and a convolution layer; the processing flow of the Mamba module is as follows:

[0099] For the surround image features input into the Mamba module, a LayerNorm linear layer is first used to ensure the stability of the data feature distribution, thereby preventing the data from falling into the saturation area of ​​the activation function and reducing the problem of gradient vanishing.

[0100] Layer Normalization normalizes the feature layer of each input sample so that its output has a mean of 0 and a variance of 1. Unlike BatchNorm, LayerNorm does not depend on the batch size and is suitable for sequence-sensitive network structures such as RNN. LayerNorm helps accelerate the convergence of the model and prevents the model from experiencing gradient vanishing or gradient exploding during training by normalizing the input data.

[0101] In addition, the present invention also designs a Panoramic State Modeling (PSM), and inputs the output features of the LayerNorm layer into the Panoramic State Modeling (PSM) to capture long-term spatial dependencies.

[0102] At the same time, a module with learnable parameters is used to adjust the contribution of jump connections, thereby controlling the impact of features at each layer on the final output; this parameter is automatically optimized during network training, allowing the weights of jump connections to change flexibly between different layers, thereby improving the expressiveness of the model and improving the gradient flow effect.

[0103] Specifically, the present invention uses a module with learnable parameters to connect the surround image features input into the Mamba module with the output features of the surround state model PSM, and the connected features are then input into a LayerNorm layer.

[0104] Set the learnable parameters to , which is used to adjust the contribution of the jump, is expressed as follows:

[0105] ;in, Represents the output of the PSM module, Represents the look-around state model, Represents the LayerNorm layer, represents the surround image features, .

[0106] After the input is flattened into a one-dimensional sequence by PSM, the originally adjacent elements are pulled apart, resulting in the loss of key pixel information. To solve this problem, the LayerNorm layer is used to normalize the output features of the PSM module, and then the local convolution layer is used to enhance the local features to restore the neighborhood similarity, reduce the negative impact of flattening, and ensure that important features are retained.

[0107] It should be noted that PSM introduces a large number of hidden states, resulting in significant channel redundancy. In addition, the process of dynamic target filtering will also lead to the loss of some spatial information. In order to solve these two problems, the present invention introduces a mixed attention mechanism, which combines the channel attention mechanism with the spatial attention mechanism.

[0108] The output features of the local convolutional layer enter the hybrid attention mechanism layer for feature weight adjustment, spatial and channel information integration, feature enhancement and information aggregation, and the output features of the hybrid attention mechanism layer are further processed by a convolutional layer.

[0109] Feature weight adjustment means that the hybrid attention mechanism can adaptively assign feature weights to different spatial positions or channels, so that the model pays more attention to areas with important information, thereby suppressing irrelevant or redundant information.

[0110] The integration of spatial and channel information means that by simultaneously utilizing spatial attention and channel attention, the hybrid attention mechanism can more comprehensively integrate the spatial and semantic information in the features, making the model more robust in complex scenarios.

[0111] Feature enhancement means that in the hybrid attention mechanism layer, important features are further enhanced and unimportant features are weakened, thereby generating more representative feature outputs to improve the performance of subsequent tasks.

[0112] Information aggregation means that after being processed by the hybrid attention mechanism, the feature information is optimized and aggregated, which facilitates more effective classification, regression or other task processing in the next layer.

[0113] Through feature weight adjustment, spatial and channel information integration, feature enhancement and information aggregation processing, the hybrid attention mechanism can improve the expressiveness of the model and make the features extracted by the local convolutional layer more adaptable to the needs of specific tasks.

[0114] The result of the convolutional layer processing is residually connected with the surround image features input to the Mamba module to output the final features.

[0115] The final features defined for the improved Mamba module output are , and its generation process is as follows:

[0116] ;

[0117] In the formula, represents the convolutional layer, represents the hybrid attention mechanism, Represents the LayerNorm layer.

[0118] like Figure 5 As shown, the PSM includes two parallel branches, which are defined as a first branch and a second branch respectively.

[0119] The features input to PSM go through two parallel branches.

[0120] In the first branch, the features of the input PSM are expanded to the number of channels through the Linear layer. times, and the GELU activation function is applied after the Linear layer; where is the expansion factor of the channel, Indicates the number of channels.

[0121] In the second branch, the features of the input PSM also enter the Linear layer to expand the number of channels to times, and then processed by the depthwise separable convolution layer DSConv and the GELU activation function. The features after GELU activation function are then converted into a one-dimensional sequence by scanning in four diagonal directions using the two-dimensional selective scanning module 2D-SSM. The S6 module is applied to effectively model the dependencies in the feature representation, and finally the sequences are merged and the two-dimensional features are reconstructed.

[0122] Finally, the output features of the first branch and the second branch are fused through the Hadamard product, and the fused features are converted back to the number of channels through a Linear layer. And output the result.

[0123] The GELU activation function is a nonlinear activation function based on Gaussian distribution, which can activate negative inputs more gently and provide more flexible nonlinear expression capabilities. Compared with the "hard" truncation of ReLU, GELU handles negative inputs more delicately, thereby improving gradient flow. Ultimately, this smooth activation enables PSM to have better generalization capabilities when dealing with dynamic scenes and continuous perspective changes. The S6 module (i.e., S6 blocks in the figure) is a module used to build a specific convolutional neural network (CNN) structure in deep learning. It is mainly used to achieve more efficient computing and information transfer in convolutional neural networks.

[0124] In order to better utilize 2D spatial information, the present invention introduces a 2D selective scanning module, namely 2D-SSM. The 2D image features are flattened into 1D sequences, where scanning is performed along four different directions: upper left to lower right, lower right to upper left, upper right to lower left, and lower left to upper right. Then, the long-range correlation of each sequence is captured according to the discrete state space equation. Finally, all sequences are merged using summation, and then a reshaping operation is performed to restore the 2D structure.

[0125] The 2D-SSM layer is mainly used to capture the spatial and semantic relationships of the input features. It can be some type of convolutional layer or spatial correlation modeling layer, which is responsible for learning the dependencies between local spatial information. Assuming that the 2D-SSM contains a convolution operation, the convolution scans the entire feature map through a fixed-size convolution kernel to learn the spatial features and semantic relationships of the local area.

[0126] Through this spatial modeling, the network is able to capture spatial correlations within local regions and generate feature maps that contain semantic information. Using 2D-SSM spatial modeling helps capture the dependencies of features in the spatial dimension, which is very important for understanding the object layout or scene context in an image, especially in tasks involving semantic segmentation or object detection.

[0127] The present invention not only improves the efficiency of feature extraction through the improved Mamba module, but also enhances the model's ability to understand the scene context, ensuring that no matter where the vehicle is in the image, the model can effectively capture its relative position information.

[0128] The present invention can efficiently extract the global receptive field of feature information through the improved Mamba module, thereby improving vehicle positioning accuracy, enhancing scene understanding ability and improving real-time performance.

[0129] Specifically, in terms of improving positioning accuracy, the expanded global receptive field enables the model to combine global and local information to more accurately determine the relative position of the vehicle in the scene.

[0130] This is particularly important for vehicle positioning in complex environments. For example, when a vehicle is partially occluded or in a dense environment with multiple vehicles, the improved Mamba module can infer the exact position of the occluded vehicle through global information.

[0131] In terms of enhancing scene understanding capabilities, the Mamba module uses efficient feature extraction to enable the network to not only identify the location of the vehicle, but also understand the layout of the entire scene, including the relative distance between vehicles, lane boundaries, etc. This global receptive field design helps the model correctly identify the location of vehicles in more complex urban roads or intersections.

[0132] In terms of improving real-time performance, the improved Mamba module reduces computational complexity without losing accuracy by optimizing feature extraction and global receptive field. Compared with traditional methods, it can handle vehicle position recognition tasks in complex scenes more quickly, thus providing real-time response capabilities for autonomous driving systems or intelligent transportation systems.

[0133] Input of GeM pooling module (w1 is 256) feature encoding, the output is 1×256, which is mainly used for feature extraction and can effectively capture important information in the image. In this embodiment, the visual feature map is passed through a GeM pooling module, and the pooling is learned in an end-to-end manner to obtain a one-dimensional global descriptor.

[0134] The GeM pooling module is explicitly mentioned in the following reference: Berman, Maxim et al. “MultiGrain: a unified image embedding for classes and instances.” ArXiv abs / 1902.05509 (2019): n. pag.

[0135] The output features of the GeM pooling module enter the ECA module, the input of the ECA module is 1×256, and the output is 1×256. The ECA module is designed to better capture the dependencies between features, thereby enhancing the network's expressive power.

[0136] Step 3. Train the robust location recognition model built in step 2 based on the data set constructed in step 1, and further use the trained model to achieve perspective-invariant robust location recognition based on surround images.

[0137] The present invention uses triplet loss Perform model training. For each training step, set each batch to contain: a query sample que, a positive sample pos, and batchsize-2 negative samples. .

[0138] Specifically, the triplet edge loss is expressed as follows:

[0139] ;in represents the batch size, represents the mapping of sample input to global descriptors, represents the Euclidean distance, Represents the threshold for avoiding negative losses.

[0140] NetVLAD trains the model by minimizing the triplet margin loss, which ensures that similar feature vectors are close to each other, while different feature vectors remain distant, thereby optimizing the feature clustering effect.

[0141] The training process ends when the model reaches the preset number of iterations or the convergence threshold of the loss function.

[0142] After the model training is completed, the vehicle position recognition is realized based on the model. The process is as follows:

[0143] First, a set of real-time 360-degree surround view images are collected using M cameras mounted on the vehicle and input into a trained robust location recognition model; the model outputs a one-dimensional global descriptor for location recognition, thereby achieving location recognition.

[0144] In addition, in order to verify the effectiveness of the method proposed in the present invention, the following experiments were carried out.

[0145] This experiment uses the nuScenes dataset, which is popular in the field of autonomous driving. It covers 1,000 driving scenarios of 20 seconds each and is equipped with a complete set of autonomous driving sensors, including six cameras, one lidar, and five radars.

[0146] The data in the nuScenes dataset were collected in four different areas in Boston and Singapore: Boston Seaport (BS), SG-One North (SON), SG-Queenstown (SQ), and SG-Holland Village (SHV). The BS subset is the largest, with a total of 550 scenes, while the others contain 215, 135, and 100 scenes respectively.

[0147] In order to train and evaluate the robustness of the location recognition model of the present invention, the BS dataset was used for training, and the generalization performance of the network was tested on the SON dataset. The data collected within 120 days on the BS dataset were divided into a training query set, a verification query set, a test query set, and a database set. Specifically, the data of the first 105 days were used as a training query set, and samples with an interval of 1 meter were selected to construct a database set; then the remaining 15 days of data were divided into a test query set and a verification query set at a ratio of 1:4; finally, those query samples for which there was no corresponding index in the database within a range of 5 meters were removed. After this series of processing, a database set containing 9686 samples, a training query set of 8407 samples, a verification query set of 1002 samples, and a test query set of 3008 samples were finally obtained on the BS dataset. On the SON dataset, there are a database set of 4796 samples and a test query set of 3308 samples, as shown in Table 1 below.

[0148] Table 1 Dataset division

[0149]

[0150] The present invention selects two representative visual position recognition algorithms, two laser radar position recognition algorithms and one millimeter wave radar position recognition algorithm as the comparison methods of the present invention for comparison.

[0151] The sizes of the global descriptors output by NetVLAD, Patch-NetVLAD, PointNetVLAD, OverlapTransformer, AutoPlace and the present invention are 9216×1, 256×1, 256×1, 256×1, 9216×1, and 256×1, respectively.

[0152] Among them, the sources of the five comparison methods are as follows:

[0153] R. Arandjelovic, P. Gronat, A. Torii, T. Pajdla and J. Sivic, "NetVLAD: CNN Architecture for Weakly Supervised Place Recognition," 2016 IEEEConference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV,USA, 2016, pp. 5297-5307, doi: 10.1109 / CVPR.2016.572。

[0154] S. Hausler, S. Garg, M. Xu, M. Milford and T. Fischer, "Patch-NetVLAD: Multi-Scale Fusion of Locally-Global Descriptors for PlaceRecognition," 2021 IEEE / CVF Conference on Computer Vision and PatternRecognition (CVPR), Nashville, TN, USA, 2021, pp. 14136-14147, doi: 10.1109 / CVPR46437.2021.01392。

[0155] M. A. Uy and G. H. Lee, “PointNetVLAD: Deep point cloud basedretrieval for large-scale PR,” in Proc. Conf. Comput. Vis. Pattern Recognit.,2018, pp. 4470–4479。

[0156] J. Ma, J. Zhang, J. Xu, R. Ai, W. Gu, and X. Chen, “OverlapTransformer: An efficient and yaw-angle-invariant transformer network for LiDAR-based place recognition,” IEEE Robot. Automat. Lett., vol. 7, no.3, pp. 6958–6965, Jul. 2022.

[0157] K. Cait, B. Wang, and CX Lu, “AutoPlace: Robust place recognition with single-chip automotive radar,” in Proc. IEEE Int. Conf. Robot. Automat., 2022, pp. 2222–2228.

[0158] The present invention uses recall rates Recall@1, Recall@5, and Recall@10 as evaluation indicators, as shown in Table 2.

[0159] Table 2 Comparison results of different methods

[0160]

[0161] It is not difficult to see from Table 2 above that the method of the present invention performs well in all evaluation indicators on the two data sets and achieves the best results. It can be seen that the method of the present invention achieves the best accuracy in the robust location recognition task, which is significantly better than all other compared methods, thereby verifying the effectiveness of the method of the present invention.

[0162] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A robust position recognition method for unmanned vehicles based on surround view images, characterized in that: The steps include: Step 1. First, obtain multiple sets of 360-degree surround images and build a dataset; The 360-degree surround view image is acquired by M cameras mounted on the vehicle, where M is a natural number; Step 2. Build a robust location recognition model based on surround view images, which includes a surround view image feature encoding module, a vertical compression module, an improved Mamba module, a GeM module, and an ECA module; The surround view image feature encoding module includes encoder branches with multiple image weights shared, which are used to capture multi-level, multi-channel fine-grained features; the number of encoder branches is equal to the number of cameras; The input of the surround view image feature encoding module is the 360-degree surround view image collected by M cameras as input data, and each encoder branch extracts the visual image features of the image data collected by one camera; The image encoder in each encoder branch uses a DenseNet network to process image features; The output features of each encoder branch in the surround image feature encoding module are spliced ​​along the width dimension, and the spliced ​​feature quantities are input into the vertical compression module for feature compression operation to obtain compressed features; Then, the compressed features are input into the improved Mamba module to extract global information from the convolutional coded surround image features and reduce channel redundancy and complexity; the improved Mamba module includes LayerNorm layer, surround state model PSM, local convolution layer, hybrid attention mechanism layer and convolution layer; For the surround image features input to the Mamba module, a LayerNorm layer is first used; the output features of the LayerNorm layer are input into the surround state model PSM to capture long-term spatial dependencies; At the same time, a module with learnable parameters is used to connect the surround image features input into the Mamba module with the output features of the surround state model PSM, and the connected features are then input into a LayerNorm layer; The concatenated features are normalized using the LayerNorm layer and then passed through the local convolution layer; The output features of the local convolutional layer enter the hybrid attention mechanism layer for feature weight adjustment, spatial and channel information integration, feature enhancement and information aggregation; the output features of the hybrid attention mechanism layer are further processed by a convolutional layer; The result of the convolutional layer processing is residually connected with the surround image features input to the Mamba module to output the final features; The output results of the improved Mamba module are then sent to the GeM pooling module and the ECA module in sequence to capture the association between multi-view images and generate the final one-dimensional global descriptor for location recognition; Step 3. Based on the data set constructed in step 1, the robust position recognition model built in step 2 is trained, and the trained model is further used to realize perspective-invariant robust position recognition based on surround view images.

2. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 1, characterized in that: The DenseNet network includes a first convolutional layer, a first pooling layer, a first Dense Block, a first transition layer, a second Dense Block, a second transition layer, a third Dense Block, a third transition layer and an output layer; The first convolutional layer uses 64 3×3 convolution kernels to convolve with the visual image of input dimension 3×h1×w1, with a step size of 1 and a dilation rate of 1. After BN and ReLU activation, a feature map of dimension 64×h1×w1 is obtained. Among them, 3, h1, and w1 in the input dimension 3×h1×w1 represent the channel dimension, height, and width respectively; The first pooling layer uses a 2×2 pooling kernel to perform maximum pooling on the feature map output by the first convolutional layer with a step size of 4, and obtains a feature map with an output dimension of 64×h1 / 4×w1 / 4; The first Dense Block is used to extract features from the feature map output by the first pooling layer through dense connections, and obtain a feature map with an output dimension of 64×h1 / 8×w1 / 8; The first transition layer uses a 1×1 convolution layer to reduce the dimension, followed by a 2×2 average pooling layer. The first transition layer first uses 128 1×1 convolution kernels to reduce the dimension of the feature map output by the first Dense Block, and the output dimension is 64×h1 / 8×w1 / 8; then uses a 2×2 average pooling kernel with a step size of 1, and the output dimension is 64×h1 / 8×w1 / 8; The second Dense Block is used to extract features from the feature map output by the first transition layer through dense connections, and obtain a feature map with an output dimension of 128×h1 / 16×w1 / 16; The second transition layer further reduces the feature map size through 1×1 convolution and average pooling. The second transition layer first uses 128 1×1 convolution kernels to reduce the dimension of the feature map output by the second Dense Block, and the output dimension is 128×h1 / 16×w1 / 16; then uses a 2×2 average pooling kernel with a step size of 1, and the output dimension is 128×h1 / 16×w1 / 16. The third Dense Block is used to extract features from the feature map output by the second transition layer through dense connections, and obtain a feature map with an output dimension of 256×h1 / 32×w1 / 32; The third transition layer further reduces the feature map size through 1×1 convolution and average pooling. The third transition layer first uses 256 1×1 convolution kernels to reduce the dimension of the feature map output by the third Dense Block, and the output dimension is 256×h1 / 32×w1 / 32; the pooling layer uses a 2×2 average pooling kernel with a step size of 1, and the output dimension is 256×h1 / 32×w1 / 32; The features output by the third transition layer are output through the output layer of the DenseNet network; The first Dense Block, the second Dense Block, and the third Dense Block are each composed of several convolutional layers, and in each convolutional layer of the corresponding Dense Block, all previous feature maps are used as inputs of the convolutional layer.

3. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 2 is characterized in that: The first Dense Block consists of three 3×3 convolutional layers; Define the three convolutional layers in the first Dense Block as convolutional layer 1-1, convolutional layer 1-2, and convolutional layer 1-3; Convolutional layer 1-1 uses 64 3×3 convolution kernels to perform convolution with the feature map output by the first pooling layer, with a step size of 2 and a dilation rate of 1. After BN and ReLU activation, a feature map with a dimension of 64×h1 / 8×w1 / 8 is obtained; The output of the convolutional layer 1-1 is used as the input of all subsequent convolutional layers in the first Dense Block; Convolutional layer 1-2 uses 64 3×3 convolution kernels and the feature map output by convolutional layer 1-1 as the input of convolutional layer 1-2. After BN and ReLU activation, the output dimension is 64×h1 / 8×w1 / 8. Convolutional layer 1-3 uses 64 3×3 convolution kernels and uses all feature maps of previous convolutional layers 1-1 and 1-2 as input of convolutional layer 1-3 to obtain a feature map with an output dimension of 64×h1 / 8×w1 / 8; The second Dense Block consists of three 3×3 convolutional layers; The three convolutional layers in the second Dense Block are defined as convolutional layer 2-1, convolutional layer 2-2, and convolutional layer 2-3; Convolutional layer 2-1 uses 64 3×3 convolution kernels to perform convolution with the feature map output by the first transition layer, with a step size of 2 and a dilation rate of 1, and the output dimension is a feature map of 128×h1 / 16×w1 / 16; The output of the convolutional layer 2-1 is used as the input of all subsequent convolutional layers in the second Dense Block; Convolutional layer 2-2 uses 64 3×3 convolution kernels and the feature map output by convolutional layer 2-1 as the input of convolutional layer 2-2. After BN and ReLU activation, a feature map with a dimension of 128×h1 / 16×w1 / 16 is obtained. Convolutional layer 2-3 uses 64 3×3 convolution kernels and uses all feature maps of previous convolutional layer 2-1 and convolutional layer 2-2 as the input of convolutional layer 2-3. It continues to use dense connections and outputs feature maps with dimensions of 128×h1 / 16×w1 / 16. The third Dense Block consists of two 3×3 convolutional layers; Define the two convolutional layers in the third Dense Block as convolutional layer 3-1 and convolutional layer 3-2 respectively; Convolutional layer 3-1 uses 64 3×3 convolution kernels to convolve the feature map output by transition layer 2, with a step size of 2 and a dilation rate of 1, and the output dimension is a feature map of 256×h1 / 32×w1 / 32; Convolutional layer 3-2 uses 64 3×3 convolution kernels to densely connect the feature map output by convolutional layer 3-1, and the output dimension is a feature map of 256×h1 / 32×w1 / 32.

4. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 1, characterized in that: Set the learnable parameter to be α∈R C , which is used to adjust the contribution of the jump, is expressed as follows: Among them, X l represents the output of the PSM module, DM represents the surround state model, LN represents the LayerNorm layer, represents the surround image features input to the Mamba module, 5. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 4, characterized in that: The final features defined for the improved Mamba module output are The generation process is as follows: In the formula, Conv represents the convolutional layer, MA represents the hybrid attention mechanism, and LN represents the LayerNorm layer.

6. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 1, characterized in that: The surround state model PSM includes two parallel branches, which are defined as a first branch and a second branch respectively; the features input to the PSM will pass through the two parallel branches; In the first branch, the features of the input PSM will be expanded to θC times through the Linear layer, and the GELU activation function will be applied after the Linear layer; where θ is the expansion coefficient of the channel and C represents the number of channels; In the second branch, the features of the input PSM also enter the Linear layer to expand the number of channels to θC times, and then are processed by the depthwise separable convolution layer DSConv and the GELU activation function. The features after the GELU activation function are then converted into a one-dimensional sequence by scanning in four diagonal directions using the two-dimensional selective scanning module 2D-SSM, and the S6 module is used to model the dependencies in the feature representation, and finally the sequences are merged and the two-dimensional features are reconstructed; Finally, the output features of the first branch and the second branch are fused through the Hadamard product, and the fused features are passed through a Linear layer to convert the number of channels back to the number of channels C and output the results.

7. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 1, characterized in that: In step 1, the number of cameras mounted on the vehicle is 6, and the 6 cameras are used to collect images of the front, rear, left, right, left front and right rear of the vehicle respectively; The images taken by 6 cameras are combined to form a 360-degree surround view image of the vehicle.

8. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 1, characterized in that: In step 3, during the training of the robust location recognition model: Use triplet loss L for training. For each training step, set each batch to contain: a query sample que, a positive sample pos, and batchsize-2 negative samples neg i ; The triplet edge loss is expressed as follows: batchsize represents the batch size, F() represents the mapping of sample input to the global descriptor, d(·,·) represents the Euclidean distance, and ω represents the threshold to avoid negative loss; The model is trained by minimizing the triplet edge loss, which ensures that similar feature vectors are close to each other, while different feature vectors remain distant, thereby optimizing the feature clustering effect; The training process ends when the model reaches the preset number of iterations or the convergence threshold of the loss function.

9. The method for robust position recognition of an unmanned vehicle based on surround view images according to claim 1, characterized in that: In step 3, the process of realizing perspective-invariant robust position recognition based on surround image is as follows: First, a set of real-time 360-degree surround view images is collected using M cameras mounted on the vehicle, and the acquired surround view images are preprocessed according to step 1 and input into the trained robust position recognition model; The robust location recognition model outputs a one-dimensional global descriptor for location recognition, thereby achieving location recognition.

Images