A lidar odometry method and system based on state space model and window attention

Abstract

The application relates to a laser radar odometer method and system based on a state space model and window attention, comprising the following steps: projecting original point cloud data and constructing an invalid point mask to filter invalid points; performing down-sampling and region aggregation to reduce the calculation amount; designing a CNN-SSM module to extract multi-scale features considering global and local information; gradually establishing cross-frame association based on a state space window attention fusion encoder; and finally converting the associated features into a pose relationship between adjacent point cloud frames. The application solves the problems of the traditional radar odometer based on deep learning, such as loss of details in global information modeling, decline of cross-frame association confidence of point clouds and high calculation cost caused by attention calculation dependence, and realizes the laser radar odometer with efficient, high real-time and high-precision pose estimation.

Description

Technical Field

[0001] This invention relates to the field of visual odometry technology, and in particular to a lidar odometry method and system based on a state-space model and window attention. Background Technology

[0002] Laser odometry (LO) is a core component of Simultaneous Localization and Mapping (SLAM). It provides the SLAM system with relative pose increments between adjacent time steps in a high-frequency, low-latency manner, allowing subsequent steps to have reliable initial motion values ​​and local maps. LiDAR (Light Detection and Ranging) has become the preferred sensor for SLAM systems due to its high accuracy and insensitivity to ambient light. Point cloud-based odometry estimates pose by optimizing the similarity between a point and a local probability distribution; geometric feature-based odometry estimates pose by minimizing the distance residual from a point to a manually designed local geometric feature (plane, edge, etc.); deep learning-based odometry uses implicit network features plus learnable metrics, achieving more robust estimations than traditional odometry designs in degraded scenes and dynamic objects. However, the accuracy of these methods is generally limited, and their real-time performance is poor. Existing technologies typically employ self-attention mechanisms to model global point cloud information and cross-attention to establish cross-frame associations. This approach suffers from high time complexity, large memory consumption, and low inference efficiency in large-scale point cloud processing. Furthermore, it lacks robust local detail modeling in establishing implicit network features of the point cloud, leading to the loss of some key feature points and hindering subsequent feature matching for cross-frame association establishment. Ultimately, this results in significant pose estimation errors. Therefore, a highly efficient method for modeling and associating global point cloud information is a key technological bottleneck for improving the accuracy and operational efficiency of laser odometry. Summary of the Invention

[0003] To address the shortcomings of existing technologies, this invention provides a lidar odometry method and system based on a state-space model and window attention. It solves the problems of traditional deep learning-based lidar odometry, such as loss of detail in global information modeling, decreased confidence in cross-frame point cloud association leading to limited pose estimation accuracy, and low efficiency and poor real-time performance due to attention computation. This invention combines CNN, SSM, and window cross-attention mechanisms to model global point cloud information and cross-frame association, aiming to improve the efficiency of point cloud information modeling while enhancing the confidence of cross-frame association. This provides high-quality and reliable input to the downstream pose estimation module, improving the performance and applicability of the entire SLAM system from the source.

[0004] Definitions of relevant abbreviations and key terms:

[0005] LO: Laser Odometry, a front-end odometer calculation method that relies solely on continuous lidar scanning data and estimates the sensor's short-term pose increment in real time through point cloud registration;

[0006] SLAM: Simultaneous Localization and Mapping, is a technique that estimates one's own pose in real time and simultaneously constructs a three-dimensional structural representation of the environment.

[0007] SSM: State Space Model, a sequence modeling framework based on linear system control theory. It describes the dynamic characteristics of a system through hidden state variables and linear differential equations. In deep learning, it is used to replace the self-attention mechanism to achieve efficient modeling of long sequence data.

[0008] CNN: Convolutional Neural Network, a deep learning model based on convolution operations, local receptive fields, and weight sharing mechanisms. It extracts local spatial features layer by layer and aggregates hierarchical information through multi-layer convolution and pooling operations to achieve efficient feature learning and pattern recognition.

[0009] To achieve the above technical objectives, the present invention provides the following technical solution: a lidar odometry method based on a state-space model and window attention, comprising the following steps:

[0010] Project consecutive point cloud frames from the original point cloud data into a two-dimensional image, construct an invalid point mask to filter invalid points, and obtain a two-dimensional filtered image;

[0011] The two-dimensional filtered image is downsampled at a fixed step to select sampling points. Each sampling point is used as the center to divide the neighborhood and perform neighborhood feature aggregation to obtain the neighborhood aggregation features of each point cloud frame.

[0012] The neighborhood aggregation features from adjacent point cloud frames are fed into the CNN-SSM module with shared parameters to extract multi-scale features respectively;

[0013] The CNN-SSM module splits the input domain aggregated features into two sub-features, which are then fed into two branches. The upper branch constructs the SS2D module to obtain global information based on the state space model and generate global features. The lower branch extracts local information based on the convolutional neural network and generates local features. Finally, the extraction results from the two branches are concatenated and shuffled, and then concatenated with the input domain aggregated features to output multi-scale features that take into account both global and local information.

[0014] Multi-scale features from adjacent point cloud frames are fed into a state-space window attention fusion encoder to establish cross-frame associations and output associated features; the state-space window attention fusion encoder includes a state-space model and a window cross-attention mechanism.

[0015] The associated features are transformed into pose relationships between adjacent point cloud frames based on the fully connected layer. The pose relationships include rotation quaternions and translation vectors.

[0016] Optionally, the method of projecting continuous point cloud frames in the original point cloud data into a two-dimensional image is cylindrical projection.

[0017] The process of constructing an invalid point mask to filter invalid points and obtain a two-dimensional filtered image includes:

[0018] The invalid point mask is multiplied element-wise with the projected two-dimensional image to filter out the invalid points and obtain a two-dimensional filtered image.

[0019] The invalid point mask is a binary mask with the same size as the two-dimensional image obtained by projection. The value at the position corresponding to the invalid point is 0, and the rest are 1.

[0020] Optionally, the step of dividing the region into domains centered on each sampling point and performing domain feature aggregation includes:

[0021] A rectangle of fixed size is divided in the two-dimensional filtered image with the sampling point as the center, and the points inside the rectangle are the neighborhood points of the sampling point.

[0022] Feature aggregation is performed on points in each domain, and the formula for this process is as follows:

[0023] ;

[0024] in, Indicates the first Domain-specific aggregation features; This represents the max pooling operation; This represents a multilayer perceptron; Indicates the first In the neighborhood of the sampling point, the first Coordinates of points in each domain; Indicates the first Coordinates of each sampling point; Indicates the first Features of each sampling point Indicates the first In the neighborhood of the sampling point, the first Features of each domain point; Indicates the first The total number of neighborhood points in the neighborhood of each sampling point; This indicates a splicing operation. Indicates the use of coordinate difference Transform to a linear projection of a tensor with the same dimensions as the neighborhood point features and the sampling point features.

[0025] Optionally, the upper branch constructs the SS2D module, which generates global features based on the state-space model to obtain global information, including:

[0026] Linear layers and deep convolutional layers are applied sequentially to the neighborhood aggregate sub-features of the upper branch of the input, and then the input is fed into the SS2D module;

[0027] The neighborhood aggregate features from the upper branch are then fed into a linear layer and concatenated with the output features of the SS2D module to obtain the global features.

[0028] The SS2D module performs channel splitting on the input features to generate two secondary sub-features, which are then fed into the introductory control weight branch and the spatial state branch, respectively.

[0029] For the second-level sub-features of the gated weight branch: apply a linear layer with an activation function to generate the gated signal;

[0030] For the secondary sub-features of the input space state branch: they are transformed into four one-dimensional sequences according to row priority, column priority, row priority reverse order, and column priority reverse order; the four one-dimensional sequences are processed in parallel using the state space model; the processed four sequences are then sequentially added and fused element by element, normalized, multiplied element by element with the gate signal, and linearly projected to obtain the output features of the SS2D module.

[0031] Optionally, the lower branch extracts local information and generates local features based on a convolutional neural network, including:

[0032] The neighborhood aggregate features of the input lower branch are sequentially processed by three two-dimensional convolutions with kernel sizes of 3×3, 3×3, and 1×1 to obtain local features.

[0033] Optionally, in the CNN-SSM module, before the upper branch inputs the features into the SS2D module, and before the lower branch performs the first convolution on the features, the invalid point scaling and suppression is performed on the features of each branch using an invalid point mask to suppress the feature values ​​at the positions corresponding to the invalid points.

[0034] Optionally, the step of feeding multi-scale features from adjacent point cloud frames into a state-space window attention fusion encoder to establish cross-frame association and outputting associated features includes:

[0035] Multi-scale features from adjacent point cloud frames are first processed through convolutional and linear layers to enhance local details, and then converted into row and column sequences to obtain four one-dimensional sequences. These sequences are then input into the state space model in parallel to establish soft correspondences between frames. Specifically, the state space models between row sequences from adjacent point cloud frames share the state transition matrix, and the state space models between column sequences from adjacent point cloud frames share the state transition matrix.

[0036] The row and column sequences from the same cloud frame, processed by the state-space model, are fed into a convolutional layer, then into a linear layer to generate soft relation features. These features are then fed into a window cross-attention mechanism to generate association features. The window cross-attention mechanism includes:

[0037] The soft relation features from adjacent point cloud frames are divided into corresponding windows to generate several windows for each.

[0038] The soft relation features of the previous cloud frame are transformed into a query through deep convolution, and the soft relation features of the next cloud frame are transformed into keys and values ​​through deep convolution.

[0039] An attention mask is constructed. Within each corresponding window of adjacent point cloud frames, based on the obtained query, key, and value, a multi-head cross-attention mechanism is executed by combining the attention mask with the relative position encoding. Then, the feature is generated by concatenating the window with a deep convolutional layer.

[0040] Optionally, the attention mask is generated based on the invalid point mask, wherein the value at the position corresponding to the invalid point is set to... The values ​​for the remaining positions are set to 0.

[0041] Optionally, the step of transforming associated features into pose relationships between adjacent point cloud frames based on a fully connected layer includes:

[0042] The associated features are concatenated with the multi-scale features from the previous point cloud frame in the adjacent point cloud frame, and then processed by a multilayer perceptron to generate multi-scale fused features. These features are then projected and normalized to obtain the weight parameters of the fully connected layer used to generate rotation quaternions and translation vectors.

[0043] The multi-scale fused features are fed into two fully connected layers, and the corresponding fully connected layer weight parameters are applied to generate rotation quaternions and translation vectors between adjacent point cloud frames.

[0044] This invention also provides a lidar odometry system based on a state-space model and window attention, for applying the aforementioned lidar odometry method based on a state-space model and window attention, comprising:

[0045] The point cloud data preprocessing module is used to project continuous point cloud frames in the original point cloud data into a two-dimensional image, construct an invalid point mask to filter invalid points, and obtain a two-dimensional filtered image.

[0046] Based on the stride sampling and grouping aggregation module, it is used to downsample the two-dimensional filtered image according to a fixed stride to select sampling points, divide the neighborhood with each sampling point as the center and perform neighborhood feature aggregation to obtain the neighborhood aggregation features of each point cloud frame;

[0047] The CNN-SSM module is used to feed the neighborhood aggregation features of adjacent point cloud frames into the CNN-SSM module with shared parameters to extract multi-scale features respectively.

[0048] The state-space attention fusion encoder module is used to feed the multi-scale features of adjacent point cloud frames into the state-space window attention fusion encoder to establish cross-frame association and output the associated features.

[0049] The pose generation module is used to transform associated features into pose relationships between adjacent point cloud frames based on a fully connected layer.

[0050] By employing the above technical solution, the present invention provides a lidar odometry method and system based on a state-space model and window attention, which has at least the following beneficial effects:

[0051] (1) This invention constructs an invalid point mask to suppress invalid points, and designs a CNN-SSM module to model the system while taking into account both local and global information, thereby improving the effectiveness of the extracted features. By establishing accurate inter-frame relationships through a shared state transition matrix and lightweight window attention, it provides high-quality input for pose calculation and improves the final positioning accuracy of the system.

[0052] (2) This invention reduces the amount of data by downsampling with a fixed stride, improves the efficiency of feature modeling by parallel processing of CNN-SSM modules, and greatly reduces the computational complexity by using a lightweight window attention mechanism, thus meeting the real-time requirements of odometers.

[0053] (3) The present invention designs a new point cloud feature extraction strategy, which uses a dual-branch feature extraction of the CNN-SSM module. The convolutional branch grasps the local details of the point cloud, while the state space model branch models the global information of the point cloud. This ensures that the extracted feature details are not lost and the global information is not missed. The number of model parameters is also greatly reduced, which improves the system operating efficiency.

[0054] (4) This invention designs a new strategy for establishing cross-frame association of point clouds. The state space window attention fusion encoder uses the state space model to perform initial interaction on point cloud information, strengthens the global modeling of the point cloud itself, and establishes a soft correspondence between two frames of point clouds. Subsequently, the window attention mechanism is used to further filter and optimize the correspondence, reduce noise interference, and enable the model to accurately learn the relative pose transformation between two frames.

[0055] (5) The present invention performs point cloud preprocessing, feature extraction, inter-frame fusion and pose generation. The entire process does not require manual feature design, which gets rid of the limitations of manual feature design. The designed model can automatically learn and adapt the feature representation of point cloud characteristics and motion laws, and has strong adaptability. Attached Figure Description

[0056] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0057] Figure 1 This is a flowchart of a lidar odometry method based on a state-space model and window attention proposed in this invention.

[0058] Figure 2 This is a flowchart of the CNN-SSM module of the present invention;

[0059] Figure 3 This is a flowchart of the state space window attention fusion encoder of the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. This will allow for a full understanding of how the present application uses technical means to solve technical problems and achieve technical effects, and to facilitate its implementation.

[0061] Those skilled in the art will understand that all or part of the steps in the implementation of the methods of the embodiments can be implemented by a program instructing related hardware. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Moreover, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0062] Please refer to Figures 1-3This illustration shows a specific implementation of the present embodiment. This embodiment projects the original point cloud data and filters out invalid points, performs downsampling and region aggregation, designs a CNN-SSM module to extract multi-scale features of adjacent point cloud frames, establishes cross-frame association based on a state space window attention fusion encoder, and finally transforms the associated features into pose relationships between adjacent point cloud frames. This solves the problems of traditional deep learning-based radar odometry in modeling global information, decreasing confidence of cross-frame point cloud association, and high computational cost caused by relying on attention calculation. It realizes a high-efficiency, high-real-time, and high-precision pose estimation lidar odometry.

[0063] Please refer to Figure 1 This embodiment proposes a lidar odometry method based on a state-space model and window attention. The method includes the following steps:

[0064] S1. Point cloud data preprocessing.

[0065] Raw point cloud data is inherently sparse, irregular, and disordered. Directly inputting it into a model leads to difficulties in feature extraction and requires adaptation to change-invariant processing logic. Therefore, continuous point cloud frames in the raw point cloud data are projected into a two-dimensional image. This invention uses cylindrical projection. Since LiDAR scanning data is 360-degree circular scanning data, the data naturally possesses azimuth continuity. Cylindrical projection can preserve this characteristic to the greatest extent and maintain good geometric accuracy on the horizontal plane. The mathematical representation of the cylindrical projection process is as follows:

[0066] ;

[0067] ;

[0068] in, , These represent the horizontal and vertical coordinates of a pixel in the projected 2D image, respectively. The 3D channel of each pixel represents its 3D spatial coordinates. ; This is a two-parameter arctangent function used to calculate polar coordinate angles; Represents the arcsine function; , These refer to the horizontal and vertical resolutions of the lidar, respectively.

[0069] Due to the sparsity of point clouds, not all pixels will have 3D projection points. Therefore, a mask is needed for identification to prevent invalid point information from being learned by the subsequent network. During the projection process, an invalid point mask is constructed to filter invalid points, resulting in a 2D filtered image. The mathematical representation of the constructed invalid point mask is as follows:

[0070] ;

[0071] in, , , Representing pixels The pixel values ​​in the x, y, and z channel dimensions. This means that the pixel does not have a corresponding 3D projection point, which is an invalid point; This represents an invalid point mask; the invalid point mask is a binary mask with the same size as the two-dimensional image obtained by projection, where the value at the position corresponding to the invalid point is 0, and the rest are 1;

[0072] The invalid point mask is multiplied element-wise with the two-dimensional image to filter out the invalid points and obtain a two-dimensional filtered image.

[0073] S2, step sampling group aggregation.

[0074] Since the number of point clouds after projection is huge, in order to reduce the burden on the subsequent network, the two-dimensional filtered image needs to be downsampled with a fixed step to select sampling points, and a rectangle of fixed size is divided with each sampling point as the center as the neighborhood of the sampling point, and the points inside the rectangle are the neighborhood points of the sampling point.

[0075] Feature aggregation is performed on each neighborhood point to obtain neighborhood aggregated features. This process is expressed by the following formula:

[0076] ;

[0077] in, Indicates the first Domain-specific aggregation features Used to represent the "global information" contained in the domain aggregation feature, corresponding to the "local information" in the domain point feature; This represents the max pooling operation; This represents a multilayer perceptron; Indicates the first In the neighborhood of the sampling point, the first Coordinates of points in each domain; Indicates the first Coordinates of each sampling point; Indicates the first Features of each sampling point Indicates the first In the neighborhood of the sampling point, the first Features of each domain point; Indicates the first The total number of neighborhood points in the neighborhood of each sampling point; This indicates a concatenation operation, which concatenates the coordinate difference, neighborhood point features, and sampling point features along the feature dimension (the coordinate difference is first linearly projected before concatenation). It is converted into a tensor of the same dimension as the features of the neighborhood points and the features of the sampling points, so as to realize the fusion of local geometric information and high-dimensional semantic information, improve feature discrimination and robustness, and solve the limitations of single information.

[0078] The S3 and CNN-SSM modules extract multi-scale features.

[0079] The CNN-SSM module is one of the core modules of this invention. Since the two-dimensional image generated by projection contains some invalid points, mask suppression is introduced in the CNN-SSM module to prevent contamination of the effectiveness of extracted features. In terms of information modeling, a parallel feature extraction strategy of CNN+SSM is designed to balance local modeling details and modeling efficiency. This invention feeds the neighborhood aggregated features from adjacent point cloud frames into the CNN-SSM module with shared parameters to extract multi-scale features. The CNN-SSM module process can be found in [reference needed]. Figure 2 .

[0080] All parameters are shared between CNN-SSM modules in adjacent point cloud frames. This is because adjacent point cloud frames have spatiotemporal consistency. The feature extraction module with shared parameters reduces the number of parameters while ensuring the consistency of the output.

[0081] As a preferred embodiment of step S3, the specific process includes:

[0082] The input domain aggregation features are channel-split, dividing them equally along the channel dimension into two sub-tensors, generating two domain aggregation sub-features, such as... Figure 2 As shown, the channel splitting process can be described as follows:

[0083] Let the feature map to be split be Its dimensions are ,in , , , These represent the feature maps to be split. Batch size, length, width, and number of channels;

[0084] According to channel dimension ( The dimension is split into two feature maps by average. , All dimensions .

[0085] Based on the above splitting method, the two sub-tensors obtained after splitting are used as neighborhood aggregate sub-features and are respectively fed into the upper and lower branches. The upper branch constructs the SS2D module, which obtains global information based on the state space model to generate global features. Specifically, linear layers and deep convolutional layers are applied sequentially to the features input to the upper branch, and then input into the SS2D module.

[0086] The SS2D module performs channel splitting on the input features to generate two secondary sub-features, which are then fed into the introductory control weight branch and the spatial state branch, respectively.

[0087] For the second-level sub-features of the gated weight branch: apply a linear layer with an activation function to generate the gated signal;

[0088] For the second-level sub-features of the input space state branch: they are transformed into four one-dimensional sequences in row-major, column-major, row-major reverse, and column-major reverse order, each sequence having a length of 1. , , These are the length and width of the input feature, respectively.

[0089] Subsequently, the four one-dimensional sequences are processed in parallel using a state-space model, performing linear recursion within each sequence. The formula is as follows:

[0090] ;

[0091] ;

[0092] in, It is the intermediate hidden state of the previous moment. Indicates time; , They represent the current time. Input and output, hour Given an initial one-dimensional sequence, subsequent time steps... This represents the output from the previous time step; the model parameters in the equation are: Here is the state transition matrix. For the input matrix, For the output matrix, and and This indicates that in order to adapt image data... and The discretized form is shown in the following formula:

[0093] ;

[0094] ;

[0095] in, For exponential operations, The time step parameter for discretization is used to control the discretization sampling interval. express The inverse matrix, It is an identity matrix.

[0096] To leverage the parallel computing advantages of GPUs, the discretized state-space model described above is equivalently transformed into a convolutional form:

[0097] ;

[0098] ;

[0099] in, The derived convolution kernel, , These are the input and output of the state-space model, respectively.

[0100] The outputs of the four sequences obtained from the state-space model are added and fused element by element. Then, the normalization layer stabilizes the feature distribution. The outputs are then multiplied element by element with the gated signal from the gated weight branch to achieve feature selection. Finally, a linear projection layer is used to restore the expanded channel dimension to the original input dimension, which is used as the output feature of the SS2D module.

[0101] The neighborhood aggregate features from the upper branch are then fed into a linear layer and concatenated with the output features of the SS2D module to obtain the global features.

[0102] More specifically, in the CNN-SSM module, before the upper branch inputs features into the SS2D module, and before the lower branch performs the first convolution on the features, the features of each branch are first scaled and suppressed using an invalid point mask. This suppresses the feature values ​​at the positions corresponding to invalid points, ensuring that invalid point features are not learned. The invalid point scaling and suppression formula is defined as follows:

[0103] ;

[0104] in, Represents the original eigenvalues. This represents the eigenvalues ​​after scaling and suppressing invalid points. The invalid point mask constructed for step S1; by... The eigenvalue of the location (i.e., the location corresponding to the invalid point) multiplied by a local minimum. To suppress it.

[0105] The lower branch extracts local information and generates local features based on a convolutional neural network. Specifically, the neighborhood aggregate features input to the lower branch are sequentially passed through three two-dimensional convolutions with kernel sizes of 3×3, 3×3, and 1×1 to obtain local features. The lower branch uses convolutional layers to capture local information, compensating for the shortcomings of local modeling in the state-space model while also reducing the computational complexity of the module. More specifically, before convolution, the features of the lower branch also undergo invalid point scaling and suppression to ensure that invalid point features are not learned.

[0106] Finally, the extraction results from the two branches are concatenated and shuffled, and then concatenated with the input neighborhood aggregation features to output multi-scale features that take into account both global and local information. For example... Figure 1 As shown, the multi-scale features generated by adjacent point cloud frames (point cloud frame 1, point cloud frame 2) are respectively represented as features. ,feature .

[0107] This invention designs a novel point cloud feature extraction strategy: a dual-branch feature extraction of the CNN-SSM module. Employing a channel splitting mechanism, the input neighborhood aggregated feature channels are divided into two independent subsets, each processed through different feature extraction paths on the upper and lower branches. This achieves diversity in feature extraction methods. The two branches focus on features of different scales and types: the lower branch excels at extracting local geometric structures and capturing local details of the point cloud, while the upper branch excels at modeling global contextual relationships. The two branches form complementary feature representations, ensuring that no feature details are lost and no global details are missed, while significantly reducing the number of model parameters and improving system efficiency.

[0108] S4, the state-space attention fusion encoder outputs associated features.

[0109] The process of constructing the state-space attention fusion encoder can be found in [link to documentation]. Figure 3 Multi-scale features from adjacent point cloud frames are first processed through convolutional and linear layers to enhance local details, and then converted into row and column sequences, resulting in four one-dimensional sequences. These sequences are then input into the state space model in parallel to establish soft correspondences between frames. Specifically, the state space model between row sequences from adjacent point cloud frames... Shared state transition matrix State-space model between column sequences from adjacent point cloud frames Shared state transition matrix .

[0110] In a state-space model, the state transition matrix (and its discrete form) has the greatest impact on the system because it controls the evolution of the current hidden state. The input and output matrices mainly affect the input and output states. Therefore, in establishing intra-frame feature interactions, the state transition matrix is ​​shared across frames. (and its discrete form) This allows the network to establish a preliminary correspondence between two frames, and between the two frames... and It remains independent and can establish point-to-point interactions within the feature maps of two frames.

[0111] Next, the row and column sequences from the same point cloud frame, after being processed by the state space model, are fed into a convolutional layer, and then into a unified linear layer to generate soft relation features. The soft relation features of adjacent point cloud frames obtained in this way are the feature maps of two frames that have established a preliminary relationship.

[0112] Next, the features of the previous frame are made to pay attention to the features of the next frame. Since the global cross-attention is computationally complex and the state space model has already established a preliminary association, this invention constructs a lightweight window cross-attention mechanism to establish further cross-frame associations and generate associated features. Specifically: First, the soft relation features from adjacent point cloud frames are divided into corresponding windows (in this embodiment, the window size is fixed at 4×4 spatial positions), generating several windows for each.

[0113] The soft relation features of the previous cloud frame are transformed into a query through deep convolution, and the soft relation features of the next cloud frame are transformed into keys and values ​​through deep convolution.

[0114] Within the corresponding windows of adjacent point cloud frames, a multi-head cross-attention mechanism is executed based on the obtained query, key, and value, combined with attention mask and relative position encoding. Taking 8 heads as an example, the flowchart of the multi-head cross-attention mechanism is expressed as follows:

[0115] ;

[0116] ;

[0117] in, , These represent the soft relation features from the previous cloud frame and the next cloud frame, respectively; Indicates the number of heads paying attention Output of the head; , , They represent the first Each element generates a query from the previous cloud frame. Key from the next cloud frame Values ​​from the next cloud frame The depthwise convolutional weights; This represents the cross-attention mechanism; This represents the softmax normalization operation; Indicates the dimensions of the head. This is a scaling factor used to stabilize training and prevent the softmax gradient from vanishing due to excessively large dot product values. This is an attention mask used to ensure that features from the previous frame only interact with valid points in the next frame. It is generated based on the mask of invalid points. The value at the position corresponding to the invalid point is set to a minimum value (e.g., The values ​​at the remaining positions are set to 0; Encoding for relative positions; This represents the output of the multi-head cross-attention mechanism; This represents the output projection matrix, used to project the outputs of each spliced ​​head.

[0118] Output of multi-head cross-attention mechanism After passing through a deep convolutional layer and concatenating with a window, the associated features are obtained. For example... Figure 1 As shown, the associated features generated by adjacent point cloud frames (point cloud frame 1, point cloud frame 2) are represented as features. .

[0119] The window partitioning process in this step can be described as follows:

[0120] 1) Assume the dimension of the feature map to be segmented is . ,in , , , These represent the batch size, length, width, and number of channels of the feature map to be segmented, respectively.

[0121] 2) Set the window size to Calculate the number of windows: The total number of windows is ,in This indicates the number of windows along the longitudinal direction. Indicates the number of windows in the width direction;

[0122] 3) Reshape the feature map to be segmented into Dimension;

[0123] 4) Next, the dimension is replaced with The first three dimensions Identify the window position;

[0124] 5) Finally, the second time, the dimension was reshaped into The resulting shape is A collection of windows, each window containing Feature vectors of spatial locations.

[0125] The window splicing process in this step can be described as follows:

[0126] 1) Assume the dimension of the feature map to be concatenated is . ,in , These represent the batch size and number of channels of the feature maps to be stitched, respectively. This invention sets... , ;

[0127] 2) Reshape the feature map to be stitched into a six-dimensional tensor containing window position information, with the following dimensions: ;

[0128] 3) Perform inverse dimensional permutation to obtain the shape as tensor;

[0129] 4) Reshape back to the same dimension as the feature map to be segmented. .

[0130] This invention designs a novel strategy for establishing cross-frame association of point clouds, namely a state-space window attention fusion encoder. The state-space model is used to perform initial interaction on point cloud information, which strengthens the global modeling of the point cloud itself and establishes a soft correspondence between point clouds in two frames. Subsequently, the window attention mechanism is used to further filter and optimize the correspondence, reduce noise interference, and enable the model to accurately learn the relative pose transformation between the two frames.

[0131] S5. Pose generation based on fully connected layers.

[0132] Associated features Combined with multi-scale features from the previous point cloud frame (point cloud frame 1) The data is then stitched together and processed by a multilayer perceptron to generate multi-scale fused features. ,in, Representing multi-scale fusion features The first in There are feature tensors, and the total number of these feature tensors is . Then, projection and normalization are performed to obtain the fully connected layer weight parameters used to generate rotation quaternions and translation vectors. = , = The mathematical representation of this process is as follows:

[0133] = ;

[0134] = ;

[0135] = ;

[0136] in, Represents the weight parameters of the fully connected layer The first in One weight parameter, Represents the weight parameters of the fully connected layer The first in One weight parameter, , These are the weight parameters of the fully connected layer. , The total number of weighted parameters; , They represent the methods used for generation. , Linear projection of.

[0137] Subsequently, multi-scale fusion features will be used. Two fully connected layers are fed into the graph to generate rotation quaternions and translation vectors between adjacent point cloud frames, as shown in the following formulas:

[0138] ;

[0139] ;

[0140] in, Represents the rotation quaternion from the previous cloud frame to the next cloud frame; This represents the translation vector from the previous cloud frame to the next cloud frame; , These represent the fully connected layers used to generate rotation quaternions and translation vectors, respectively. , Representing fully connected layers , The number of output channels, For the real number space, This indicates element-wise multiplication. This indicates the calculation of the modulus.

[0141] In practical applications, the step-by-step sampling grouping aggregation step of this invention can be superimposed with the feature extraction step of the CNN-SSM module in up to four layers, such as... Figure 1 As shown, the output of the first layer will serve as the input to the second layer. The sampling point features and neighborhood point features in the strided sampling group aggregation come from the feature map output of the previous layer. When the data enters the strided sampling group aggregation for the first time after data preprocessing, the sampling point features and neighborhood point features will be filled with all-zero vectors. After four layers of feature extraction, the high-level multi-scale features are processed by a state-space window attention fusion encoder to generate associated features. Associated features Multi-scale features extracted from point cloud frame 1 After concatenation, the weight parameters are calculated, and these weight parameters are then passed through two fully connected layers to generate rotation quaternions between adjacent point clouds. Translation vector During the stacking process, the parameters shared by CNN-SSM modules are shared within each layer, and are not shared between different layers.

[0142] In practical applications, the process from step S1 to step S4 is constructed into a lidar odometry network model using a computer language (such as Python). The weight parameters involved in the method of this invention are obtained by training the lidar odometry network model, and the training loss is designed as follows:

[0143] ;

[0144] ;

[0145] ;

[0146] in, Indicates total training loss. Indicates translation loss. Indicates rotational loss; Represents the actual translation vector. Represents a true rotation quaternion. This represents the predicted translation vector, which is the set of translation vectors between adjacent point cloud frames obtained from the LiDAR odometry network model inference. This represents the predicted rotation quaternion, which is the set of rotation quaternions between adjacent point cloud frames obtained from the inference of the lidar odometry network model. ∥⋅∥2 represents the L1 norm, used for translation loss; ∥⋅∥2 represents the L2 norm, used for rotation loss. and These are learnable scalar parameters used to adaptively adjust the weights of translation and rotation losses, respectively, and are combined with the natural logarithm base. Constructed , The two terms are adaptive weighting terms for translation loss and rotation loss, respectively, which play an adaptive weighting role. When the prediction error is large, , Reduce gradient explosion and minimize weight loss. , Increase efforts to promote refined adjustments.

[0147] This application also provides a lidar odometry system based on a state-space model and window attention, for applying the aforementioned lidar odometry method based on a state-space model and window attention, including:

[0148] The point cloud data preprocessing module is used to project continuous point cloud frames in the original point cloud data into a two-dimensional image, construct an invalid point mask to filter invalid points, and obtain a two-dimensional filtered image.

[0149] Based on the stride sampling and grouping aggregation module, it is used to downsample the two-dimensional filtered image according to a fixed stride to select sampling points, divide the neighborhood with each sampling point as the center and perform neighborhood feature aggregation to obtain the neighborhood aggregation features of each point cloud frame;

[0150] The CNN-SSM module is used to feed the neighborhood aggregation features of adjacent point cloud frames into the CNN-SSM module with shared parameters to extract multi-scale features respectively.

[0151] The state-space attention fusion encoder module is used to feed the multi-scale features of adjacent point cloud frames into the state-space window attention fusion encoder to establish cross-frame association and output the associated features.

[0152] The pose generation module is used to transform associated features into pose relationships between adjacent point cloud frames based on a fully connected layer.

[0153] This invention suppresses invalid points by constructing an invalid point mask, improves the effectiveness of extracted features by designing a CNN-SSM module to model in a way that considers both local and global information, and establishes accurate inter-frame relationships through a shared state transition matrix and lightweight window attention, providing high-quality input for pose calculation and improving the final positioning accuracy of the system. Furthermore, this invention reduces data volume through downsampling based on a fixed stride, improves feature modeling efficiency through parallel processing of the CNN-SSM module, and significantly reduces computational complexity using a lightweight window attention mechanism, adapting to the real-time requirements of odometry. In summary, this invention performs point cloud preprocessing, feature extraction, inter-frame fusion, and pose generation without requiring manual feature design, overcoming the limitations of manual feature design. The designed model can automatically learn and adapt feature representations to point cloud characteristics and motion laws, exhibiting strong adaptability.

[0154] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of those different embodiments or examples.

[0155] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus or device (such as a computer-based system, a processor-included system or other system that can fetch and execute instructions from, an instruction execution system, apparatus or device).

[0156] The above embodiments provide a detailed description of the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A lidar odometry method based on a state-space model and window attention, characterized in that, include: Project consecutive point cloud frames from the original point cloud data into a two-dimensional image, construct an invalid point mask to filter invalid points, and obtain a two-dimensional filtered image; The two-dimensional filtered image is downsampled at a fixed step to select sampling points. Each sampling point is used as the center to divide the neighborhood and perform neighborhood feature aggregation to obtain the neighborhood aggregation features of each point cloud frame. The neighborhood aggregation features from adjacent point cloud frames are fed into the CNN-SSM module with shared parameters to extract multi-scale features respectively; The CNN-SSM module splits the input domain aggregated features into two sub-features, which are then fed into two branches. The upper branch constructs the SS2D module to obtain global information based on the state space model and generate global features. The lower branch extracts local information based on the convolutional neural network and generates local features. Finally, the extraction results from the two branches are concatenated and shuffled, and then concatenated with the input domain aggregated features to output multi-scale features that take into account both global and local information. Multi-scale features from adjacent point cloud frames are fed into a state space window attention fusion encoder to establish cross-frame associations and output associated features. The state-space window attention fusion encoder includes a state-space model and a window cross-attention mechanism; The associated features are transformed into pose relationships between adjacent point cloud frames based on the fully connected layer. The pose relationships include rotation quaternions and translation vectors.

2. The lidar odometry method based on state-space model and window attention according to claim 1, characterized in that: The method of projecting continuous point cloud frames in the original point cloud data into a two-dimensional image is to use cylindrical projection. The process of constructing an invalid point mask to filter invalid points and obtain a two-dimensional filtered image includes: The invalid point mask is multiplied element-wise with the projected two-dimensional image to filter out the invalid points and obtain a two-dimensional filtered image. The invalid point mask is a binary mask with the same size as the two-dimensional image obtained by projection. The value at the position corresponding to the invalid point is 0, and the rest are 1.

3. The lidar odometry method based on state-space model and window attention according to claim 1, characterized in that: The process of dividing the area into regions centered on each sampling point and aggregating region features includes: A rectangle of fixed size is divided in the two-dimensional filtered image with the sampling point as the center, and the points inside the rectangle are the neighborhood points of the sampling point. Feature aggregation is performed on points in each domain, and the formula for this process is as follows: ； in, Indicates the first Domain-specific aggregation features; This represents the max pooling operation; This represents a multilayer perceptron; Indicates the first In the neighborhood of the sampling point, the first Coordinates of points in each domain; Indicates the first Coordinates of each sampling point; Indicates the first Features of each sampling point Indicates the first In the neighborhood of the sampling point, the first Features of each domain point; Indicates the first The total number of neighborhood points in the neighborhood of each sampling point; This indicates a splicing operation. Indicates the use of coordinate difference Transform to a linear projection of a tensor with the same dimensions as the neighborhood point features and the sampling point features.

4. The lidar odometry method based on state-space model and window attention according to claim 1, characterized in that: The upper branch constructs the SS2D module, which obtains global information based on the state-space model to generate global features, including: Linear layers and deep convolutional layers are applied sequentially to the neighborhood aggregate sub-features of the upper branch of the input, and then the input is fed into the SS2D module; The neighborhood aggregate features from the upper branch are then fed into a linear layer and concatenated with the output features of the SS2D module to obtain the global features. The SS2D module performs channel splitting on the input features to generate two secondary sub-features, which are then fed into the introductory control weight branch and the spatial state branch, respectively. For the second-level sub-features of the gated weight branch: apply a linear layer with an activation function to generate the gated signal; For the secondary sub-features of the input space state branch: they are transformed into four one-dimensional sequences according to row priority, column priority, row priority reverse order, and column priority reverse order; the four one-dimensional sequences are processed in parallel using the state space model; the processed four sequences are then sequentially added and fused element by element, normalized, multiplied element by element with the gate signal, and linearly projected to obtain the output features of the SS2D module.

5. The lidar odometry method based on state-space model and window attention according to claim 1, characterized in that: The lower branch extracts local information and generates local features based on a convolutional neural network, including: The neighborhood aggregate features of the input lower branch are sequentially processed by three two-dimensional convolutions with kernel sizes of 3×3, 3×3, and 1×1 to obtain local features.

6. The lidar odometry method based on state-space model and window attention according to claim 1, characterized in that: The CNN-SSM module, before the upper branch inputs the features into the SS2D module, and before the lower branch performs the first convolution on the features, respectively, uses an invalid point mask to scale and suppress the invalid point features, thereby suppressing the feature values ​​at the positions corresponding to the invalid points.

7. The lidar odometry method based on state-space model and window attention according to claim 1, characterized in that: The step of feeding multi-scale features from adjacent point cloud frames into a state space window attention fusion encoder to establish cross-frame association and outputting associated features includes: Multi-scale features from adjacent point cloud frames are first processed through convolutional and linear layers to enhance local details, and then converted into row and column sequences to obtain four one-dimensional sequences. These sequences are then input into the state space model in parallel to establish soft correspondences between frames. Specifically, the state space models between row sequences from adjacent point cloud frames share the state transition matrix, and the state space models between column sequences from adjacent point cloud frames share the state transition matrix. The row and column sequences from the same cloud frame, processed by the state-space model, are fed into a convolutional layer, then into a linear layer to generate soft relation features. These features are then fed into a window cross-attention mechanism to generate association features. The window cross-attention mechanism includes: The soft relation features from adjacent point cloud frames are divided into corresponding windows to generate several windows for each. The soft relation features of the previous cloud frame are transformed into a query through deep convolution, and the soft relation features of the next cloud frame are transformed into keys and values ​​through deep convolution. An attention mask is constructed. Within each corresponding window of adjacent point cloud frames, based on the obtained query, key, and value, a multi-head cross-attention mechanism is executed by combining the attention mask with the relative position encoding. Then, the feature is generated by concatenating the window with a deep convolutional layer.

8. The lidar odometry method based on state-space model and window attention according to claim 7, characterized in that: The attention mask is generated based on the invalid point mask, wherein the value at the position corresponding to the invalid point is set to... The values ​​for the remaining positions are set to 0.

9. The lidar odometry method based on state-space model and window attention according to claim 1, characterized in that: The process of transforming associated features into pose relationships between adjacent point cloud frames based on a fully connected layer includes: The associated features are concatenated with the multi-scale features from the previous point cloud frame in the adjacent point cloud frame, and then processed by a multilayer perceptron to generate multi-scale fused features. These features are then projected and normalized to obtain the weight parameters of the fully connected layer used to generate rotation quaternions and translation vectors. The multi-scale fused features are fed into two fully connected layers, and the corresponding fully connected layer weight parameters are applied to generate rotation quaternions and translation vectors between adjacent point cloud frames.

10. A lidar odometry system based on a state-space model and window attention, used to apply the lidar odometry method based on a state-space model and window attention as described in any one of claims 1-9, characterized in that, include: The point cloud data preprocessing module is used to project continuous point cloud frames in the original point cloud data into a two-dimensional image, construct an invalid point mask to filter invalid points, and obtain a two-dimensional filtered image. Based on the stride sampling and grouping aggregation module, it is used to downsample the two-dimensional filtered image according to a fixed stride to select sampling points, divide the neighborhood with each sampling point as the center and perform neighborhood feature aggregation to obtain the neighborhood aggregation features of each point cloud frame; The CNN-SSM module is used to feed the neighborhood aggregation features of adjacent point cloud frames into the CNN-SSM module with shared parameters to extract multi-scale features respectively. The state-space attention fusion encoder module is used to feed the multi-scale features of adjacent point cloud frames into the state-space window attention fusion encoder to establish cross-frame association and output the associated features. The pose generation module is used to transform associated features into pose relationships between adjacent point cloud frames based on a fully connected layer.

Images