Sequence multi-modal scene recognition method based on state space model

Abstract

The invention belongs to the technical field of scene recognition, and discloses a sequence multi-modal scene recognition method based on a state space model. Laser point clouds and visual images in the tracks are jointly coded to form multi-modal sequence data, the multi-modal sequence data are processed into unique global descriptors through a global descriptor coding network, and the global descriptors of the two tracks are used as maps and queries respectively; and in the query process, finding the most similar data in the map through a nearest neighbor search algorithm to complete scene recognition. According to the invention, fusion coding is carried out on laser point cloud and image data, the data dimension is increased, and the data complexity is compressed. A global descriptor is obtained through a single frame module and a sequence module based on a state space model. The calculation efficiency is improved through the cross scanning design in the single-frame module, and the position recognition problem of track change in a scene is solved through the sequence full combination representation strategy in the sequence module. The method has the characteristics of high calculation efficiency, high accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of scene recognition, and in particular to a sequence multimodal scene recognition method based on a state space model. Background Art

[0002] Scene recognition algorithms have shown broad application prospects in the fields of autonomous driving, robotics, virtual reality, etc. Efficient and robust scene recognition algorithms are crucial to promoting the development of these fields. Traditional scene recognition methods based on vision or point cloud are limited by the limitations of single-modal data and cannot fully capture the rich information of the scene. The fusion of serialized multimodal data provides a promising way to improve scene recognition performance. However, the introduction of multimodal data also brings new challenges: on the one hand, the feature extraction of multimodal data is computationally intensive, and how to balance model accuracy and complexity is a difficult problem; on the other hand, how to effectively extract the variation features of serialized data and express them globally is also an urgent problem to be solved. In recent years, deep learning methods have made significant progress in the field of scene recognition and gradually replaced traditional methods based on geometric features. By encoding and reducing the dimension of multimodal data, the model can be effectively simplified and the computational efficiency can be improved. Researchers are also paying more and more attention to the important role of serialized data in scene recognition. Many studies have encoded serialized data. Although a large number of scene recognition algorithms based on serialized data have emerged, few methods can effectively utilize the temporal and spatial variation characteristics of data, and there is still room for improvement in large-scale outdoor scene recognition.

[0003] The paper "Luo L, Cao SY, Han B, et al. BVMatch: Lidar-Based Place Recognition Using Bird's-Eye View Images [J]. IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 6076-6083. 2021." projects the 3D point cloud onto the bird's-eye view image, and then uses a set of Log-Gabor filters to construct the maximum index map to encode the directional information of the structure in the image. Although the computational efficiency of descriptor generation is improved, the loss of vertical information of the scene cannot be avoided.

[0004] The paper "Komorowski J. MinkLoc3D: Point Cloud Based Large-Scale Place Recognition [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, pp. 1790-1799.2021." computes discriminative 3D point cloud descriptors based on sparse voxelized point cloud representation and sparse 3D convolution. Using sparse 3D convolution to extract global features of point clouds ensures real-time calculation, but it cannot effectively cope with the rotation changes of the scene.

[0005] The paper "Li L, Kong X, Zhao X, et al. RINet: Efficient 3D lidar-based place recognition using rotation invariant neural network[J]. IEEE Robotics and Automation Letters, vol.7, no.2, pp.4321-4328.2022." combines semantic and geometric features to improve the description ability, and then uses a rotation-invariant twin neural network to predict the similarity between descriptors. The combination of semantic and geometric information ensures the integrity of the information, but the computational efficiency is lower than other methods.

[0006] The paper "Zhao S, Yin P, Yi G, et al. Spherevlad++: Attention-based and signal-enhanced viewpoint invariant descriptor [J]. IEEE Robotics and Automation Letters, vol. 8, no. 1, pp. 256-263. 2023" projects point clouds on spherical perspectives of multiple unique regions and captures the contextual connection between local features and their dependencies on the global 3D geometric distribution, and has rotation invariance. The local directional equivalent features are encoded by spherical convolution, but the correlation between features is not utilized.

[0007] In summary, current research cannot guarantee recognition accuracy, real-time performance, and complete scene expression at the same time, and has not yet effectively processed the feature fusion of multimodal data and the spatiotemporal feature extraction of sequence data. Summary of the invention

[0008] Based on the shortcomings of the prior art, the present invention proposes a sequence multimodal scene recognition method based on a state space model, which performs scene recognition based on sequence multimodal data, encodes image and point cloud data into a fusion feature representation that can be efficiently calculated, and uses the spatiotemporal variation characteristics of sequence data to solve the recognition problem of trajectory changes in the scene. A balance is achieved between scene recognition accuracy and real-time performance, and the robustness of scene recognition is improved.

[0009] The technical solution of the present invention is as follows: a sequential multimodal scene recognition method based on a state-space model, which jointly encodes the laser point cloud and visual image in the trajectory to form multimodal sequence data, which is processed into a unique global descriptor through a global descriptor encoding network, and the global descriptors of the two trajectories are used as a map and a query respectively; during the query process, the most similar data in the map is found through a nearest neighbor search algorithm to complete scene recognition.

[0010] Further, the global descriptor encoding network includes a single frame module and a sequence module;

[0011] The multimodal sequence data is input into a global descriptor encoding network, and after passing through a single frame module and a sequence module in sequence, a global descriptor is obtained;

[0012] The single-frame module includes an ECA backbone network, a multimodal state space network and a VLAD aggregation network; each group of multimodal sequence data is first processed by the ECA backbone network, and the five-channel multimodal data is subjected to channel attention fusion to obtain the main features; the multimodal state space network further selectively learns deep features and uses the VLAD aggregation network to obtain a single-frame data descriptor;

[0013] The sequence module includes a sequence state space network and a pooling layer; the sequence state space network fuses multiple single-frame data descriptors, and then uses the GeM pooling layer to aggregate to generate a global descriptor with a dimension of 256;

[0014] The multimodal state space network and the sequence state space network are mainly composed of multiple basic state space models.

[0015] Furthermore, the ECA backbone network is a five-layer convolutional neural network that uses a channel attention mechanism to fuse information from different channels; the multimodal state space network uses a cross-scanning method to achieve a global receptive field in multimodal sequence data, and the multimodal sequence data is scanned in opposite directions starting from the upper left, lower left, upper right and lower right corners to ensure that each pixel can fully utilize the neighborhood information, and then the different sequences are merged into the original size to obtain different sequences; each sequence is processed by the basic state space model.

[0016] Furthermore, the basic state space model input is divided into two parts; one part passes through the Linear layer and convolution and enters the SS2D module, while activating the SinLU function; the other part passes through the Linear layer and activates the SinLU function at the same time; the outputs of the two branches are merged and then merged with the original input after passing through the Linear layer and input into the VLAD layer module.

[0017] Furthermore, after the sequence module obtains multiple single-frame descriptors, it connects the multiple single-frame descriptors in sequence and arranges the connection order in combination; the connection descriptors of multiple combination orders are respectively input into the basic state space model of the single layer, and the results of multiple arrangements are fused by weighted addition, and input into the GeM pooling layer to generate a sequence global descriptor.

[0018] Furthermore, the joint coding is specifically:

[0019] Align and correlate visual images and laser point clouds at the data level;

[0020] First, the laser point cloud P = {x, y, z} is projected from the laser radar coordinate system to the visual image plane I = {u, v}, the formula is as follows:

[0021]

[0022] The homogeneous representation of the laser point cloud coordinates is X = [x, y, z, 1] T , the homogeneous representation of the visual image coordinates is Y = [u, v, 1] T ;M 1 and M 2 They are the camera internal parameter matrix and the laser radar to camera external parameter matrix, Z c Represents the z coordinate of a spatial point in the camera coordinate system;

[0023] After projection, the corresponding RGB color information is assigned to each projection point, and the laser point cloud format is converted to P' = {x, y, z, I, r, g, b}, where I represents the reflection intensity;

[0024] A polar coordinate system with the lidar sensor as the origin is used to divide the 3D space into several sectors according to the set horizontal angular resolution and vertical angular resolution. Each sector contains a set of laser point clouds after format conversion. Among these laser point clouds, the laser point with the largest distance from the lidar sensor is selected as the representative point. The representative point represents the maximum detectable range of the lidar sensor in this direction, which is defined as follows:

[0025]

[0026] Among them, P x , P y , P zare the Cartesian coordinates of the point, L 2 Indicates L 2 norm, Max represents the maximum value, Φ() represents the point corresponding to the maximum value, and n represents the number of points in the sector; the multidimensional information contained in the points is encoded into a two-dimensional image according to the arrangement order of the sectors to obtain multimodal sequence data, called SMRD, which is defined as follows:

[0027] SMRD(k)=P′ ij (k)∈Ri,j∈(V s , H s ),

[0028] k∈(P′ I , P′ R , P′ r , P′ g , P′ b ).

[0029] Among them, V s and H s Refers to the number of rows and columns that divide the sector, P′ I , P′ R , P′ r , P′ g , P′ b They refer to the reflection intensity of each point, the distance from the point to the sensor, and the data domain of the RGB information;

[0030] The SMRD is a five-channel two-dimensional structure data containing reflection intensity I, range R and color (r, g, b) information.

[0031] Beneficial effects of the present invention: In view of the inherent defects of a single modality in current scene recognition tasks and the inability to effectively utilize the spatiotemporal characteristics of sequence data, the present invention proposes a sequence multimodal scene recognition method based on a state-space model to achieve highly robust scene recognition. First, it is proposed to fuse the laser point cloud and image data to increase the data dimension and compress the complexity of the data. Subsequently, a global descriptor is obtained through a single-frame module and a sequence module based on a state-space model. The cross-scanning design in the single-frame module improves the computational efficiency, and the sequential full combination representation strategy in the sequence module solves the problem of position recognition of trajectory changes in the scene. The invention can be effectively used in real-time global point cloud scene recognition tasks due to its computational efficiency, high accuracy and robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a flow chart of the method of the present invention;

[0033] Figure 2 Encode a network flow graph for the global descriptor;

[0034] Figure 3 Schematic diagram of the basic state space model. DETAILED DESCRIPTION

[0035] Figure 1 This is the overall flow chart of the technical solution of the present invention. The present invention proposes a sequential multimodal scene recognition method based on a state space model. The laser point cloud and visual image in the trajectory are first processed into a unique global descriptor through the proposed global descriptor encoding network. The global descriptors of the two trajectories are used as a map and a query respectively. Then, in the query process, the nearest neighbor search algorithm is used to find the most similar data in the map, thus realizing scene recognition.

[0036] Joint encoding of multimodal data:

[0037] Panoramic images and whole scene point clouds show inherent alignment at the data level, and the two modalities are strongly correlated. Data feature-level fusion is of great significance in multimodal scene recognition. By aligning and correlating vision and point clouds at the data level, the dimensionality of the data can be increased, thereby enriching the place description.

[0038] First, the point cloud (P = {x, y, z}) is projected from the lidar coordinate system to the image plane (I = {u, v}) using the following formula:

[0039]

[0040] The homogeneous representation of point cloud and image coordinates is represented as X = [x, y, z, 1] T and Y = [u,v,1] T , M 1 and M 2 They are the camera internal parameter matrix and the laser radar to camera external parameter matrix, Z c Represents the z coordinate of the spatial point in the camera coordinate system. After projection, each point is assigned corresponding RGB color information, and the laser point cloud format is converted to P'={x, y, z, i, r, g, b}, where i represents the reflection intensity.

[0041] The horizontal field of view of a lidar sensor is usually 360 degrees, and its vertical field of view is usually between 30 and 60 degrees. A polar coordinate system with the lidar sensor as the origin is used to divide the 3D space into several sectors according to the set horizontal angular resolution and vertical angular resolution. Each sector contains a set of format-converted laser point clouds. Among these format-converted laser point clouds, the point with the largest distance from the lidar sensor is selected as the representative point. The representative point represents the maximum detectable range of the lidar sensor in this direction, which has geographic resolution significance. The definition is as follows:

[0042]

[0043] Among them, P x , P y , P z are the Cartesian coordinates of the point, L 2 Indicates L 2 norm, Max represents the maximum value, Φ() represents the point corresponding to the maximum value, and n represents the number of points in the sector. The multi-dimensional information contained in the points is encoded into a two-dimensional image according to the arrangement order of the sectors, and the multi-modal fusion coding is obtained, which is called SMRD and is defined as follows:

[0044] SMRD(k)=P′ ij (k)∈R i,j∈(V s , H s ),

[0045] k∈(P′ I , P′ R , P′ r , P′ g , P′ b ).

[0046] It is a five-channel two-dimensional structure data containing intensity I, range R and color (r, g, b) information.

[0047] The global descriptor encoding network is as follows Figure 2 As shown in the figure; the input of the global descriptor encoding network is multimodal sequence data. After being processed by the single-frame module and the sequence module, the global descriptor encoding network outputs a global descriptor. In the single-frame module, each set of multimodal sequence data is first processed by the ECA backbone network, which performs channel attention fusion on the five-channel multimodal data to obtain the main features. Subsequently, the multimodal state-space network further selectively learns deep features and uses VLAD aggregation to obtain the descriptor of the single-frame data. Subsequently, the sequence state-space network fuses multiple single-frame data descriptors. Finally, the GeM pooling layer is used to aggregate the features to generate a global descriptor with a dimension of 256.

[0048] The single frame module is specifically as follows:

[0049] The ECA backbone network is a five-layer convolutional neural network that uses a channel attention mechanism to fuse information from different channels. The multimodal state-space network uses a cross-scan approach to achieve a global receptive field in a two-dimensional image, where the two-dimensional image is scanned in opposite directions starting from the top left, bottom left, top right, and bottom right corners. This ensures that each pixel can fully utilize the neighborhood information, and then merges different sequences into the original size. Each sequence is processed by the underlying state-space model.

[0050] The basic state space model is Figure 3As shown in the figure, one part passes through the Linear layer and convolution and then enters the SS2D module, activating the SinLU function at the same time; the other part passes through the Linear layer and activates the SinLU function at the same time; the outputs of the two branches are merged and then added to the input in a form similar to cross-layer connection. This combination constitutes the basic state space model. Downsampling is performed under the stacking of multiple layers. The extracted valid features are then aggregated into a simple single-frame descriptor by the VLAD layer.

[0051] The sequence module is specifically as follows: after obtaining multiple single-frame descriptors, they are fused to generate a sequence global descriptor. This is different from traditional sequence-based methods, which rely on the order of the sequence to remain unchanged. Our goal is to avoid limiting the situation of entering the same scene to a fixed trajectory, because a scene usually involves a variety of different travel trajectories, including reverse travel or rotation. Taking the fusion of three single-frame descriptors as an example, their full sequence combination arrangements are listed, a total of 6. Multiple state-space network modules are used to process each arrangement in a manner similar to single-frame cross scanning, and then the features are fused into unique sequence global features, which are then aggregated into global descriptors through the GeM pooling layer.

[0052] The global descriptor encoding network proposed in this invention is not only computationally efficient and can utilize the features of different modalities, but can also effectively utilize the spatiotemporal features of sequence data to enhance scene recognition. The proposed global descriptor encoding network can be effectively applied to scene recognition tasks, ensuring real-time performance while ensuring high accuracy of scene recognition.

[0053] In order to verify that this method can achieve a balance between real-time and accuracy, the present invention was tested on two datasets, NCLT and self-recorded data-level NEU-dataset, respectively. The experimental platform is equipped with Core i7-10700 CPU@2.9GHz and Nvidia RTX 3090GPU. The AR@1% (average recall rate) on the two datasets is 92.2% and 99.0% respectively. It can be seen that our method can maintain good results under datasets of different types of laser sensors, while ensuring the accuracy of scene recognition, it can still maintain good real-time performance, and can be effectively used for tasks such as large-scale scene recognition.

Claims

1. A method for sequential multimodal scene recognition based on a state space model, characterized in that: The laser point cloud and visual image in the trajectory are jointly encoded to form multimodal sequence data, which are processed into unique global descriptors through a global descriptor encoding network. The global descriptors of the two trajectories are used as maps and queries respectively. During the query process, the most similar data in the map is found through the nearest neighbor search algorithm to complete scene recognition.

2. The method for sequential multimodal scene recognition based on a state space model according to claim 1, characterized in that: The global descriptor encoding network includes a single frame module and a sequence module; The multimodal sequence data is input into a global descriptor encoding network, and after passing through a single frame module and a sequence module in sequence, a global descriptor is obtained; The single-frame module includes an ECA backbone network, a multimodal state space network and a VLAD aggregation network; each group of multimodal sequence data is first processed by the ECA backbone network, and the five-channel multimodal data is subjected to channel attention fusion to obtain the main features; the multimodal state space network further selectively learns deep features and uses the VLAD aggregation network to obtain a single-frame data descriptor; The sequence module includes a sequence state space network and a pooling layer; the sequence state space network fuses multiple single-frame data descriptors, and then uses the GeM pooling layer to aggregate to generate a global descriptor with a dimension of 256; The multimodal state space network and the sequence state space network are mainly composed of multiple basic state space models.

3. The method for sequential multimodal scene recognition based on a state space model according to claim 2, characterized in that: The ECA backbone network is a five-layer convolutional neural network that uses a channel attention mechanism to fuse information from different channels; the multimodal state space network uses a cross-scanning method to achieve a global receptive field in multimodal sequence data. The multimodal sequence data is scanned in opposite directions starting from the upper left, lower left, upper right and lower right corners to ensure that each pixel can fully utilize the neighborhood information, and then the different sequences are merged into the original size to obtain different sequences; each sequence is processed by the basic state space model.

4. The method for sequential multimodal scene recognition based on a state space model according to claim 3, characterized in that: The basic state space model input is divided into two parts; one part passes through the Linear layer and convolution and enters the SS2D module, activating the SinLU function at the same time; the other part passes through the Linear layer and activates the SinLU function at the same time; the outputs of the two branches are merged and then merged with the original input after passing through the Linear layer and input into the VLAD layer module.

5. The method for sequential multimodal scene recognition based on a state space model according to claim 2, characterized in that: After the sequence module obtains multiple single-frame descriptors, the multiple single-frame descriptors are connected in sequence and the connection order is combined and arranged; the connection descriptors of multiple combination orders are respectively input into the basic state space model of the single layer, and the results of multiple arrangements are fused by weighted addition, and input into the GeM pooling layer to generate a sequence global descriptor.

6. The method for sequential multimodal scene recognition based on a state space model according to any one of claims 1 to 5, characterized in that: The joint coding is specifically: Align and correlate visual images and laser point clouds at the data level; First, the laser point cloud P = {x, y, z} is projected from the laser radar coordinate system to the visual image plane I = {u, v}, the formula is as follows: The homogeneous representation of the laser point cloud coordinates is X = [x, y, z, 1] T , the homogeneous representation of the visual image coordinates is Y = [u, v, 1] T ; M1 and M2 are the camera internal parameter matrix and the laser radar to camera external parameter matrix, respectively. c Represents the z coordinate of a spatial point in the camera coordinate system; After projection, the corresponding RGB color information is assigned to each projection point, and the laser point cloud format is converted to P' = {x, y, z, I, r, g, b}, where I represents the reflection intensity; A polar coordinate system with the lidar sensor as the origin is used to divide the 3D space into several sectors according to the set horizontal angular resolution and vertical angular resolution. Each sector contains a set of laser point clouds after format conversion. Among these laser point clouds, the laser point with the largest distance from the lidar sensor is selected as the representative point. The representative point represents the maximum detectable range of the lidar sensor in this direction, which is defined as follows: Among them, P x , P y , P z are the Cartesian coordinates of the points, L2 represents the L2 norm, Max represents the maximum value, Φ() represents the point corresponding to the maximum value, and n represents the number of points in the sector; the multidimensional information contained in the points is encoded into a two-dimensional image according to the arrangement order of the sectors to obtain multimodal sequence data, called SMRD, which is defined as follows: SMRD(k)=P′ ij (k)∈R i,j∈(V s ,H s ), k∈(P′ I ,P′ R ,P′ r ,P′ g ,P′ b ). Among them, V s and H s Refers to the number of rows and columns that divide the sector, P′ I , P′ R , P′ r , P′ g , P′ b They refer to the reflection intensity of each point, the distance from the point to the sensor, and the data domain of the RGB information; The SMRD is a five-channel two-dimensional structure data containing reflection intensity I, range R and color (r, g, b) information.

Images