A Sequential Multimodal Scene Recognition Method Based on State-Space Model

Abstract

This invention belongs to the field of scene recognition technology and discloses a sequential multimodal scene recognition method based on a state-space model. It involves jointly encoding laser point clouds and visual images from a trajectory to form multimodal sequence data, which is then processed into unique global descriptors through a global descriptor encoding network. The global descriptors of the two trajectories serve as the map and the query, respectively. During the query process, a nearest neighbor search algorithm is used to find the most similar data in the map, completing scene recognition. This invention proposes to fuse and encode laser point cloud and image data, increasing data dimensionality and compressing data complexity. Global descriptors are obtained through a single-frame module and a sequence module based on a state-space model. The cross-scan design in the single-frame module improves computational efficiency, while the sequential full-combination representation strategy in the sequence module solves the problem of position recognition for trajectory changes within the scene. This invention features high computational efficiency, high accuracy, and robustness.

Description

Technical Field

[0001] This invention relates to the field of scene recognition technology, and in particular to a sequence multimodal scene recognition method based on a state-space model. Background Technology

[0002] Scene recognition algorithms have shown broad application prospects in fields such as autonomous driving, robotics, and virtual reality. Efficient and robust scene recognition algorithms are crucial for driving the development of these fields. Traditional scene recognition methods based on vision or point clouds are limited by the constraints of single-modal data, making it difficult to fully capture the rich information of a scene. The fusion of serialized multimodal data provides a promising approach to improving scene recognition performance. However, the introduction of multimodal data also brings new challenges: on the one hand, feature extraction from multimodal data is computationally intensive, and balancing model accuracy and complexity is a difficult problem; on the other hand, effectively extracting the changing features of serialized data and representing them globally is also an urgent issue to be addressed. In recent years, deep learning methods have made significant progress in scene recognition, gradually replacing traditional geometric feature-based methods. By encoding and dimensionality reduction of multimodal data, models can be effectively simplified and computational efficiency improved. Researchers are also increasingly focusing on the important role of serialized data in scene recognition, and many studies have focused on encoding serialized data. Although a large number of scene recognition algorithms based on serialized data have emerged, few methods can effectively utilize the spatiotemporal variation characteristics of the data, and there is still room for improvement in large-scale outdoor scene recognition.

[0003] The paper "Luo L, Cao SY, Han B, et al. BVMatch: Lidar-Based Place Recognition Using Bird's-Eye View Images[J]. IEEE Robotics and Automation Letters, vol. 6, no. 3, pp. 6076-6083. 2021." projects 3D point clouds onto bird's-eye view images and then uses a set of Log-Gabor filters to construct a maximum index map to encode the orientation information of structures in the image. Although this improves the computational efficiency of descriptor generation, it cannot avoid the loss of vertical orientation information of the scene.

[0004] The paper "Komorowski J. MinkLoc3D: Point Cloud Based Large-Scale Place Recognition [C] / / Proceedings of the IEEE / CVF Winter Conference on Applications of Computer Vision, pp. 1790-1799. 2021." uses sparse voxelized point cloud representation and sparse 3D convolution to compute discriminative 3D point cloud descriptors. While using sparse 3D convolution to extract global features of the point cloud ensures real-time computation, it cannot effectively handle scene rotation changes.

[0005] The paper "Li L, Kong X, Zhao X, et al. RINet: Efficient 3D lidar-based place recognition using rotation invariant neural network[J]. IEEE Robotics and Automation Letters, vol.7, no.2, pp.4321-4328.2022." combines semantic and geometric features to improve descriptive ability, and then uses a rotation-invariant Siamese neural network to predict the similarity between descriptors. While utilizing the combination of semantic and geometric information ensures the integrity of the information, its computational efficiency is lower than other methods.

[0006] The paper "Zhao S, Yin P, Yi G, et al. Spherevlad++: Attention-based and signal-enhanced viewpoint invariant descriptor[J]. IEEE Robotics and Automation Letters, vol.8, no.1, pp.256-263.2023." projects point clouds onto a spherical viewpoint in multiple unique regions, capturing the contextual connections between local features and their dependencies on the global 3D geometric distribution, and exhibiting rotation invariance. It encodes equivalent features in local orientations through spherical convolution, but does not utilize the correlations between features.

[0007] In summary, current research cannot simultaneously guarantee recognition accuracy, real-time performance, and complete scene representation, and has not yet effectively handled feature fusion of multimodal data and spatiotemporal feature extraction of sequence data. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention proposes a sequential multimodal scene recognition method based on a state-space model. This method performs scene recognition based on sequential multimodal data, encoding image and point cloud data into efficiently computed fusion feature representations, and utilizing the spatiotemporal variation features of the sequential data to solve the problem of identifying trajectory changes within the scene. This achieves a balance between scene recognition accuracy and real-time performance, while also improving the robustness of scene recognition.

[0009] The technical solution of the present invention is as follows: a sequential multimodal scene recognition method based on a state-space model, which jointly encodes laser point clouds and visual images in a trajectory to form multimodal sequence data, processes them into unique global descriptors through a global descriptor encoding network, and uses the global descriptors of the two trajectories as the map and the query, respectively; during the query process, the most similar data in the map is found through the nearest neighbor search algorithm to complete scene recognition.

[0010] Furthermore, the global descriptor encoding network includes a single-frame module and a sequence module;

[0011] The multimodal sequence data is input into the global descriptor encoding network, and after passing through the single-frame module and the sequence module in sequence, a global descriptor is obtained.

[0012] The single-frame module includes an ECA backbone network, a multimodal state space network, and a VLAD aggregation network. Each set of multimodal sequence data is first processed by the ECA backbone network, which performs channel attention fusion on the five-channel multimodal data to obtain the main features. The multimodal state space network further selectively learns deep features and uses the VLAD aggregation network to obtain the single-frame data descriptor.

[0013] The sequence module includes a sequence state space network and a pooling layer; the sequence state space network fuses multiple single-frame data descriptors, and then uses the GeM pooling layer to aggregate them to generate a global descriptor with a dimension of 256;

[0014] Both the multimodal state space network and the sequence state space network are mainly composed of multiple basic state space models.

[0015] Furthermore, the ECA backbone network is a five-layer convolutional neural network that uses a channel attention mechanism to fuse information from different channels; the multimodal state space network uses a cross-scanning method to realize the global receptive field in the multimodal sequence data. The multimodal sequence data is scanned in opposite directions starting from the top left, bottom left, top right, and bottom right corners to ensure that each pixel can make full use of the neighborhood information. Then, the different sequences are merged into the original size to obtain different sequences; each sequence is processed by the basic state space model.

[0016] Furthermore, the input of the basic state space model is divided into two parts; one part passes through a Linear layer and convolution before entering the SS2D module, while activating the SinLU function; the other part passes through a Linear layer while activating the SinLU function; the outputs of the two branches are merged, passed through a Linear layer, and then merged with the original input before being input into the VLAD layer module.

[0017] Furthermore, after obtaining multiple single-frame descriptors, the sequence module sequentially connects the multiple single-frame descriptors and combines and arranges the connection order; the connection descriptors of various combination orders are respectively input into the basic state space model of a single layer, and the results of multiple arrangements are fused by adding them according to weights, and input into the GeM pooling layer to generate a sequence global descriptor.

[0018] Furthermore, the joint encoding specifically refers to:

[0019] Align and correlate visual images and laser point clouds at the data level;

[0020] First, the laser point cloud P = {x, y, z} is projected from the lidar coordinate system onto the visual image plane I = {u, v}, as shown in the following formula:

[0021]

[0022] The homogeneous representation of laser point cloud coordinates is X = [x, y, z, 1]. T The homogeneous representation of the visual image coordinates is Y = [u, v, 1]. T M1 and M2 are the camera intrinsic parameter matrix and the lidar extrinsic parameter matrix from the camera, respectively. Z c This represents the z-coordinate of a point in the camera coordinate system.

[0023] After projection, each projection point is assigned corresponding RGB color information, and the laser point cloud format is converted to P'={x,y,z,I,r,g,b}, where I represents the reflection intensity;

[0024] A polar coordinate system with the lidar sensor as the origin is used, and the 3D space is divided into several sectors according to the set horizontal and vertical angular resolutions. Each sector contains a set of format-converted lidar point clouds. Among these lidar point clouds, the lidar point with the largest distance from the lidar sensor is selected as the representative point. The representative point represents the maximum detectable range of the lidar sensor in that direction, defined as follows:

[0025]

[0026] Among them, P x P y P zLet be the Cartesian coordinates of the points, L2 represent the L2 norm, Max represent the maximum value, Φ() represent the point corresponding to the maximum value, and n represent the number of points within the sector. The multidimensional information contained in the points is encoded into a two-dimensional image according to the sector arrangement order, resulting in multimodal sequence data, called SMRD, defined as follows:

[0027] SMRD(k)=P′ ij (k)∈Ri,j∈(V s H s ),

[0028] k∈(P′ I , P′ R , P′ r , P′ g , P′ b ).

[0029] Among them, V s and H s P′ refers to the number of rows and columns used to divide a sector. I , P′ R , P′ r , P′ g , P′ b These refer to the reflection intensity at each point, the distance from the point to the sensor, and the data domain of the RGB information, respectively.

[0030] The SMRD is a five-channel two-dimensional structured data containing information on reflection intensity I, range R, and color (r, g, b).

[0031] The beneficial effects of this invention are as follows: Addressing the inherent limitations of single-modality scene recognition in current scene recognition tasks and the inability to effectively utilize the spatiotemporal features of sequential data, this invention proposes a sequential multimodal scene recognition method based on a state-space model, achieving highly robust scene recognition. First, it proposes fusing and encoding laser point cloud and image data, increasing data dimensionality and compressing data complexity. Then, a global descriptor is obtained through a single-frame module and a sequence module based on the state-space model. The cross-scan design in the single-frame module improves computational efficiency, while the sequential full-combination representation strategy in the sequence module solves the problem of position recognition for trajectory changes within the scene. This invention, with its computational efficiency, high accuracy, and robustness, can be effectively used for real-time global point cloud scene recognition tasks. Attached Figure Description

[0032] Figure 1 This is a flowchart of the method of the present invention;

[0033] Figure 2 Encode the network flowchart for the global descriptor;

[0034] Figure 3A schematic diagram of the basic state-space model. Detailed Implementation

[0035] Figure 1 This is a flowchart of the overall technical solution of the present invention. The present invention proposes a sequential multimodal scene recognition method based on a state-space model. First, the laser point cloud and visual image in the trajectory are processed into unique global descriptors through a proposed global descriptor encoding network. The global descriptors of the two trajectories serve as the map and the query, respectively. Then, during the query process, the nearest neighbor search algorithm is used to find the most similar data in the map, thus achieving scene recognition.

[0036] Multimodal data joint coding:

[0037] Panoramic images and full-scene point clouds exhibit inherent alignment at the data level, and the two modalities are highly correlated. Data feature-level fusion is crucial for multimodal scene recognition. By aligning and associating visual and point cloud data at the data level, the dimensionality of the data can be increased, thereby enriching the location description.

[0038] First, the point cloud (P = {x, y, z}) is projected from the LiDAR coordinate system onto the image plane (I = {u, v}), as shown in the following formula:

[0039]

[0040] The homogeneous representations of point cloud and image coordinates are represented as X = [x, y, z, 1], respectively. T And Y = [u,v,1] T M1 and M2 are the camera intrinsic parameter matrix and the lidar extrinsic parameter matrix from the camera, respectively. Z c This represents the z-coordinate of a point in the camera coordinate system. After projection, each point is assigned corresponding RGB color information, and the laser point cloud format is converted to P'={x,y,z,i,r,g,b}, where i represents the reflection intensity.

[0041] LiDAR sensors typically have a horizontal field of view of 360 degrees and a vertical field of view typically between 30 and 60 degrees. Using a polar coordinate system with the LiDAR sensor as the origin, the 3D space is divided into several sectors according to the set horizontal and vertical angular resolutions. Each sector contains a set of format-converted LiDAR point clouds. From these format-converted LiDAR point clouds, the point with the largest distance from the LiDAR sensor is selected as the representative point. This representative point represents the maximum detectable range of the LiDAR sensor in that direction and has geographic resolution significance. The definition is as follows:

[0042]

[0043] Among them, P x Py P z Let be the Cartesian coordinates of the points, L2 represent the L2 norm, Max represent the maximum value, Φ() represent the point corresponding to the maximum value, and n represent the number of points within the sector. Encoding the multidimensional information contained in the points into a two-dimensional image according to the sector arrangement yields the multimodal fusion coding, called SMRD, defined as follows:

[0044] SMRD(k)=P′ ij (k)∈R i,j∈(V s H s ),

[0045] k∈(P′ I , P′ R , P′ r , P′ g , P′ b ).

[0046] It is a five-channel two-dimensional structured data containing intensity I, range R, and color (r, g, b) information.

[0047] Global descriptor encoding networks such as Figure 2 As shown, the input to the global descriptor encoding network is multimodal sequence data. After processing by the single-frame module and the sequence module, the global descriptor encoding network outputs a global descriptor. In the single-frame module, each set of multimodal sequence data is first processed by the ECA backbone network, which performs channel attention fusion on the five-channel multimodal data to obtain the main features. Subsequently, the multimodal state space network selectively learns deeper features and uses VLAD aggregation to obtain the descriptor of the single-frame data. Then, the sequence state space network fuses multiple single-frame data descriptors. Finally, a GeM pooling layer is used to aggregate features and generate a global descriptor with a dimension of 256.

[0048] The single-frame module is specifically as follows:

[0049] The ECA backbone is a five-layer convolutional neural network that uses a channel attention mechanism to fuse information from different channels. The multimodal state-space network employs a cross-scanning approach to achieve a global receptive field in a two-dimensional image, scanning in opposite directions starting from the top-left, bottom-left, top-right, and bottom-right corners. This ensures that each pixel fully utilizes its neighborhood information, and then the different sequences are merged back to the original size. Each sequence is processed by the underlying state-space model.

[0050] Basic state-space model such as Figure 3As shown, one part passes through a Linear layer and convolution before entering the SS2D module, simultaneously activating the SinLU function; another part passes through a Linear layer, also activating the SinLU function. The outputs of the two branches are merged and added to the input in a manner similar to a cross-layer connection. This combination constitutes the basic state-space model. Downsampling is performed on multiple stacked layers. The extracted effective features are then aggregated by VLAD layers into simple single-frame descriptors.

[0051] The sequence module works as follows: after obtaining multiple single-frame descriptors, they are fused to generate a sequence global descriptor. Unlike traditional sequence-based methods, which rely on the invariant order of the sequence, our goal is to avoid restricting the entry into the same scene to a fixed trajectory, as a scene typically involves multiple different travel trajectories, including reverse travel or rotation. Taking the fusion of three single-frame descriptors as an example, we list their full sequential combinations, totaling six. Each permutation is processed using multiple state-space network modules in a manner similar to single-frame cross-scanning, and the features are then fused into unique sequence global features, which are then aggregated into a global descriptor through a GeM pooling layer.

[0052] The global descriptor encoding network proposed in this invention is not only computationally efficient and capable of utilizing features from different modalities, but also effectively leverages the spatiotemporal features of sequence data to enhance scene recognition. The proposed global descriptor encoding network can be effectively applied to scene recognition tasks, ensuring both high accuracy and real-time performance.

[0053] To verify that our method achieves a balance between real-time performance and accuracy, we tested it on two datasets: NCLT and the self-recorded NEU-dataset. The experimental platform was equipped with a Core i7-10700 CPU@2.9GHz and an Nvidia RTX 3090 GPU. The AR@1% (average recall) on the two datasets were 92.2% and 99.0%, respectively. This demonstrates that our method maintains good performance on datasets with different types of laser sensors, ensuring both scene recognition accuracy and good real-time performance, making it effective for large-scale scene recognition tasks.

Claims

1. A sequence multimodal scene recognition method based on a state-space model, characterized in that, The laser point cloud and visual image in the trajectory are jointly encoded to form multimodal sequence data, which are then processed into unique global descriptors through a global descriptor encoding network. The global descriptors of the two trajectories are used as the map and the query, respectively. During the query process, the nearest neighbor search algorithm is used to find the most similar data in the map to complete scene recognition; the global descriptor encoding network includes a single-frame module and a sequence module. The multimodal sequence data is input into the global descriptor encoding network, and after passing through the single-frame module and the sequence module in sequence, a global descriptor is obtained. The single-frame module includes an ECA backbone network, a multimodal state space network, and a VLAD aggregation network. Each set of multimodal sequence data is first processed by the ECA backbone network, which performs channel attention fusion on the five-channel multimodal data to obtain the main features. The multimodal state space network further selectively learns deep features and uses the VLAD aggregation network to obtain the single-frame data descriptor. The sequence module includes a sequence state space network and a pooling layer; the sequence state space network fuses multiple single-frame data descriptors, and then uses the GeM pooling layer to aggregate them to generate a global descriptor with a dimension of 256; Both the multimodal state space network and the sequence state space network are mainly composed of multiple basic state space models. The ECA backbone network is a five-layer convolutional neural network that uses a channel attention mechanism to fuse information from different channels. The multimodal state space network uses a cross-scanning method to achieve a global receptive field in the multimodal sequence data. The multimodal sequence data is scanned in opposite directions starting from the top left, bottom left, top right, and bottom right corners to ensure that each pixel can fully utilize neighborhood information. Then, the different sequences are merged back to the original size to obtain different sequences. Each sequence is processed by a basic state space model. The input of the basic state space model is divided into two parts: one part passes through a linear layer and convolution before entering the SS2D module, while activating the SinLU function; the other part passes through a linear layer while activating the SinLU function. The outputs of the two branches are merged, passed through a linear layer, and then merged with the original input before being input to VLAD. The layer module; after obtaining multiple single-frame descriptors, the sequence module sequentially connects the multiple single-frame descriptors and combines and arranges the connection order; the connection descriptors of various combination orders are respectively input into the basic state space model of the single layer, and the results of multiple arrangements are fused by weighted addition, and input into the GeM pooling layer to generate a sequence global descriptor; the joint encoding specifically includes: Align and correlate visual images and laser point clouds at the data level; First, project the laser point cloud P = {x, y, z} from the lidar coordinate system onto the visual image plane I = {u, v}, as shown in the following formula: ； The homogeneous representation of the laser point cloud coordinates is X = [x, y, z, 1]. T The homogeneous representation of the visual image coordinates is Y = [u, v, 1]. T ; M1 and M2 are the camera intrinsic parameter matrix and the lidar-to-camera extrinsic parameter matrix, respectively. Z c This represents the z-coordinate of a point in the camera coordinate system. After projection, each projection point is assigned corresponding RGB color information, and the laser point cloud format is converted to P' = {x,y,z,I,r,g,b}, where I represents the reflection intensity; A polar coordinate system with the lidar sensor as the origin is used, and the 3D space is divided into several sectors according to the set horizontal and vertical angular resolutions. Each sector contains a set of format-converted lidar point clouds. Among these lidar point clouds, the lidar point with the largest distance from the lidar sensor is selected as the representative point. The representative point represents the maximum detectable range of the lidar sensor in that direction, defined as follows: ； Among them, P x P y P z Let be the Cartesian coordinates of the points, L2 represent the L2 norm, Max represent the maximum value, Φ() represent the point corresponding to the maximum value, and n represent the number of points within the sector. The multidimensional information contained in the points is encoded into a two-dimensional image according to the sector arrangement order, resulting in multimodal sequence data, called SMRD, defined as follows: ； Among them, V s and H s This refers to the number of rows and columns used to divide a sector. These refer to the reflection intensity at each point, the distance from the point to the sensor, and the data domain of the RGB information, respectively. The SMRD is a five-channel two-dimensional structured data containing reflection intensity I, range D, and color RGB information.

Images