Hyperspectral image and laser radar image fusion classification method based on state space model

Abstract

The invention discloses a hyperspectral image and laser radar image fusion classification method based on a state space model, and belongs to the technical field of remote sensing image processing and application. Firstly, a spectrograph and a laser radar are used for collecting a hyperspectral image and a laser radar image of the same area; secondly, local features of the hyperspectral image and the laser radar image are extracted by using a convolutional neural network, and global features are extracted by using a visual state space model; thirdly, the features are input into the designed state and feature dual fusion neural network after being sorted, fused features are obtained, and a multi-layer perceptron is used for classifying the fused features to obtain a classification result. According to the hyperspectral and laser radar image fusion classification method, the problems of low accuracy and poor generalization ability of an existing fusion classification method can be solved, high-accuracy and high-generalization-ability hyperspectral and laser radar image fusion classification is realized, and an effective solution is provided for hyperspectral and laser radar image classification so as to improve the reliability and accuracy of the hyperspectral and laser radar image classification in practical application.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image processing and application, and relates to a multimodal remote sensing image fusion classification method, and in particular to a hyperspectral image and laser radar image fusion classification method based on a state space model. Background Art

[0002] Remote sensing image classification is an important research branch in the field of remote sensing image processing and application technology. With the development of imaging technology and earth observation technology, the modalities of remote sensing images in the same geographical area are becoming increasingly rich. Hyperspectral images contain reflection information collected from visible light to near-infrared bands by passive imaging, and lidar images contain elevation information obtained from microwave bands by active imaging. Multimodal information fusion analysis of hyperspectral images and lidar images is performed, and the complementarity of information is used to significantly improve the accuracy of object classification, thereby further improving the efficiency of earth observation. People have conducted a lot of research on the fusion classification methods of hyperspectral images and lidar images, and have achieved remarkable results:

[0003] Li Jiaxin, Hong Danfeng, Gao Lianru et al. (Reference: LI J, HONG D, GAO L, et al. Deep learning in multimodal remote sensing data fusion: A comprehensive review [J / OL]. International Journal of Applied Earth Observation and Geoinformation, 2022, 112: 102926.) reviewed the research background and significance of hyperspectral image and lidar image fusion classification in the article, and introduced the existing methods, existing problems and vision for future development.

[0004] Yao Dong et al. (China invention patent, application number: CN202410998576.5) invented a method for detecting and analyzing hyperspectral and lidar images based on noise filtering and spectral distribution information, which solved the problem that the existing technology could not efficiently detect spectral echoes and accurately analyze spectra, thereby achieving accurate detection and identification of targets. However, its algorithm relies on manual feature extraction, and its adaptability and portability are poor.

[0005] Wang Wenzhen et al. (Chinese invention patent, patent application number: CN202410717318.5) invented a classification method based on convolutional neural network and prototype enhancement, which solved the problem of data differences from different sources, and thus greatly improved the accuracy of ground object classification in complex scenes. However, its model structure is highly linear and overly relies on data similarity within the same data set for classification, and the generalization ability of the model needs to be improved. Meng Xiangchao et al. (Chinese invention patent, patent application number: CN202410854745.8) invented a category-aware fusion network hyperspectral and lidar joint classification method based on an attention model, which solved the problem that the existing technology failed to fully utilize the features of hyperspectral images and lidar images for mining uncertain categories, and achieved targeted information aggregation, thereby improving the classification accuracy of the overall model. However, its adaptability to special environments is low, and it is difficult to cope with classification needs in complex weather.

[0006] In summary, it is urgent to develop a deep learning model with wide coverage, good adaptability and the ability to represent highly nonlinear relationships to effectively extract, represent and fuse the global and local information contained in hyperspectral and lidar images, so as to meet the application scenarios of accurate joint classification of hyperspectral images and lidar images in different environments. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the prior art and propose a hyperspectral image and laser radar image fusion classification method based on a state space model to solve the problem that the existing fusion classification technology is difficult to effectively fuse global and local information, resulting in poor model generalization ability. The state space model includes a visual selection state space model (Vision Mamba, ViM), a selection state space model (Mamba) and a state and feature dual fusion neural network proposed by the present invention. The method uses a visual selection state space model to extract image features to solve the problem that the existing method cannot effectively and deeply mine hyperspectral and laser radar information; and uses a selection state space model and a state and feature dual fusion neural network to efficiently fuse hyperspectral and laser radar information to solve the problem of poor model generalization ability.

[0008] In order to achieve the above object, the technical solution adopted by the present invention is:

[0009] A hyperspectral image and laser radar image fusion classification method based on a state space model. The hyperspectral image and laser radar image fusion classification method mainly solves the two technical problems of being unable to effectively extract features and being unable to effectively fuse features in the fusion classification of hyperspectral images and laser radar images, and proposes a technical solution that uses a state space model to effectively solve these two technical problems. The hyperspectral image and laser radar image fusion classification method first uses a spectrometer and a laser radar to collect hyperspectral images H and laser radar images L of the same area. Secondly, the hyperspectral image and the laser radar image are respectively extracted using a convolutional neural network to obtain local features H1 and L1, and the hyperspectral image and the laser radar image are respectively extracted using a visual state space model to obtain global features H2 and L2. Thirdly, the previously obtained features are sorted according to H1, L1, H2, and L2, and then input into the state and feature dual fusion neural network designed by the present invention to obtain the fused feature Z. Finally, the fused feature Z is classified using a multilayer perceptron to obtain a classification result. Specifically, the following steps are included:

[0010] Step 1, extracting features from hyperspectral images and lidar images;

[0011] The hyperspectral image H is obtained by observing the region G with a spectrometer. The lidar image L is obtained by observing the region G with a lidar.

[0012] Step 1.1, global feature extraction based on visual selection state space model;

[0013] The visual selection state space model is used to extract global features from hyperspectral images and lidar respectively, and the hyperspectral global feature vector sequence h1,h2,…,h is obtained. N and the lidar global feature vector sequence l1,l2,…,l N .

[0014] The specific steps of the visual selection state space model are as follows: First, the image H is cut and classified into a number of hyperspectral image blocks k 11 ,k 12 ,…,k NN , Secondly, all the hyperspectral image blocks are arranged from top to bottom and from left to right into a vector sequence a1, a2, …, a N . Third, a1, a2, …, a N Two identical visual selection state-space models are input and their outputs are added together to obtain their global feature vector sequence.

[0015] Step 1.2, local feature extraction based on convolutional neural network;

[0016] Convolutional neural networks are used to extract local features from hyperspectral images and lidar images respectively.

[0017] For hyperspectral images, a 3D convolutional neural network is first used to extract the spatial and spectral local features of the hyperspectral image. Secondly, a 2D convolution is used to further extract the local spatial information. Finally, the convolution output is flattened in the spatial dimension to obtain a hyperspectral local feature vector sequence p1, p2, …, p containing spatial and spectral information. N .

[0018] For the lidar image, first, a 2D convolutional neural network is used to extract local spatial information; secondly, another 2D convolution is used to further extract local spatial information. Finally, the convolution output is flattened in the spatial dimension to obtain the lidar local feature vector sequence q1, q2, …, q N .

[0019] Step 2: local-global bidirectional feature sorting;

[0020] For the hyperspectral global feature vector sequence h1,h2,…,h obtained in step 1 N 、LiDAR global feature vector sequence l1,l2,…,l N , hyperspectral local feature vector sequence p1,p2,…,p N And the laser radar local feature vector sequence q1,q2,…,q N , the bottom-up vector sequence F is obtained by arranging them in order from local to global, and the top-down vector sequence B is obtained by arranging them in order from global to local, where the order of hyperspectral and lidar is arbitrary.

[0021] Step 3: dual fusion neural network of state and feature;

[0022] The present invention proposes a new state and feature dual fusion neural network ( Figure 1 ), three selected state space models, denoted as Mamba1, Mamba2, Mamba and the state feature dual fusion module StateFusion ( Figure 2 ). The specific steps of the state and feature dual fusion neural network are as follows:

[0023] Step 3.1 Bidirectional feature analysis;

[0024] The bottom-up vector sequence F and the top-down vector sequence B obtained in step 2 are input into the two selection state space models Mamba1 and Mamba2 respectively to obtain the output features F1 and B1. The final states s1 and s2 of the two selection state space models Mamba1 and Mamba2 are also used as outputs. That is:

[0025] F1,s1=Mamba1(F) (3.1)

[0026] B1,s2=Mamba2(B) (3.2)

[0027] Step 3.2 state fusion;

[0028] Input the final states s1 and s2 of the two selection state space models Mamba1 and Mamba2 obtained in step 3.1 into the state feature dual fusion module StateFusion to obtain the bidirectional fusion state s. That is:

[0029] s t =A1s1+A2s2 (3.3)

[0030] s=Cs t +D (3.4)

[0031] Among them, s t is the intermediate state after preliminary fusion, s is the final fusion state, A1, A2, C, D are all learnable parameters, A1 is the transformation matrix of state s1, A2 is the transformation matrix of state s2, C is the intermediate state s t The transformation matrix, D is the intermediate state s t The offset vector.

[0032] Step 3.3 The fusion state is used as the initial state of the selected state space model;

[0033] The bidirectional fusion state s obtained in step 3.2 is used as the initial state s0 of the selection state space model Mamba, and the output features F1 and B1 obtained in step 3.1 are concatenated and input into the selection state space model Mamba to obtain the fusion feature vector sequence Z. That is:

[0034]

[0035] Step 4, classifying the fused feature vector sequence Z obtained in step 3;

[0036] Use the multi-layer perceptron as the classifier, select the fused feature vector sequence Z output by the state space model Mamba in step 3.3, take the last vector for multi-layer perceptron calculation, and obtain the classification vector y. Take the dimension number of the maximum component in the classification vector as the classification result C.

[0037] Beneficial effects of the present invention:

[0038] First, the present invention utilizes a visual selection state space model and a convolutional neural network, and through step 1, a global feature extraction method of the visual selection state space model and a local feature extraction method of the convolutional neural network, effectively extracts, characterizes and fuses the global and local information contained in the hyperspectral image and the lidar, thereby solving the problem that features cannot be effectively extracted in the prior art.

[0039] Second, the present invention designs a feature vector sequence sorting method. In step 2, the hyperspectral local feature vector sequence, the hyperspectral global feature vector sequence, the lidar local feature vector sequence, and the lidar global feature vector sequence are sorted from top to bottom and from bottom to top, so as to maximize the analytical ability of the selected state space model.

[0040] Third, the present invention designs a state and feature dual fusion neural network to efficiently fuse global and local features, thereby improving the classification accuracy, and provides a deep learning fusion classification model with wide coverage, good adaptability and the ability to characterize highly nonlinear relationships.

[0041] In summary, the present invention can achieve high-accuracy classification tasks for the collected hyperspectral images and lidar images, and provides an effective solution for achieving high-precision ground object observation, thereby meeting the application scenarios of accurate joint classification of hyperspectral images and lidar images in different environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of the present invention.

[0043] Figure 2 Schematic diagram of the state fusion mechanism designed for the present invention.

[0044] Figure 3 It is a true category diagram and a model classification result diagram of the method of the present invention; Figure 3 (a) in the figure is the true category of the test dataset; Figure 3 (b) in the figure is the model classification result. DETAILED DESCRIPTION

[0045] The present invention is further described below in conjunction with specific implementation cases.

[0046] A hyperspectral image and lidar image fusion classification method based on a state space model, wherein the task of the state space model is to classify ground objects at the pixel level according to the hyperspectral image and the lidar image, specifically comprising the following steps:

[0047] Step 1, extract features from the hyperspectral image H and the lidar image L;

[0048] The hyperspectral image H is obtained by observing the region G with a spectrometer. The lidar image L is obtained by observing the region G with a lidar.

[0049] The hyperspectral image and lidar image of the same area G collected by the spectrometer and lidar are denoted as H and L respectively. The hyperspectral image and lidar image were collected at the University of Houston and its surrounding areas. The spectral coverage of the hyperspectral image is 380nm-1050nm, with a total of 144 bands. The spatial resolution of the hyperspectral image and lidar is 2.5 meters. The feature labeling information contains label information with the same resolution as H and L, which is manually collected and denoted as Y. The labels contain 15 categories of features, including healthy grassland, stressed grassland, synthetic grassland, trees, soil, water, residential areas, commercial areas, roads, highways, railways, first-class parking lots, second-class parking lots, stadiums and runways.

[0050] Step 1.1 Global feature extraction based on state space model;

[0051] The state space model is a visual selection state space model, and the visual selection state space model used is the same deep learning model as the reference (reference: Zhu L, Liao B, Zhang Q, et al. Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model [A / OL]. arXiv, 2024 [2024-01-19]. http: / / arxiv.org / abs / 2401.09417). The specific steps are as follows:

[0052] Step 1.1.1: Use a sliding window to cut the hyperspectral image H to obtain a hyperspectral image block k with a shape of c×w×w (where c represents the number of channels and w represents the width). 11 ,k 12 ,…,k NN The sliding window is centered at the training sample pixel position and has a square shape with a side length of 3.

[0053] Step 1.1.2: Cut each hyperspectral image block k obtained by sliding the window 11 ,k 12 ,…,k NN Sort by window sliding order from top to bottom and from left to right, i.e. k 11 ,k 12 ,…,k NN , flattened to a c×l shape in the image dimension, where l=w×w.

[0054] Step 1.1.3, input the feature sequence into two identical visual selection state space models to analyze the relationship between their features. The state dimension of the visual selection state space model is 32. Two feature sequences with the same tensor dimension as the input sequence are obtained.

[0055] Step 1.1.4, the outputs of the visual selection state space model in the two directions are weightedly added through the primary features to obtain a hyperspectral global feature vector sequence h1,h2,…,h with a shape of c1×l1 N .

[0056] Step 1.1.5: Perform steps 1.1.1 to 1.1.4 on the lidar image to obtain a sequence of lidar global feature vectors l1, l2, ..., l with a shape of c1 × l1. N .

[0057] Step 1.2, local feature extraction of hyperspectral image and lidar image;

[0058] In step 1.2.1, the hyperspectral image H is passed through a 3D convolutional layer of size 6 × 3 × 3, and then through a normalization layer and a linear rectification activation layer to obtain the primary spatial-spectral local texture features of the hyperspectral image.

[0059] Step 1.2.2, the primary spatial-spectral local texture features are passed through a 2D convolutional layer of size 3×3, and then through a normalization layer to obtain a tensor of shape c1×w1×w1, and finally flattened in the spatial dimension to c1×l1 to obtain a hyperspectral local feature vector sequence p1,p2,…,p N .

[0060] In step 1.2.3, the lidar image L sample passes through a two-dimensional convolution layer of size 3×3 and then through a normalization layer to obtain the primary local texture features.

[0061] Step 1.2.4, the primary local texture features are passed through a 3×3 two-dimensional convolutional layer again, and then through a normalization layer to obtain a tensor of shape c1×w1×w1, and finally flattened to c1×l1 in the spatial dimension to obtain the lidar local feature vector sequence q1,q2,…,q N .

[0062] Step 2: local-global bidirectional feature sorting;

[0063] For the hyperspectral global feature vector sequence h1,h2,…,h obtained in step 1 N 、LiDAR global feature vector sequence l1,l2,…,l N , hyperspectral local feature vector sequence p1,p2,…,p NAnd the laser radar local feature vector sequence q1,q2,…,q N , the bottom-up vector sequence F is obtained by arranging from local to global, and the top-down vector sequence B is obtained by arranging from global to local. The order of hyperspectral and lidar is arbitrary, such as:

[0064] F=p1,p2,…,p N ,q1,q2,…,q N ,h1,h2,…,h N ,l1,l2,…,l N (2.1)

[0065] B=l1,l2,…,l N ,h1,h2,…,h N ,q1,q2,…,q N ,p1,p2,…,p N (2.2)

[0066] Step 3: dual fusion neural network of state and feature;

[0067] The present invention proposes a new state and feature dual fusion neural network, which is composed of three state fusion modules designed by the state space model. Figure 2 The three selection state space models are denoted as Mamba1, Mamba2, and Mamba.

[0068] Step 3.1, input the bottom-up vector sequence F and the top-down vector sequence B obtained in step 2 into the two selection state space models Mamba1 and Mamba2 respectively, and the final states s1 and s2 of the two selection state space models Mamba1 and Mamba2 are also used as outputs, as shown in formulas (3.1) and (3.2).

[0069] The definition of the selective state space model (reference: Gu A, Dao T. Mamba: Linear-Time Sequence Modeling with Selective State Spaces [A / OL]) is:

[0070] s(t)=As(t-1)+Bx(t)(3.6)

[0071] y(t)=Cs(t)+D(3.7)

[0072] Among them, A and D are learnable parameters, B and C are parameters generated by the parameter generation network (composed of one-dimensional convolution and linear layers) according to the input, s(t) is the state sequence, t is the sequence number of the feature sequence and the state sequence, x(t) is the input feature vector sequence, and y(t) is the output feature vector sequence. The state vector dimension parameter of the state space model is selected as 32.

[0073] Step 3.2, input the final states s1 and s2 of the two selection state space models Mamba1 and Mamba2 obtained in step 3.1 into the state fusion module StateFusion to obtain the bidirectional fusion state s. As shown in formula 3.3, the detailed state fusion formula is shown in formulas (3.1) and (3.2).

[0074] Step 3.3, the bidirectional fusion state s obtained in step 3.2 is used as the initial state s0 of the selection state space model Mamba, and the output features F1 and B1 obtained in step 3.1 are concatenated and input into the selection state space model Mamba to obtain the fused feature vector sequence Z. As shown in formula (3.5).

[0075] The state vector dimension parameter of the selected state space model Mamba is 32.

[0076] Step 3.4 classifies the fused feature vector sequence Z obtained in step 3;

[0077] Step 3.4.1 takes the last one-dimensional vector y0 of the fused feature vector sequence Z with a shape of c2×l2.

[0078] Step 3.4.1 uses a multilayer perceptron as a classifier to calculate y0 and obtain the classification vector y.

[0079] Step 3.4.2 obtains the classification result C by calculating the dimension number of the maximum component in the classification vector.

[0080] C = argmax(y)(3.8)

[0081] Among them, argmax(y) represents all components of vector y y1,y2,…,y N The dimension number of the largest component in .

[0082] The following is a simulation experiment to illustrate the technical effect of the embodiment of the present invention:

[0083] First, simulation conditions;

[0084] The experimental settings of the present invention and the comparative method are: a desktop computer with an Intel Core i5-12400F central processor, 64GB memory, and a GeForce RTX 3090 graphics processor. The software environment is the PyTorch deep learning framework. The number of training cycles of the model is 100, the model optimization algorithm is the Adam algorithm, the learning rate is 0.01, the data set is the 2013 hyperspectral and lidar data set of the University of Houston, and the division of the training set and the test set uses the default standard division method of the data set, that is, a small number of samples specified in the data set without special cloud and fog occlusion environment are selected as the training set, and the rest are used as the test set.

[0085] The comparison method is specifically: a currently widely studied and used attention-based deep neural network method (reference: WANG X, FENG Y, SONG R, et al. Multi-attentive hierarchical dense fusion net for fusion classification of hyperspectral and LiDAR data [J / OL]. Information Fusion, 2022, 82: 1-18. DOI: 10.1016 / j.inffus.2021.12.008 .), this method first uses a convolutional neural network to extract spectral and spatial features from the hyperspectral image, and also uses a convolutional neural network to extract lidar features. The extracted features are then put into three attention modules, and finally the outputs of the three attention modules are concatenated and classified using a multi-layer perceptron.

[0086] Second, the simulation results are shown in Table 1.

[0087] Table 1, simulation results

[0088]

[0089]

[0090] According to the simulation result data in Table 1, the classification results of almost all categories of the present invention are better than those of the control method, and the two categories with the largest lead are highways blocked by clouds and fog and a type of parking lot. This shows that the present invention has a significant effect in improving special weather and improving the adaptability and discrimination ability of the model. The above-mentioned embodiments only express the implementation methods of the present invention, but they cannot be understood as limiting the scope of the patent of the present invention. It should be pointed out that for those skilled in the art, without departing from the concept of the present invention, several variations and improvements can be made, which all belong to the protection scope of the present invention.

Claims

1. A hyperspectral image and lidar image fusion classification method based on a state space model, characterized in that: The hyperspectral image and laser radar image fusion classification method first uses a spectrometer and a laser radar to collect a hyperspectral image H and a laser radar image L of the same area; secondly, a convolutional neural network is used to extract local features of the hyperspectral image and the laser radar image to obtain H1 and L1, and a visual state space model is used to extract global features of the hyperspectral image and the laser radar image to obtain H2 and L2; thirdly, the features obtained above are sorted as H1, L1, H2, and L2, and then input into the state and feature dual fusion neural network designed by the present invention to obtain a fused feature Z; finally, a multi-layer perceptron is used to classify the fused feature Z to obtain a classification result.

2. The hyperspectral image and laser radar image fusion classification method based on state space model according to claim 1 is characterized in that: The hyperspectral image and lidar image fusion classification method specifically includes the following steps: Step 1, extracting features from hyperspectral images and lidar images; A hyperspectral image H is obtained by observing the region G with a spectrometer; a lidar image L is obtained by observing the region G with a lidar; Step 1.1, global feature extraction based on visual selection state space model; The visual selection state space model is used to extract global features from hyperspectral images and lidar respectively, and the hyperspectral global feature vector sequence h1,h2,…,h is obtained. N and the lidar global feature vector sequence l1,l2,…,l N ; Step 1.2, local feature extraction based on convolutional neural network; Use convolutional neural networks to extract local features from hyperspectral images and lidar images respectively; Step 2: local-global bidirectional feature sorting; For the hyperspectral global feature vector sequence h1,h2,…,h obtained in step 1 N 、LiDAR global feature vector sequence l1,l2,…,l N , hyperspectral local feature vector sequence p1,p2,…,p N And the laser radar local feature vector sequence q1,q2,…,q N , the bottom-up vector sequence F is obtained by arranging them in order from local to global, and the top-down vector sequence B is obtained by arranging them in order from global to local, where the order of hyperspectral and lidar is arbitrary; Step 3: dual fusion neural network of state and feature; Design a state and feature dual fusion neural network, which consists of three selected state space models, denoted as Mamba1, Mamba2, Mamba and the designed state feature dual fusion module StateFusion; Step 4, classifying the fused feature vector sequence Z obtained in step 3; Use a multi-layer perceptron as a classifier, select the fused feature vector sequence Z output by the state-space model Mamba in step 3.3, take the last vector for multi-layer perceptron calculation, and obtain the classification vector y; take the dimension number of the maximum component in the classification vector as the classification result C.

3. The method for fusion classification of hyperspectral image and laser radar image based on state space model according to claim 2 is characterized in that: In step 1.1, the specific steps of the visual selection state space model are as follows: first, the image H is cut and classified into a number of hyperspectral image blocks k 11 ,k 12 ,…,k NN , Secondly, all the hyperspectral image blocks are arranged from top to bottom and from left to right into a vector sequence a1, a2, …, a N ; Finally, a1, a2, …, a N Two identical visual selection state-space models are input and their outputs are added together to obtain their global feature vector sequence.

4. The hyperspectral image and laser radar image fusion classification method based on state space model according to claim 2 is characterized in that: In step 1.2, the specific steps of local feature extraction are: For hyperspectral images, a 3D convolutional neural network is first used to extract the spatial and spectral local features of the hyperspectral image. Secondly, a 2D convolution is used to further extract the local spatial information. Finally, the convolution output is flattened in the spatial dimension to obtain a hyperspectral local feature vector sequence p1, p2, …, p containing spatial and spectral information. N ; For the lidar image, first, a 2D convolutional neural network is used to extract local spatial information; secondly, another 2D convolution is used to further extract local spatial information. Finally, the convolution output is flattened in the spatial dimension to obtain the lidar local feature vector sequence q1, q2, …, q N .

5. The method for fusion classification of hyperspectral image and laser radar image based on state space model according to claim 2 is characterized in that: In step 3, the specific design steps of the state and feature dual fusion neural network are as follows: Step 3.1 Bidirectional feature analysis; The bottom-up vector sequence F and the top-down vector sequence B obtained in step 2 are input into the two selection state space models Mamba1 and Mamba2 respectively to obtain the output features F1 and B1; the final states s1 and s2 of the two selection state space models Mamba1 and Mamba2 are also used as outputs; that is: F1,s1=Mamba1(F) (3.1) B1,s2=Mamba2(B) (3.2) Step 3.2 state fusion; The final states s1 and s2 of the two selection state space models Mamba1 and Mamba2 obtained in step 3.1 are input into the state feature dual fusion module StateFusion to obtain the bidirectional fusion state s; that is: s t =A1s1+A2s2 (3.3) s=Cs t +D (3.4) Among them, s t is the intermediate state after preliminary fusion, s is the final fusion state, A1, A2, C, D are all learnable parameters, A1 is the transformation matrix of state s1, A2 is the transformation matrix of state s2, C is the intermediate state s t The transformation matrix, D is the intermediate state s t The offset vector. Step 3.3 The fusion state is used as the initial state of the selected state space model; The bidirectional fusion state s obtained in step 3.2 is used as the initial state s0 of the selection state space model Mamba, and the output features F1 and B1 obtained in step 3.1 are concatenated and input into the selection state space model Mamba to obtain the fusion feature vector sequence Z; that is:

Images