Multi-path signal intelligent recognition method of multi-modal feature fusion under complex and changeable environment

Abstract

The application discloses a multi-path signal intelligent recognition method for multi-modal feature fusion in a complex and changeable environment, and the method comprises the following steps: acquiring original satellite data and performing cutting and preprocessing; performing image processing on the original satellite data, generating two-dimensional image data and three-dimensional image data, and integrating the image data sets; constructing a deep learning framework for multi-modal feature fusion, which comprises a feature extraction module and a feature fusion module; training the deep learning framework by using the image data set and satellite observation values at corresponding moments; inputting real-time GNSS baseband signals converted into image data into the trained model to complete signal classification. By using the application, high-precision signal recognition can be realized under the condition of low signal-to-noise ratio or high signal path overlap. The application can be widely applied to the field of signal recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of signal recognition, and in particular to a multi-modal feature fusion multi-path signal intelligent recognition method in a complex and variable environment. BACKGROUND

[0002] Global Navigation Satellite System (GNSS) as the core of modern positioning and navigation technology, plays an important role in national defense, transportation, agriculture, communication and disaster rescue and other fields. With the rapid development of emerging technologies such as intelligent driving and unmanned aerial vehicles, the demand for high-precision GNSS positioning is further improved. However, the multi-path effect in complex environments is still one of the main obstacles to GNSS positioning accuracy.

[0003] Multi-path effect will cause the arrival time estimation of satellite signal to deviate, resulting in errors in pseudorange measurement and carrier phase measurement, thereby directly affecting the positioning accuracy of the receiver. For key application scenarios, such as autonomous vehicles navigating in urban canyons or unmanned aerial vehicles performing tasks in forest environments, traditional GNSS receiving technology is difficult to meet the requirements of centimeter-level or even millimeter-level positioning accuracy. Research on how to effectively distinguish pure direct signals from direct satellite signals contaminated by multi-path has become a key technical path to solve the multi-path problem.

[0004] Most existing GNSS multi-path signal recognition methods are based on deep learning models. Since they do not focus on satellite observation modalities and do not compare the relationship between image features in multiple dimensions, classification in complex environments with more noise is easily affected, thereby limiting the classification accuracy in complex and variable environments. SUMMARY

[0005] Therefore, in order to solve the technical problem that the existing multi-path signal recognition method does not introduce multi-modal features for complex environments, thereby the classification accuracy is not high, the present application provides a multi-modal feature fusion multi-path signal intelligent recognition method in a complex and variable environment, which comprises the following steps:

[0006] Obtain the original satellite data and perform cropping preprocessing;

[0007] Perform image processing on the original satellite data and generate two-dimensional image data and three-dimensional image data, and integrate to obtain an image dataset;

[0008] Construct a multi-modal feature fusion deep learning framework, which includes a feature extraction module and a feature fusion module;

[0009] Train the deep learning framework using the image dataset and the corresponding satellite observation values at the moment;

[0010] The real-time GNSS baseband signal is converted into image data and input into the trained model to complete signal classification.

[0011] In some embodiments, the original satellite data is acquired and preprocessed by cropping, specifically:

[0012] The data file is received, a large file is cropped into small files of one second each according to the size, and the file is named as the determined time when the satellite signal is sent.

[0013] In some embodiments, the original satellite data is processed to generate two-dimensional image data and three-dimensional image data, and the image data set is integrated by representing the signal in the form of an image, specifically including:

[0014] The original satellite data is calculated to obtain the required features from the original data, including correlation intensity, carrier characteristics and code number, and the three-dimensional image data is generated by combination;

[0015] The original satellite data is processed to convert the correlation intensity into color intensity for representation, find the code number of the maximum peak of the correlation intensity, and generate two-dimensional image data by taking a certain range of code numbers as the center;

[0016] The generated three-dimensional image data and two-dimensional image data are labeled and assigned labels to obtain the image data set.

[0017] In some embodiments, the deep learning framework is trained using the image data set and the satellite observation value at the corresponding time to generate an identification model, which specifically includes:

[0018] The constructed multi-dimensional satellite signal image data set (including three-dimensional image data and two-dimensional image data) and the satellite observation value at the corresponding time are input into the deep learning framework;

[0019] The feature extraction module is used to extract features from the three-dimensional image data and the two-dimensional image data respectively to obtain two-dimensional image features and three-dimensional image features, wherein the feature extraction module includes a convolutional neural network, a convolutional attention network and a feedforward neural network;

[0020] The two-dimensional image features, the three-dimensional image features and the satellite observation value at the corresponding time are processed based on the feature fusion module, wherein the feature fusion module includes a multi-head attention network, a multi-dimensional image feature fusion network, a self-attention block, a cross-attention block and a fully connected neural network;

[0021] The differences between the features of the pure direct signal and the satellite signal disturbed by multipath interference are learned in three dimensions and two dimensions, and the model parameters are adjusted in combination with the loss function.

[0022] Based on the above scheme, the application provides a multi-modal feature fusion multi-path signal intelligent recognition method in a complex and variable environment, which converts GNSS baseband signals into two-dimensional images and three-dimensional images, uses image processing technology to recognize and classify multi-path signals, relies on multi-dimensional image features, and signal classification can be more effectively performed through a deep learning model. In addition, through the multi-modal feature fusion technology, the application has stronger adaptability and robustness in a complex and dynamic environment, can still maintain high recognition accuracy under the condition of low signal-to-noise ratio or highly overlapped signal paths, and solves the problem that the traditional method fails in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 is a step flow chart of a multi-modal feature fusion multi-path signal intelligent recognition method in a complex and variable environment of the application;

[0024] Figure 2 is a schematic diagram of the construction process of the image data set of the application;

[0025] Figure 3 is a schematic diagram of the steps of generating a two-dimensional image data by the specific embodiment of the application;

[0026] Figure 4 is a schematic diagram of a three-dimensional image of a pure direct signal of the application;

[0027] Figure 5 is a schematic diagram of a two-dimensional image of a pure direct signal of the application;

[0028] Figure 6 is a structural schematic diagram of a deep learning framework of multi-modal feature fusion of the application;

[0029] Figure 7 is a data flow schematic diagram of a feature extraction module of a specific embodiment of the application;

[0030] Figure 8 is a structural schematic diagram of a convolutional neural network of a specific embodiment of the application;

[0031] Figure 9 is a structural schematic diagram of a convolutional attention network of a specific embodiment of the application;

[0032] Figure 10 is a data flow schematic diagram of a feature fusion module of a specific embodiment of the application;

[0033] Figure 11 is a structural schematic diagram of a multi-dimensional image feature fusion network of a specific embodiment of the application. DETAILED DESCRIPTION

[0034] With reference to the drawings and brief description of the drawings, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments of the present application. Based on the embodiments in the present application, all the other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0035] It should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings. The embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0036] It should be understood that the "system", "device", "unit" and / or "module" used in the present application is a method for distinguishing different components, elements, parts, sections or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0037] As shown in the present application and claims, unless the context clearly indicates otherwise, "one", "a", "an" and / or "the" do not refer to the singular, but also include the plural. Generally, the terms "comprise" and "include" only indicate that the steps and elements explicitly identified are included, and these steps and elements do not constitute an exclusive list, and the method or device can also include other steps or elements. The element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, product or device comprising the element.

[0038] In the description of the embodiments of the present application, "a plurality of" means two or more than two. The following terms "first", "second" are only for the purpose of description, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include one or more features.

[0039] In addition, flowcharts are used in the present application to illustrate the operations performed by the system according to the embodiments of the present application. It should be understood that the preceding or subsequent operations are not necessarily performed in sequence. On the contrary, each step can be processed in reverse order or simultaneously. At the same time, other operations can be added to these processes, or one or more steps of operation can be removed from these processes.

[0040] Reference Figure 1 The flowchart of an optional example of the multi-path signal intelligent recognition method of multi-modal feature fusion in complex and variable environment proposed by the present application can be applied to a computer device. The recognition method proposed in the present embodiment can include but is not limited to the following steps:

[0041] Step S1, obtaining original satellite data and constructing a multi-dimensional satellite signal image data set;

[0042] Step S2, constructing a deep learning framework of multi-modal feature fusion based on a feature extraction module and a feature fusion module;

[0043] Step S3, training the deep learning framework based on the multi-dimensional satellite signal image data set and the satellite observation value mode at the corresponding moment, to obtain an identification model.

[0044] In some possible embodiments, step S1 further comprises:

[0045] preprocessing the original satellite data.

[0046] For satellite signal data files received by a labsat receiver in complex and variable environments, such as urban complex overpass environments, urban canyons, urban forests, and urban tunnels, the original satellite data file in the ls3w suffix format is obtained.

[0047] Since this part of the file usually contains a long time (more than thirty minutes) of original satellite data file, the file is large, and therefore a preliminary cutting process is needed through a python script. According to the satellite receiver parameter file matched with the original satellite data file, the python script will calculate the size of one second of data that needs to be cut according to the sampling rate, sampling bit depth, and channel number of the original satellite data file collected by the receiver, and the formula is:

[0048]

[0049] wherein S is the sampling rate of the receiver collecting satellite signal data, with the unit of Hz; D is the bit depth of the receiver, i.e. the bit number of each data, the bit number for encoding, and the bit depth of a general receiver is 2 bits; C is the channel number, with the unit of bytes, and therefore the unit is divided by 8, and the formula is processed to obtain S, which is the size of one second that needs to be cut.

[0050] The result is input into the python script, and the code cuts a large file into one-second small files according to the size, and names the file as the determined time when the satellite signal is sent, i.e. the time format accurate to the second, for example, the data collected at 3:11:11 pm on October 21, the file data name of the original satellite of this second is 1021031111, which is used as an index for querying the label data matched with the satellite data according to the file name for subsequent access. Through this processing, a series of original satellite small data files in seconds are obtained for the next step of processing.

[0051] In some possible embodiments, the construction process of the multi-dimensional satellite signal image data set in step S1, with reference to Figure 2 , specifically includes:

[0052] First, the satellite baseband signal is collected, demodulated, filtered, and preprocessed for signal data such as sampling.

[0053] S1.1, three-dimensional image data generation.

[0054] The correlation strength between signals is extracted, and the formula is as follows:

[0055]

[0056] Where S r (t) is the received baseband signal processed as a small data file, S ref (t-γ) is a locally generated pseudo-random code reference signal, γ is the time delay, and T is the integration time. By sliding the time delay parameter γ, the strength of the correlation function R(γ) is calculated, and the correlation distribution of the signal is obtained.

[0057] Then the carrier characteristics of the correlation distribution need to be calculated, and the frequency and phase characteristics of the carrier can help identify the propagation path of the signal. When extracting the carrier characteristics, the Fourier transform is used to calculate the signal spectrum, and the formula is:

[0058]

[0059] Through spectral analysis, the carrier frequency f and phase information e -j2πft of the baseband signal are extracted, and combined with the time domain characteristics x(t), the carrier characteristics can be used as a dimension of the signal information.

[0060] Furthermore, the number of chips needs to be determined. GNSS signals are usually modulated by pseudo-random codes, and the number of chips directly reflects the periodic structure of the signal in the time domain. Using the periodic characteristics of the correlation function, the number of chips N chip can be accurately extracted.

[0061]

[0062] Where T signal is the sampling time of the signal, and T chip is the period of a single chip.

[0063] The three characteristics extracted above are used as the three-dimensional features of the signal to generate image data: the X-axis is the number of chips N chip, Y axis is carrier frequency f, Z axis is correlation intensity R(γ), through the peak shape diagram of baseband signal formed by the three, the basic signal shape of the approximate signal in this period of time can be basically understood, by observing the signal peak type gap between the pure direct signal and the direct signal disturbed by multipath, the difference between the characteristics of the two in the peak type is understood, through this point the unknown real signal data set can be labeled (divided into pure direct signal and direct signal disturbed by multipath), which lays the foundation for more in-depth classification processing later.

[0064] S1.2, two-dimensional image data generation, process reference Figure 3 .

[0065] The first step is to search for the peak value of the correlation intensity of the processed satellite signal sequence, to determine the chip number and carrier frequency. The second step is to convert the three-dimensional correlation intensity (Z axis) into a two-dimensional image represented by color intensity, that is, to a two-dimensional image. The third step is to adjust the search range to the chip number of the peak value of the correlation intensity, with the chip number of the peak value of the correlation intensity as the center, and the upper and lower limits as 1.5 chips. The fourth step is to perform image correlation processing to obtain a two-dimensional signal data image.

[0066] It should be noted that this process is to generate a two-dimensional image directly from satellite signal data, rather than generating a two-dimensional image from a three-dimensional image, to ensure that the two-dimensional image data has the uniqueness, independence and contrast of the original generated data.

[0067] The correlation intensity is represented as color intensity to convert the three-dimensional correlation intensity to a two-dimensional image, and the formula is:

[0068]

[0069] Where I(x,y) represents the color value of the pixel point in the two-dimensional image, R max , R min is the minimum and maximum value of the correlation intensity, used for normalization.

[0070] For the correlation intensity maximum peak value R max , adjust the search range of the chip number N chip , limit the floating range above and below the peak value, which is specifically:

[0071] N chip ∈[N chip,max -1.5,N chip,max +1.5]#(6)

[0072] At the same time, the carrier frequency f is adjusted accordingly, and the upper and lower limits of the frequency range corresponding to the peak value are taken. In this way, the main signal characteristics can be highlighted, and irrelevant noise interference can be highlighted.

[0073] S1.3, assign a label.

[0074] After obtaining the two-dimensional signal data image, the next step is to label these image data for subsequent model training. The purpose of data labeling is to divide the image into two categories: pure direct signal and direct signal disturbed by multipath interference, and assign a label to each image.

[0075] By observing the features in the two-dimensional image, it can be found that the pure direct signal and the multipath interference signal have significant differences in peak type and distribution:

[0076] Pure direct signal: when represented on a three-dimensional image, it usually appears as a single peak, with strong and symmetrical correlation intensity, high and sharp peak value, and reference Figure 4 , for a three-dimensional image of a pure direct signal; when represented on a two-dimensional image, it is a relatively concentrated circular red pattern, reference Figure 5 , for a two-dimensional image of a pure direct signal.

[0077] Multipath interference signal: when represented on a three-dimensional image, it usually appears as multiple peaks, with relatively dispersed correlation intensity distribution, low and wide peak value, and various phenomena such as Doppler shift due to reflection, refraction, etc. When represented on a two-dimensional image, it cannot be consistent with a pure direct signal. This is an important point for distinguishing between pure direct signals and satellite signals disturbed by multipath interference.

[0078] Based on this difference, the three-dimensional image and two-dimensional image obtained by collecting satellite signal data are labeled, which is used as a classification label for subsequent input into a classification model. The higher the accuracy of the label, the higher the classification accuracy of the final classification model.

[0079] In some possible embodiments, the deep learning framework in step S2 is specifically a model framework represented as Figure 6 , which specifically includes:

[0080] S2.1, build a multi-dimensional image feature extraction module based on a convolutional attention network mechanism.

[0081] For the prepared two-dimensional image data and three-dimensional image data, they will be respectively transmitted into the feature extraction module based on the convolution attention network mechanism, for extracting the key features of this dimension and optimizing. First, the image data will pass through a convolutional neural network (CNN) containing residual connections, which gradually learns and extracts image features through multiple two-dimensional convolution layers and pooling layers. Then, the extracted feature vectors will be transmitted to the convolutional block attention module (CBAM) to further enhance and optimize important feature performance using attention mechanism. Finally, these optimized features will be input into a fully connected neural network containing residual connections to achieve more in-depth processing and understanding. Figure 7 The data processing process of the feature extraction module is shown, and the specific structure of the multi-dimensional image feature extraction module is shown.

[0082] The convolutional neural network (CNN) has a structure as shown in Figure 8 The network first receives the RGB image vector (128x128) input from the image encoder, and uses 64 5x5 convolution kernels (Conv2D) in the convolution layer 1 to perform preliminary feature extraction on the image. Then, a 2x2 max pooling layer (Pooling Layer) is used to reduce the dimension of the feature map, reducing its size by half. The calculation formula of the convolution layer is:

[0083] W = Relu(conv2(H, S) + R) #(7)

[0084] Where H is the input feature map, S is the convolution kernel (also known as the filter), and R is the bias. After the convolution operation by the conv function, a Relu activation function is used to introduce non-linear characteristics.

[0085] Next, 128 3x3 convolution kernels and 256 3x3 convolution kernels are used to extract deeper features from the feature map. These convolution layers gradually learn more complex image features. In the final stage, the extracted features are input into a fully connected layer (Fully Connected Layer) and flattened into a one-dimensional vector form, i.e., each pixel value of the feature map is arranged in a row, resulting in a final one-dimensional image feature as the output result of the network.

[0086] Convolutional Block Attention Module (CBAM). After the image features are extracted by the convolutional neural network, they are input into the convolutional attention network for further optimization. The specific situation is shown in Figure 9

[0087] Channel Attention Module (CAM). The design of the channel attention module aims to capture global importance information in the input feature map. First, the feature map is processed by average pooling and max pooling. The average pooling calculates the global average value of the feature map to obtain the overall information, while the max pooling extracts the global maximum value to capture the significant area. Finally, two 1x1xC feature maps are generated, which are further processed by a shared weight multi-layer perceptron (MLP).

[0088] The shared MLP contains two layers: the first layer uses a hidden unit number of C / r (where r is the reduction rate) and an activation function of ReLU, and the second layer has a neuron number of C. By sharing weights, not only the model parameter amount is significantly reduced, but also the model's generalization ability is improved. Then, the two processed feature maps are added element by element, and the channel attention weights are generated through the sigmoid activation function. These weights are used to weight the original input feature map, significantly improving the model's sensitivity and attention to key features. The formula is expressed as:

[0089]

[0090] where σ represents the sigmoid function, W0∈RC / r×C, W1∈RC×C / r, and W0 and W1 are two parameters in the MLP, which are shared.

[0091] Spatial attention module (SAM), the spatial attention module aims to capture the spatial distribution information of the feature map, so that the model can more accurately focus on the important areas in the input image. Its input is the feature map optimized by the channel attention module. First, global average pooling and global max pooling are used to generate two feature maps, which describe the importance of features from different angles. Then, they are concatenated in the channel dimension to form a new feature representation.

[0092] ​Then, the concatenated feature map is passed through a 7x7 convolution operation to reduce the channel number to 1, generating a spatial attention feature map. This convolution operation can capture local and global spatial relationships in the feature map, enhancing the representation ability. Finally, the convolution result is normalized by a sigmoid activation function to generate spatial attention weights. After element-wise multiplication of the input feature map and the weights, the generated output feature pays more attention to the regions with significant spatial information in the input image. The formula is expressed as:

[0093]

[0094] where σ represents the sigmoid function, f 7×7 represents the size of the filter is 7x7.

[0095] Feed Forward Neural Network (FFN). The optimized feature vector extracted by the attention module is input into the feed forward network composed of fully connected layers for further processing and understanding. The feed forward neural network can achieve efficient information representation and classification by learning the nonlinear relationship of the feature vector. Its core structure includes two fully connected layers, which are used for feature transformation and output generation, respectively. The first layer realizes linear transformation through a fully connected operation, and the weight matrix W1 and the bias vector b1 are used to map the input features to the hidden layer. The ReLU activation function is used to introduce nonlinearity to help the network capture complex feature patterns. Then the second layer maps the hidden layer features to the final output features through another fully connected operation. The formula is as follows:

[0096] FFN(x) = Relu(xW1 + b1)W2 + b2 # (10)

[0097] where x is the input vector, W1 and W2 are the weight matrices, b1 and b2 are the bias vectors, and the introduction of the ReLU activation function not only increases the representation ability of the network, but also avoids the problem of gradient vanishing, making the training more stable and efficient. Through this structure, the feed forward neural network can fully learn the complex relationships between features, suitable for various task scenarios such as classification and regression, and provide strong feature representation ability for the subsequent decision module.

[0098] S2.2, build an attention network module based on multi-modal feature fusion

[0099] To solve the problem of poor fusion of multi-modal features, a specific structure is invented as Figure 10The multi-modal feature fusion module is shown. This module mainly uses multi-head attention and multi-dimensional feature fusion network (MDFN) to coordinate with each other to complete the extraction operation of multi-modal features. At the same time, the multi-dimensional image features obtained are fused, and then the cross attention layer is used to enhance the correlation between different features. Finally, the full connection neural network is input, and the final multi-modal feature fusion feature vector is output, and the extraction operation is completed.

[0100] Multi-head attention network. In the design of this module, the multi-head attention network is the core component of feature extraction, which is particularly suitable for processing multi-dimensional image modal features and satellite observation value modal features. Its role is to extract fine-grained global information from different attention subspaces, so as to better capture the relationship between multi-modal features. The input multi-modal features (such as two-dimensional image features, three-dimensional image features, and satellite observation value features (signal-to-noise ratio features)) are projected into query (Query), key (Key), and value (Value) vector spaces respectively, and the specific formula is:

[0101] Q = W Q × X#(11)

[0102] K = W K × X#(12)

[0103] Q = W V × X#(13)

[0104] Here, W Q , W K , and W V are learned weight matrices, and X represents the input features.

[0105] Through the multi-head mechanism, the attention weights in multiple subspaces are calculated respectively, and each subspace is realized by scaled dot-product attention:

[0106]

[0107] Where, is the scaling factor, which aims to balance the gradient size and ensure the stability of training. Each attention head is calculated independently in different subspaces to capture the detailed correlation between multi-dimensional image modal features and satellite observation modal features. The formula of the multi-head mechanism is:

[0108] MultiHead(Q,K,V)=Concat(head1,head2,……,head h )W o #(15)

[0109] where Concat represents concatenating the outputs of multiple heads, and W o is a transformation matrix used to fuse the results of multiple attention heads. Multi-head attention can handle the complex correlation between two-dimensional and three-dimensional image features simultaneously, and can mine potential high-order dependencies. Since the attention mechanism covers all input features, the model can model global image information and avoid focusing only on local regions.

[0110] Multi-dimensional image feature fusion network (MDFN). To address the difficulties of multi-dimensional feature fusion and the difficulty of capturing the correlation between two-dimensional and three-dimensional features, this module designs a multi-dimensional image feature fusion network. This network can fuse two-dimensional image features and three-dimensional image features, and significantly enhance the correlation and fusion effect of different dimensional features by automatically adjusting the fusion weight between multi-dimensional features. The specific structure of this network is shown in Figure 11 Multi-dimensional image feature fusion network. Its main modules include two-dimensional feature embedding, three-dimensional feature embedding, multi-dimensional feature fusion, and attention mechanism, which are used to realize adaptive optimization and weighted fusion of global and local information.

[0111] For two-dimensional image features, the network embeds the features through global average pooling (GAP) to extract global information of the two-dimensional image. The specific calculation formula is:

[0112]

[0113] where X RH×S×R is the image feature, H is the height, S is the width, and R is the number of channels. Through global average pooling, the overall information of the two-dimensional image is embedded into a compact global feature representation, avoiding information loss.

[0114] For the extracted three-dimensional image features, the network designs a fully connected neural network (FCNN) containing a residual structure for embedding. The formula is:

[0115] f(y) = w1y + w2(ReLU(w1y))#(17)

[0116] where: y ∈ Rd is a multi-dimensional image feature; w1 ∈ Rd*c and w2 ∈ Rc*c are weight matrices used for feature dimension alignment. The addition of the residual structure improves the stability of three-dimensional feature embedding and can perform nonlinear enhancement on three-dimensional features to extract high-order spatial information.

[0117] After multi-dimensional image feature embedding, the fusion feature function is used for feature fusion, and the fusion result is:

[0118] θ = g(x) + f(y)#(18)

[0119] At the same time, in order to further improve the discrimination ability of the fused features, the attention mechanism is introduced in this module, which uses the Sigmoid function to dynamically adjust the weight of global information. The specific formula is as follows:

[0120] S = H(β, ε) = sigmoid(Q1(ReLU(Q2β)))#(19)

[0121] where H(β, ε) is the sigmoid function activation operation, the weight matrix Q1 ∈ RC×C / r, Q2 ∈ RC / r×C; the dimension reduction ratio r represents the reduction of the number of channels between the two fully connected layers, and r is taken as 16. After dimension reduction and activation of the input features through two fully connected layers, the weight S is obtained, which is used to represent the attention coefficient of the model to different dimensional image (two-dimensional or three-dimensional) features. Apply the weight S to the input features x and multi-dimensional features y respectively, and perform element-wise multiplication operation in the channel direction. The result is:

[0122] L = f j (f c (S, x), f c (S, y))#(20)

[0123] where L ∈ RH×S×R, fj is the addition of corresponding channel elements, and fc is the multiplication operation of corresponding channel elements.

[0124] Through the design of feature embedding and fusion functions, two-dimensional and three-dimensional image features can achieve efficient information interaction at the global and local levels. The attention mechanism dynamically adjusts the feature weights of different dimensions, effectively avoiding information loss in feature fusion. The residual structure and global pooling mechanism ensure the stable performance of the module in complex tasks, while being able to adapt to different types of image features.

[0125] Self-Attention Block. The self-attention block is the core unit of the multi-head attention mechanism, used to process image features of the same dimension and establish global dependencies within the features. For multi-dimensional image features, this module uses self-attention blocks when processing two-dimensional and three-dimensional image features.

[0126] By calculating the attention weights between image features of the same dimension, the self-attention block can capture the internal relationships of the features. The weight calculation formula is consistent with formula (14), which ensures that each feature point can dynamically perceive the importance of other feature points, thereby strengthening the global representation. Using multiple heads to capture the detailed information of different subspaces of image features improves the richness of feature expression.

[0127] For example, the texture information of two-dimensional images and the depth information of three-dimensional images can be extracted separately through the self-attention block, ultimately achieving more fine-grained feature modeling. The two-dimensional image features are enhanced in depth, extracting details such as texture and color. The three-dimensional image features extract spatial structure information, enhancing the representation ability of three-dimensional features.

[0128] Cross Attention Block. It aims to fuse image features from different dimensions, such as the relationship between two-dimensional and three-dimensional image features. This module achieves deep fusion of multi-dimensional features by constructing cross-dimensional correlations. The features in one dimension are used to adjust the features in the other dimension to enhance information exchange and complementarity between the two dimensions, represented by the formula:

[0129]

[0130] where x and y represent the feature maps of two different dimensional images, W x and W y are the weight matrices for dimension reduction, d kis the scaling factor of feature dimension, which normalizes the attention weight by softmax. In order to ensure the bidirectionality of information flow, the model also calculates in reverse, that is, taking the three-dimensional image feature as the query and the two-dimensional image feature as the key and value, so as to further strengthen the feature interaction. The feature output by the cross-attention block fuses the global context information of image features of different dimensions, not only improving the discriminability of the feature, but also providing a more robust input for the subsequent fully connected layer.

[0131] Multimodal feature fusion loss (MFFL). In order to fully exploit the potential of the multi-feature fusion network (MDFN) and ensure that it can effectively adapt to the image classification task, this scheme proposes a loss function that combines the characteristics of multi-dimensional feature fusion, adaptive weight optimization and classification task requirements. Its form is:

[0132]

[0133] wherein, is the standard cross-entropy loss, used for classification supervision; is the multi-dimensional image feature fusion regularization loss, used to improve the compactness and discriminability of the fused feature; is the consistency loss of attention mechanism, used to optimize the attention allocation weight, and a and β are adjustment hyperparameters that control the contribution of different loss terms.

[0134] Cross-entropy loss is the core loss function in the classification task, which is used to measure the difference between the predicted distribution and the true label distribution, and its formula is:

[0135]

[0136] wherein, N is the number of samples, C is the number of classification categories, y ij is the true label distribution of sample i, is the predicted probability distribution of sample i. Through the cross-entropy loss, it ensures that the network can effectively learn the class discriminative information of the input image.

[0137] In order to further optimize the representation ability of the fused feature (θ), a multi-dimensional feature fusion regularization loss is designed to constrain the two-dimensional feature g(x) and the three-dimensional feature f(y) to maintain a certain compactness and diversity during the fusion process. Its formula is:

[0138]

[0139] wherein, is the l2 norm, used to minimize the difference between the two-dimensional and three-dimensional features; KL(g(x)|||f(x)) is the KL divergence, used to measure the similarity between the two feature distributions; and are balance weights, controlling the contribution of the two regularization terms. By minimizing the feature difference and aligning the distribution, the representation ability of the multi-dimensional feature fusion is enhanced, and the classification performance of the model is improved.

[0140] To optimize the learning effect of the attention mechanism, a regularization loss based on attention weight consistency is proposed. Its formula is:

[0141]

[0142] where S x and S y respectively represent the attention weights calculated for two-dimensional and three-dimensional features, is the l2 norm, used to constrain the consistency of attention allocation. It ensures that the model can dynamically adjust the weights of different dimensional image features during feature fusion, while avoiding the instability of attention allocation.

[0143] Finally, to ensure that different loss terms contribute equally to the training process, we introduce a dynamic weight adjustment strategy to adaptively adjust the values of α and β according to the convergence of the loss during training:

[0144]

[0145] where t is the current training iteration number, and ∈ is a smoothing factor to avoid a denominator of 0.

[0146] In some feasible embodiments, step S3 specifically comprises:

[0147] For the obtained three-dimensional and two-dimensional satellite signal coherence image modal data and the corresponding satellite observation value modal (the satellite observation value used here is signal-to-noise ratio), this patent constructs a multi-modal feature fusion deep learning framework. By building an image feature extraction module based on convolutional attention mechanism, the image features of two-dimensional satellite signal images and three-dimensional satellite signal images are extracted; these features are input into the multi-modal feature fusion deep learning framework containing a multi-head attention network together with the satellite observation value modal (signal-to-noise ratio) at the corresponding time. By comparing the differences between the features of three-dimensional satellite signal images and two-dimensional satellite signal images, the differences between the features of pure direct signal and satellite signal disturbed by multipath interference are learned in three-dimensional and two-dimensional dimensions. At the same time, the satellite observation value, i.e. signal-to-noise ratio, which can better explain the current satellite data features, is used for learning and training.

[0148] In summary, the application provides a multi-modal feature fusion multi-path signal intelligent recognition method in a complex and variable environment.

[0149] In some possible embodiments, further comprising:

[0150] Converting the real-time GNSS baseband signal into to-be-measured image data;

[0151] Inputting the to-be-measured image data into the recognition model to complete signal classification.

[0152] Based on the above scheme

[0153] A multi-modal feature fusion multi-path signal intelligent recognition device in a complex and variable environment:

[0154] At least one processor;

[0155] At least one memory for storing at least one program;

[0156] When the at least one program is executed by the at least one processor, the at least one processor implements the multi-modal feature fusion multi-path signal intelligent recognition method in a complex and variable environment as described above.

[0157] The contents in the above method embodiments are all applicable to the device embodiments, the device embodiments specifically implement the functions of the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0158] A storage medium having processor-executable instructions stored therein, the processor-executable instructions, when executed by a processor, are used to implement the multi-modal feature fusion multi-path signal intelligent recognition method in a complex and variable environment as described above.

[0159] The contents in the method embodiments are applicable to the storage medium embodiments, the storage medium embodiments specifically implement the same functions as the method embodiments, and achieve the same beneficial effects as the method embodiments.

[0160] The above is a specific description of the preferred embodiments of the application, but the application is not limited to the embodiments described, and those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. A multi-path signal intelligent recognition method for multi-modal feature fusion in a complex and variable environment, characterized in that, The method comprises the following steps: acquiring original satellite data and constructing a multi-dimensional satellite signal image data set; constructing a deep learning framework based on a feature extraction module and a feature fusion module; training the deep learning framework based on the multi-dimensional satellite signal image data set and satellite observation value modalities at corresponding time points to obtain an identification model; the step of acquiring original satellite data and constructing a multi-dimensional satellite signal image data set specifically comprises: capturing original satellite data; solving the original satellite data, extracting correlation intensity, carrier frequency and code chip number as three-dimensional features, and generating three-dimensional image data; the calculation formula of the code chip number is as follows: wherein, is the number of chips, is the time of a signal sample, is the period of a single chip; processing the original satellite data, converting the correlation intensity into color strength for representation, searching around the code chip number of the maximum peak of the correlation intensity, and outputting two-dimensional image data; allocating labels to the three-dimensional image data and the two-dimensional image data to obtain a multi-dimensional satellite image data set.

2. The multi-modal feature fusion multi-path signal intelligent identification method in a complex and variable environment according to claim 1, further comprising: converting real-time GNSS baseband signals into image data to be tested; inputting the image data to be tested into the identification model to complete signal classification. 3.The multi-path signal intelligent recognition method of multi-modal feature fusion in a complex and variable environment according to claim 2, characterized in that, The calculation formula of the correlation intensity is as follows: wherein, is a baseband signal of the received raw satellite data, is a locally generated pseudo-random code reference signal, is a time delay, T is an integration time.

4. The multi-path signal intelligent recognition method of multi-modal feature fusion in a complex and variable environment according to claim 1, characterized in that, the step of processing the original satellite data, converting the correlation intensity into color strength for representation, searching around the code chip number of the maximum peak of the correlation intensity, and outputting two-dimensional image data specifically comprises: determining the correlation intensity, code chip number and carrier frequency of the original satellite data; representing the correlation intensity by color strength; adjusting the search range of the code chip number for the maximum peak of the correlation intensity, limiting it to a floating range centered on the peak, and outputting two-dimensional image data.

5. The multi-path signal intelligent recognition method of multi-modal feature fusion in a complex and variable environment according to claim 4, characterized in that, The formula for representing the correlation intensity by color strength is as follows: wherein is the color value of a pixel in the normalized two-dimensional image, is the maximum value of the correlation strength, is the minimum value of the correlation strength, represents the original signal strength value of the three-dimensional correlation strength at the position .​ 6. The multi-modal feature fusion-based multipath signal intelligent recognition method in a complex and variable environment according to claim 1, characterized in that, the step of training the deep learning framework based on the multi-dimensional satellite signal image data set and satellite observation value modalities at corresponding time points to obtain an identification model specifically comprises: inputting the multi-dimensional satellite signal image data set and satellite observation value modalities at corresponding time points into the deep learning framework; extracting features from three-dimensional image data and two-dimensional image data based on the feature extraction module to obtain two-dimensional image features and three-dimensional image features; processing the two-dimensional image features, the three-dimensional image features and satellite observation value modalities at corresponding time points based on the feature fusion module to learn the difference between pure direct signals and satellite signals disturbed by multi-path interference; adjusting model parameters in combination with a loss to obtain an identification model.

7. The multi-modal feature fusion-based multipath signal intelligent recognition method in a complex and variable environment according to claim 6, characterized in that, The feature extraction module comprises a convolutional neural network, a convolutional attention network and a feedforward neural network.

8. The multi-path signal intelligent recognition method of multi-modal feature fusion in a complex and variable environment according to claim 6, characterized in that, The feature fusion module comprises a multi-head attention network, a multi-dimensional image feature fusion network, a self-attention block, a cross-attention block and a fully connected neural network.

9. The multi-modal feature fusion-based multipath signal intelligent recognition method in a complex and variable environment according to claim 8, characterized in that, The loss function expression of the feature fusion module is as follows: wherein, is a standard cross-entropy loss, is a multi-dimensional image feature fusion regularization loss, and is a tuning hyperparameter, is a consistency loss of the attention mechanism.

Images