Target echo classification method and system based on multi-modal data fusion

Abstract

The invention discloses a target echo classification method and system based on multi-modal data fusion, relates to the technical field of computational electromagnetics, and solves the technical problems that the classification accuracy of a model is reduced in a noise environment, the generalization ability is limited, and the performance fluctuation is large when modal data distribution changes. The time sequence dependency is modeled through the Bi LSTM, the weight of each feature is analyzed, after the features are removed, global information is reserved through cross-modal fusion without removing the features, key discrimination features are focused through cross fusion after the features are removed, and the information integrity and the calculation efficiency can be both considered through combination of the two. And dynamically adjusting the contribution of different modal features to a classification result through the hidden state of the GRU. If certain modal data is missing, the original fusion path can still utilize other modal information, and the fusion path screening can depend on the key features of the remaining modals. And the sensitivity of the model to a specific noise mode can be reduced through the balance of redundant feature reservation and key feature screening.

Description

Technical Field

[0001] The present invention belongs to the field of computational electromagnetics, and in particular relates to a target echo classification method and system based on multimodal data fusion. Background Art

[0002] Target echo refers to the signal that is scattered and returned by the target after the electromagnetic waves emitted by detection equipment such as radar hit the target. These echo signals carry the target's physical characteristics (such as shape, size, and material), motion state (such as speed and direction), and environmental interference information (such as clutter and noise). In complex detection scenarios, the echo signals of different targets may overlap or interfere with each other, making it difficult for traditional single-modal classification methods (such as those that rely only on RF or time domain features) to meet high-precision requirements. The target echo classification system based on multimodal data fusion improves the accuracy and robustness of target echo classification by integrating multi-source heterogeneous data (such as RF signals, time domain waveforms, polarization characteristics, spatial position information, etc.).

[0003] Existing target echo classification schemes for multimodal data fusion directly employ cross-modal fusion classification methods. Using only cross-modal fusion without feature elimination, such as directly concatenating all modal features, can introduce redundant or noisy features, reducing classification accuracy and limiting generalization in noisy environments. Existing technologies may employ only a single fusion path, failing to balance global information integrity and key feature focus, resulting in significant performance fluctuations when the modal data distribution changes. Summary of the Invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a target echo classification method and system based on multimodal data fusion, which is used to solve the technical problems of decreased classification accuracy of the model in a noisy environment, limited generalization ability, and large performance fluctuations when the modal data distribution changes.

[0005] To solve the above problems, the first aspect of the present invention provides a target echo classification method and system based on multimodal data fusion, comprising the following steps:

[0006] Data collection nodes are set up in the radar signal collection area. The data collection nodes collect raw echo data through millimeter wave radar, aerosol distribution data through lidar, and target thermal radiation data through infrared sensors.

[0007] Divide the radar signal collection area into a grid, set up generation processing nodes in the grid, perform lightweight feature extraction on the data collection nodes, and add the corresponding node position data as data labels;

[0008] The processing node extracts the weather parameters of the data collection node based on the authorized weather station data, adds the weather parameters to the data tag, and sends it to the cloud processing platform together with the extracted feature data;

[0009] The cloud processing platform performs cross-modal attention fusion on radar echo features, aerosol distribution features, and infrared features to obtain cross-modal fusion features;

[0010] The BiLSTM modeling method uses temporal dependencies to analyze the weights of radar echo features, aerosol distribution features, and infrared features. The feature with the lowest weight among these three features is removed. The remaining two features are then fused using a two-stream Transformer network using cross-attention modal complementation to generate cross-fusion features.

[0011] According to the cross-modal fusion features and cross-fusion features, a classifier based on a gated recurrent unit plus a fully connected layer outputs the probability distribution of the classification.

[0012] Optionally, in an example of the above aspect, performing lightweight feature extraction on a data acquisition node and adding position data of the corresponding node as a data label includes the following steps:

[0013] The millimeter-wave radar collects raw echo data and generates a time-frequency spectrum through STFT. The time-frequency features of the time-frequency spectrum are extracted through MobileNetV3-Smal l, and the 256-dimensional radar echo features are output.

[0014] The laser radar collects aerosol distribution images and the infrared sensor collects target thermal radiation images, performs histogram equalization processing, extracts image spatial features through ShuffleNetV2, and outputs 256-dimensional feature data;

[0015] The lidar scanning point cloud is projected into the WGS84 coordinate system to generate the longitude and latitude coordinates of the corresponding nodes, and the longitude and latitude coordinates are used as lightweight features to extract data labels.

[0016] Optionally, in an example of the above aspect, a proxy processing node is set in the grid, and the proxy processing node extracts weather parameters of the data collection node based on the authorized weather station data, including the following steps:

[0017] By merging and re-splitting the grids, each grid is controlled to contain a preset number of data acquisition nodes, and generation processing nodes are set in the resulting grid;

[0018] The processing node obtains data from the authorized weather station, extracts the weather parameters of the data collection node, including temperature, humidity, wind speed and air pressure, and normalizes the collected data into a [-1,1] vector;

[0019] The normalized weather parameter vector is aligned with the data timestamp collected by the data acquisition node, and a 64-dimensional environment encoding vector is generated through the MLP network.

[0020] Optionally, in an example of the above aspect, the cloud processing platform performs cross-modal attention fusion on the radar echo features, aerosol distribution features, and infrared features to obtain cross-modal fusion features, including the following steps:

[0021] The radar echo feature Fr, aerosol distribution feature Fa, infrared feature Ft and environmental code vector Fe are spliced ​​together to construct the joint feature tensor Fjoint = [Fr, Fa, Ft, Fe];

[0022] Perform adaptive weight calculation and set the three-way cross attention of radar echo feature Fr, aerosol distribution feature Fa, and infrared feature Ft as: (i,j)∈{r,a,t};

[0023] Calculating attention weights in, is the modality-specific learnable parameter matrix, Scaling factor to prevent gradient vanishing;

[0024] The gating mechanism introduces environmental coding to control the information flow g = σ(Fe·Wg); Wg is the gating parameter matrix, σ is the Sigmoid activation function;

[0025] Calculate cross-modal fusion features Ffusion = g·∑ i,j αij+(1-g)·LayerNorm(Fjoint).

[0026] Optionally, in an example of the above aspect, temporal dependencies are modeled using BiLSTM to analyze weights of radar echo features, aerosol distribution features, and infrared features, including the following steps:

[0027] Radar echo features, aerosol distribution features and infrared features are vector spliced ​​to form a spliced ​​feature vector Xt;

[0028] The feature sequence is modeled using BiLSTM, which includes two LSTM units: one forward LSTM from the beginning to the end of the sequence: htf = LSTMf(Xt, h(t-1)f), and the other backward LSTM from the end to the beginning of the sequence: htb = LSTMb(Xt, h(t+1)b);

[0029] The forward LSTM and the backward LSTM perform a bidirectional hidden state fusion ht = [htf; htb], where ht is the final hidden state at time step t, which contains information from both the forward and backward LSTMs.

[0030] The attention mechanism is applied to the output of BiLSTM to calculate the attention weights of radar echo features, aerosol distribution features, and infrared features, and correspond to the weights of radar echo features, aerosol distribution features, and infrared features.

[0031] Optionally, in an example of the above aspect, an attention mechanism is applied to the output of the BiLSTM, and the attention weights of the radar echo features, aerosol distribution features, and infrared features are calculated as follows:

[0032] Ar=softmax(Wa·tanh(Wh·Fr+bh)) / softmax(Wa·tanh(Wh·ht+bh))

[0033] Aa=softmax(Wa·tanh(Wh·Fa+bh)) / softmax(Wa·tanh(Wh·ht+bh))

[0034] At=softmax(Wa·tanh(Wh·Ft+bh)) / softmax(Wa·tanh(Wh·ht+bh))

[0035] Where: Ar, Aa and At are the attention weights of radar echo features, aerosol distribution features and infrared features respectively, Fr, Fa and Ft represent the radar echo feature vector, aerosol distribution feature vector and infrared feature vector respectively, Wa and Wh are learnable weight matrices, bh is the bias vector, and tanh is the activation function used to introduce nonlinearity.

[0036] Optionally, in an example of the above aspect, through a two-stream Transformer network, the remaining two features are complemented by cross-attention modalities to perform feature fusion to obtain cross-fusion features, including the following steps:

[0037] Eliminate the features with the lowest weights among radar echo features, aerosol distribution features, and infrared features, and set the remaining two features as x and y;

[0038] Construct an independent Transformer branch to process the two selected modal features Fx and Fy;

[0039] Establish a bidirectional feature interaction channel between the two modalities:

[0040]

[0041] Among them, Attnx→y is the attention weight of the feature interaction channel from feature x to feature y, Attny→x is the attention weight of the feature interaction channel from feature y to feature x, T is the time series length, d is the feature dimension, Wq, Wk, Wv are modality-specific learnable parameters;

[0042] Aggregate the attention output of the bidirectional feature interaction channel to obtain the aggregated feature vector:

[0043] Fx′=LayerNorm(Fx+Attnx→y),

[0044] Fy′=LayerNorm(Fy+Attny→x);

[0045] The obtained aggregated feature vectors are spliced ​​to obtain cross-fusion features in, Represents a splicing operation.

[0046] Optionally, in an example of the above aspect, according to the cross-modal fusion features and the cross-fusion features, a classifier based on a gated recurrent unit (GRU) plus a fully connected layer outputs a probability distribution of the classification, including the following steps:

[0047] The cross-modal fusion feature Fcross and the cross-fusion feature Fcross are concatenated to obtain a new feature vector Fcombined = [Fcross; Fcross];

[0048] Use GRU to perform temporal modeling on Fcombined to capture the temporal dependencies between features and obtain the hidden state hT of the last time step;

[0049] Input hT into the fully connected layer and softmax function to obtain the probability distribution of the classification.

[0050] According to another aspect of the present disclosure, a target echo classification system based on multimodal data fusion is provided. The system adopts the target echo classification method based on multimodal data fusion as described above to implement target echo classification.

[0051] Compared with the prior art, the present invention has the following beneficial effects:

[0052] This paper uses BiLSTM to model temporal dependencies. BiLSTM captures the temporal dependencies of features through forward and backward LSTMs, and uses gradient descent optimization to adaptively adjust weights over time. Eliminating the lowest-weighted feature reduces the model's computational complexity, while considering only two highly correlated features, which helps better demonstrate modal complementarity.

[0053] The present invention retains global information through cross-modal fusion without feature elimination, and focuses on key discriminant features after feature elimination. The combination of the two can take into account both information integrity and computational efficiency. The contribution of different modal features to the classification results is dynamically adjusted through the hidden state of GRU. If a certain modality data is missing, the original fusion path can still use other modal information, while the filtered fusion path can rely on the key features of the remaining modalities. The balance between redundant feature retention and key feature screening can reduce the model's sensitivity to specific noise patterns and improve classification stability. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 Schematic diagram of the system framework of the present invention;

[0056] Figure 2 A flow chart of the method for adding data tags according to the present invention; Figure 3 Schematic diagram of the probability distribution of cross-modal fusion feature and cross-fusion feature analysis classification of the present invention. DETAILED DESCRIPTION

[0057] The technical solutions of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the embodiments described are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0058] See also Figure 1 - Figure, an embodiment of the first aspect of the present invention provides a target echo classification method and system based on multimodal data fusion, comprising the following steps:

[0059] Data collection nodes are set up in the radar signal collection area. The data collection nodes collect raw echo data through millimeter wave radar, aerosol distribution data through LiDAR laser radar, and target thermal radiation data through infrared sensors.

[0060] Divide the radar signal collection area into a grid, set up generation processing nodes in the grid, perform lightweight feature extraction on the data collection nodes, and add the corresponding node position data as data labels;

[0061] The processing node extracts the weather parameters of the data collection node based on the authorized weather station data, adds the weather parameters to the data tag, and sends it to the cloud processing platform together with the extracted feature data;

[0062] The cloud processing platform performs cross-modal attention fusion on radar echo features, aerosol distribution features, and infrared features to obtain cross-modal fusion features;

[0063] The BiLSTM modeling method uses temporal dependencies to analyze the weights of radar echo features, aerosol distribution features, and infrared features. The feature with the lowest weight among these three features is removed. The remaining two features are then fused using a two-stream Transformer network using cross-attention modal complementation to generate cross-fusion features.

[0064] According to the cross-modal fusion features and cross-fusion features, a classifier based on the GRU gated recurrent unit plus a fully connected layer outputs the probability distribution of the classification.

[0065] Specifically, in this embodiment, millimeter-wave radar nodes are deployed, and FMCW millimeter-wave radars (e.g., 24 GHz / 77 GHz) are placed at the edge of the acquisition area or in key locations to cover the target area. Raw I / Q echo data (including distance, velocity, and angle information) is collected, and the sampling rate must meet the Nyquist criterion (e.g., sampling rate ≥ 1 GHz for a bandwidth of 500 MHz). Multi-node data is synchronized using PTP (Precision Time Protocol) or GPS clocks.

[0066] Deploy LiDAR nodes, co-located or interleaved with millimeter-wave radar. Choose mechanical or solid-state LiDAR (such as the 16-line Velodyne or Ouster OS1) with a vertical angular resolution of ≤ 0.3°. Collect aerosol (PM2.5 / PM10) distribution data.

[0067] Deploy infrared sensor nodes and place uncooled infrared arrays (such as FLIR Lepton 3.5) with a resolution of ≥160×120 at regional high points or above the path of dynamic targets to collect infrared image data.

[0068] Perform grid division, dividing the 3D grid into grids based on the sensor resolution (e.g., a LiDAR horizontal angular resolution of 0.1° corresponds to approximately 0.3m @ 100m on the ground). For example, a 100m×100m area can be divided into a 1m×1m×1m voxel grid, with each grid associated with a unique ID.

[0069] Set up a proxy processing node. For each sub-area, such as a 10×10 grid, a proxy processing node is set up at the center to coordinate 5-10 surrounding data collection nodes. The proxy processing node performs data aggregation, time synchronization, and lightweight feature fusion.

[0070] Position encoding is performed, and the grid ID is embedded as a tag in the data header, such as [Grid_ID:X12_Y34_Z2,Timestamp:1620000000]. The LiDAR ENU coordinates are converted to the radar's local coordinate system using the seven-parameter method. Synchronization is performed using the NTPv4 protocol, with inter-node time error ≤50μs.

[0071] Obtain authorized weather station data, weather station data source integration, data types, including:

[0072] Basic parameters: temperature (°C), humidity (%), air pressure (hPa), wind speed (m / s), wind direction (°);

[0073] Advanced parameters: visibility (km), precipitation intensity (mm / h), cloud height (m);

[0074] Real-time API: Connect to authorized open interfaces (such as RESTful API) such as the Meteorological Bureau and NOAA, and pull data every 5 minutes.

[0075] Perform data quality verification and rationality checks, including: temperature range [-50℃, 60℃], humidity [0%, 100%], and mark abnormal values ​​as NaN.

[0076] Spatial mapping is performed by spreading the weather station data onto the grid area using kriging interpolation to generate a spatially continuous meteorological field. For example, the data from weather stations A (100m above sea level) and B (200m above sea level) are corrected by the elevation-temperature gradient (0.6°C / 100m) to obtain the temperature at the grid center at an altitude of 150m.

[0077] Time synchronization is performed to align the meteorological data timestamps to the sensor data frames (e.g., the 100ms period of millimeter-wave radar), and the time scale differences are reduced by sliding window averaging (window = 5 frames).

[0078] Extract the values ​​of basic and advanced parameters of data collection nodes, add the extracted values ​​to data tags, and send them to the cloud processing platform together with the extracted feature data;

[0079] Through the cross-modal attention mechanism, the weights of radar echo features, aerosol distribution features, and infrared features can be dynamically adjusted. For example, in rainy and foggy weather, the thermal radiation gradient of infrared features and the micro-Doppler characteristics of radar echoes may be given higher weights, while the weight of aerosol distribution features in low-visibility scenes may be reduced, achieving environmentally adaptive feature enhancement. Cross-modal attention calculates the correlation between features of different modalities through the attention matrix. For example, the offset between the target motion trajectory detected by the radar and the infrared thermal radiation center can be associated with the attention weight, thereby discovering implicit physical laws such as "drone propeller rotation causes local temperature anomalies" and improving feature interpretability.

[0080] Different modal noise characteristics vary significantly, such as radar clutter and infrared noise. Cross-modal attention can suppress single-modal noise through feature alignment. For example, when a radar echo causes a false alarm due to strong ground clutter, if the infrared feature indicates that there is no heat source in the area, the attention weight can be used to suppress this echo feature, reducing the false detection rate.

[0081] The BiLSTM modeling method uses temporal dependencies to analyze the weights of radar echo features, aerosol distribution features, and infrared features. The feature with the lowest weight among these three features is removed. The remaining two features are then fused using a two-stream Transformer network using cross-attention modal complementation to generate cross-fusion features.

[0082] The process is as follows: data collection → BiLSTM time series modeling → feature elimination → two-stream Transformer fusion → output. BiLSTM captures the temporal dependencies of features through forward and backward LSTMs, and weights are adaptively adjusted over time through gradient descent optimization. Eliminating the lowest-weighted features helps reduce the model's computational complexity. Furthermore, considering only two highly correlated features helps better demonstrate modal complementarity.

[0083] According to the cross-modal fusion features and cross-fusion features, a classifier based on the GRU gated recurrent unit plus a fully connected layer outputs the probability distribution of the classification.

[0084] Target echo signals typically have temporal characteristics (such as the time-varying nature of pulse trains). GRUs dynamically capture long- and short-term dependencies through update and reset gates, making them more effective than traditional RNNs. Cross-modal fusion without feature elimination preserves global information, while cross-fusion after feature elimination focuses on key discriminant features. The combination of the two balances information integrity and computational efficiency. The contribution of different modal features to the classification results is dynamically adjusted through the GRU's hidden state. If data from a particular modality is missing (such as when the radar is obscured), the original fusion path can still utilize information from other modalities, while the filtered fusion path can rely on the key features of the remaining modalities. This balance between retaining redundant features and filtering key features can reduce the model's sensitivity to specific noise patterns and improve classification stability.

[0085] In one embodiment of the present invention, lightweight feature extraction is performed on data acquisition nodes, and location data of corresponding nodes is added as data labels, including the following steps:

[0086] The millimeter-wave radar collects raw echo data and generates a time-spectrogram through STFT. The time-frequency features of the time-spectrogram are extracted through MobileNetV3-Small, and the radar echo features with 256 dimensions are output.

[0087] The laser radar collects aerosol distribution images and the infrared sensor collects target thermal radiation images, performs histogram equalization processing, extracts image spatial features through ShuffleNetV2, and outputs 256-dimensional feature data;

[0088] The lidar scanning point cloud is projected into the WGS84 coordinate system to generate the longitude and latitude coordinates of the corresponding nodes, and the longitude and latitude coordinates are used as lightweight features to extract data labels.

[0089] In this embodiment, radar echo data is generated by STFT (128×128 resolution) to generate a time-frequency spectrum, which is normalized to [0, 1], and MobileNetV3-Smal l is used to extract time-frequency features.

[0090] Aerosol distribution images and infrared images: Histogram equalization is used to enhance contrast, and the images are cropped to 224×224 ROI regions. Spatial features are extracted using ShuffleNetV2.

[0091] Weather data: normalized to [-1,1] vectors, aligned with radar / infrared data timestamps.

[0092] In one embodiment of the present invention, a proxy processing node is set in the grid, and the proxy processing node extracts weather parameters of the data collection node based on the authorized weather station data, including the following steps:

[0093] By merging and re-splitting the grids, each grid is controlled to contain a preset number of data acquisition nodes, and generation processing nodes are set in the resulting grid;

[0094] The processing node obtains data from the authorized weather station, extracts the weather parameters of the data collection node, including temperature, humidity, wind speed and air pressure, and normalizes the collected data into a [-1,1] vector;

[0095] The normalized weather parameter vector is aligned with the data timestamp collected by the data acquisition node, and a 64-dimensional environment encoding vector is generated through the MLP network.

[0096] In one embodiment of the present invention, a cloud processing platform performs cross-modal attention fusion on radar echo features, aerosol distribution features, and infrared features to obtain cross-modal fusion features, including the following steps:

[0097] The radar echo feature Fr, aerosol distribution feature Fa, infrared feature Ft and environmental code vector Fe are spliced ​​together to construct the joint feature tensor Fjoint = [Fr, Fa, Ft, Fe];

[0098] Perform adaptive weight calculation and set the three-way cross attention of radar echo feature Fr, aerosol distribution feature Fa, and infrared feature Ft as: (i,j)∈{r,a,t};

[0099] Calculating attention weights in, is the modality-specific learnable parameter matrix, Scaling factor to prevent gradient vanishing;

[0100] The gating mechanism introduces environmental coding to control the information flow g = σ(Fe·Wg); Wg is the gating parameter matrix, σ is the Sigmoid activation function;

[0101] Calculate cross-modal fusion features Ffusion = g·∑ i,j αij+(1-g)·LayerNorm(Fjoint).

[0102] In one embodiment of the present invention, BiLSTM is used to model temporal dependencies and analyze the weights of radar echo features, aerosol distribution features, and infrared features, including the following steps:

[0103] Radar echo features, aerosol distribution features and infrared features are vector spliced ​​to form a spliced ​​feature vector Xt;

[0104] The feature sequence is modeled using BiLSTM, which includes two LSTM units: one forward LSTM from the beginning to the end of the sequence: htf = LSTMf(Xt, h(t-1)f), and the other backward LSTM from the end to the beginning of the sequence: htb = LSTMb(Xt, h(t+1)b);

[0105] The forward LSTM and the backward LSTM perform a bidirectional hidden state fusion ht = [htf; htb], where ht is the final hidden state at time step t, which contains information from both the forward and backward LSTMs.

[0106] The attention mechanism is applied to the output of BiLSTM to calculate the attention weights of radar echo features, aerosol distribution features, and infrared features, and correspond to the weights of radar echo features, aerosol distribution features, and infrared features.

[0107] In one embodiment of the present invention, an attention mechanism is applied to the output of BiLSTM to calculate the attention weights of radar echo features, aerosol distribution features, and infrared features as follows:

[0108] Ar=softmax(Wa·tanh(Wh·Fr+bh)) / softmax(Wa·tanh(Wh·ht+bh))

[0109] Aa=softmax(Wa·tanh(Wh·Fa+bh)) / softmax(Wa·tanh(Wh·ht+bh))

[0110] At=softmax(Wa·tanh(Wh·Ft+bh)) / softmax(Wa·tanh(Wh·ht+bh))

[0111] Where: Ar, Aa and At are the attention weights of radar echo features, aerosol distribution features and infrared features respectively, Fr, Fa and Ft represent the radar echo feature vector, aerosol distribution feature vector and infrared feature vector respectively, Wa and Wh are learnable weight matrices, bh is the bias vector, and tanh is the activation function used to introduce nonlinearity.

[0112] In one embodiment of the present invention, through a two-stream Transformer network, the remaining two features are complemented by cross-attention modalities to perform feature fusion to obtain cross-fusion features, including the following steps:

[0113] Eliminate the features with the lowest weights among radar echo features, aerosol distribution features, and infrared features, and set the remaining two features as x and y;

[0114] Construct an independent Transformer branch to process the two selected modal features Fx and Fy;

[0115] Establish a bidirectional feature interaction channel between the two modalities:

[0116]

[0117] Among them, Attnx→y is the attention weight of the feature interaction channel from feature x to feature y, Attny→x is the attention weight of the feature interaction channel from feature y to feature x, T is the time series length, d is the feature dimension, Wq, Wk, Wv are modality-specific learnable parameters;

[0118] Aggregate the attention output of the bidirectional feature interaction channel to obtain the aggregated feature vector:

[0119] Fx′=LayerNorm(Fx+Attnx→y),

[0120] Fy′=LayerNorm(Fy+Attny→x);

[0121] The obtained aggregated feature vectors are spliced ​​to obtain the cross-fusion feature Fcross = [Fx′⊕Fy′], where ⊕ represents the splicing operation.

[0122] In one embodiment of the present invention, based on the cross-modal fusion features and the cross-fusion features, a classifier based on a gated recurrent unit (GRU) plus a fully connected layer outputs a probability distribution of the classification, including the following steps:

[0123] The cross-modal fusion feature Fcross and the cross-fusion feature Fcross are concatenated to obtain a new feature vector Fcombined = [Fcross; Fcross];

[0124] Use GRU to perform temporal modeling on Fcombined to capture the temporal dependencies between features and obtain the hidden state hT of the last time step;

[0125] The calculation process of the GRU unit is as follows:

[0126] Update gate: zt = σ(Wz[ht-1,xt]+bz)

[0127] Reset gate: rt = σ(Wr[ht-1,xt]+br)

[0128] Candidate hidden state: h~t=tanh(Wh[rt⊙ht-1,xt]+bh)

[0129] Hidden state update: ht=(1-zt)⊙ht-1+zt⊙h~t

[0130] Where: xt is the input feature at time step t, ht-1 is the hidden state at the previous time step, Wz, Wr and Wh are learnable weight matrices, bz, br and bh are bias vectors respectively, σ is the sigmoid activation function, and ⊙ is element-wise multiplication.

[0131] Input hT into the fully connected layer and softmax function to obtain the probability distribution of the classification.

[0132] The last hidden state hT of GRU is classified through the fully connected layer, the formula is: o = WohT + bo;

[0133] Through the softmax function, the probability distribution of the classification is obtained, the formula is:

[0134] P(y=a)=softmax(o)i=exp(oa) / ∑exp(on)

[0135] Where: Wo is the weight matrix of the fully connected layer, bo is the bias vector of the fully connected layer, o is the output vector of the fully connected layer, and P(y=a) is the probability of category a.

[0136] In this embodiment, the obtained radar echo features, aerosol distribution features, infrared features, and environmental coding cross-modal fusion features are combined; and through a two-stream Transformer network, two of the radar echo features, aerosol distribution features, and infrared features are complemented by cross-attention modalities to obtain cross-fusion features. Based on the cross-modal fusion features and the cross-fusion features, a classifier based on a gated recurrent unit (GRU) plus a fully connected layer outputs a probability distribution of the classification.

[0137] The temporal dependency between features is captured by the calculation of the GRU unit to obtain the hidden state hT of the last time step;

[0138] The last hidden state hT of GRU is classified through the fully connected layer, and the probability distribution of the classification is obtained through the softmax function.

[0139] In another embodiment of the present invention, a target echo classification system based on multimodal data fusion is provided. The system adopts the target echo classification method based on multimodal data fusion as described above to implement target echo classification.

[0140] The above embodiments are only used to illustrate the technical method of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A target echo classification method and system based on multimodal data fusion, characterized in that: The following steps are involved: Data collection nodes are set up in the radar signal collection area. The data collection nodes collect raw echo data through millimeter wave radar, aerosol distribution data through lidar, and target thermal radiation data through infrared sensors. Divide the radar signal collection area into a grid, set up generation processing nodes in the grid, perform lightweight feature extraction on the data collection nodes, and add the corresponding node position data as data labels; The processing node extracts the weather parameters of the data collection node based on the authorized weather station data, adds the weather parameters to the data tag, and sends it to the cloud processing platform together with the extracted feature data; The cloud processing platform performs cross-modal attention fusion on radar echo features, aerosol distribution features, and infrared features to obtain cross-modal fusion features; The BiLSTM modeling method uses temporal dependencies to analyze the weights of radar echo features, aerosol distribution features, and infrared features. The feature with the lowest weight among these three features is removed. The remaining two features are then fused using a two-stream Transformer network using cross-attention modal complementation to generate cross-fusion features. According to the cross-modal fusion features and cross-fusion features, a classifier based on a gated recurrent unit plus a fully connected layer outputs the probability distribution of the classification.

2. The target echo classification method based on multimodal data fusion according to claim 1, characterized in that: Perform lightweight feature extraction on data collection nodes and add the corresponding node location data as data labels, including the following steps: The millimeter-wave radar collects raw echo data and generates a time-frequency spectrum through STFT. The time-frequency features of the time-frequency spectrum are extracted through MobileNetV3-Smal l, and the 256-dimensional radar echo features are output. The laser radar collects aerosol distribution images and the infrared sensor collects target thermal radiation images, performs histogram equalization processing, extracts image spatial features through ShuffleNetV2, and outputs 256-dimensional feature data; The lidar scanning point cloud is projected into the WGS84 coordinate system to generate the longitude and latitude coordinates of the corresponding nodes, and the longitude and latitude coordinates are used as lightweight features to extract data labels.

3. The target echo classification method based on multimodal data fusion according to claim 1, characterized in that: Set up a proxy processing node in the grid. The proxy processing node extracts the weather parameters of the data collection node based on the authorized weather station data, including the following steps: By merging and re-splitting the grids, each grid is controlled to contain a preset number of data acquisition nodes, and generation processing nodes are set in the resulting grid; The processing node obtains data from the authorized weather station, extracts the weather parameters of the data collection node, including temperature, humidity, wind speed and air pressure, and normalizes the collected data into a [-1,1] vector; The normalized weather parameter vector is aligned with the data timestamp collected by the data acquisition node, and a 64-dimensional environment encoding vector is generated through the MLP network.

4. The target echo classification method based on multimodal data fusion according to claim 1, characterized in that: The cloud processing platform performs cross-modal attention fusion on radar echo features, aerosol distribution features, and infrared features to obtain cross-modal fusion features, including the following steps: The radar echo feature Fr, aerosol distribution feature Fa, infrared feature Ft and environmental code vector Fe are spliced ​​together to construct the joint feature tensor Fjoint = [Fr, Fa, Ft, Fe]; Perform adaptive weight calculation and set the three-way cross attention of radar echo feature Fr, aerosol distribution feature Fa, and infrared feature Ft as: Calculating attention weights in, is the modality-specific learnable parameter matrix, Scaling factor to prevent gradient vanishing; The gating mechanism introduces environmental coding to control the information flow g = σ(Fe·Wg); Wg is the gating parameter matrix, σ is the Sigmoid activation function; Calculate cross-modal fusion features Ffusion = g·∑ i,j αij+(1-g)·LayerNorm(Fjoint).

5. The target echo classification method based on multimodal data fusion according to claim 1, characterized in that: BiLSTM is used to model temporal dependencies and analyze the weights of radar echo characteristics, aerosol distribution characteristics, and infrared characteristics. The following steps are involved: Radar echo features, aerosol distribution features and infrared features are vector spliced ​​to form a spliced ​​feature vector Xt; The feature sequence is modeled using BiLSTM, which includes two LSTM units: one forward LSTM from the beginning to the end of the sequence: htf = LSTMf(Xt, h(t-1)f), and the other backward LSTM from the end to the beginning of the sequence: htb = LSTMb(Xt, h(t+1)b); The forward LSTM and the backward LSTM perform a bidirectional hidden state fusion ht = [htf; htb], where ht is the final hidden state at time step t, which contains information from both the forward and backward LSTMs. The attention mechanism is applied to the output of BiLSTM to calculate the attention weights of radar echo features, aerosol distribution features, and infrared features, and correspond to the weights of radar echo features, aerosol distribution features, and infrared features.

6. The target echo classification method based on multimodal data fusion according to claim 5, characterized in that: Applying the attention mechanism to the output of BiLSTM, the attention weights of radar echo features, aerosol distribution features, and infrared features are calculated as follows: Ar=softmax(Wa·tanh(Wh·Fr+bh)) / softmax(Wa·tanh(Wh·ht+bh)) Aa=softmax(Wa·tanh(Wh·Fa+bh)) / softmax(Wa·tanh(Wh·ht+bh)) At=softmax(Wa·tanh(Wh·Ft+bh)) / softmax(Wa·tanh(Wh·ht+bh))where: Ar, Aa and At are the attention weights of radar echo features, aerosol distribution features and infrared features, respectively. Fr, Fa and Ft represent the radar echo feature vector, aerosol distribution feature vector and infrared feature vector, respectively. Wa and Wh are learnable weight matrices, bh is the bias vector, and tanh is the activation function used to introduce nonlinearity.

7. The target echo classification method based on multimodal data fusion according to claim 1, characterized in that: Through the two-stream Transformer network, the remaining two features are complemented by cross-attention mode and feature fusion is performed to obtain cross-fusion features, including the following steps: Eliminate the features with the lowest weights among radar echo features, aerosol distribution features, and infrared features, and set the remaining two features as x and y; Construct an independent Transformer branch to process the two selected modal features Fx and Fy; Establish a bidirectional feature interaction channel between the two modalities: Among them, Attnx→y is the attention weight of the feature interaction channel from feature x to feature y, Attny→x is the attention weight of the feature interaction channel from feature y to feature x, T is the time series length, d is the feature dimension, Wq, Wk, Wv are modality-specific learnable parameters; Aggregate the attention output of the bidirectional feature interaction channel to obtain the aggregated feature vector: Fx′=LayerNorm(Fx+Attnx→y), Fy′=LayerNorm(Fy+Attny→x); The obtained aggregated feature vectors are spliced ​​to obtain the cross-fusion feature Fcross = [Fx′⊕Fy′], where ⊕ represents the splicing operation.

8. The target echo classification method based on multimodal data fusion according to claim 1, characterized in that: Based on the cross-modal fusion features and cross-fusion features, a classifier based on a gated recurrent unit (GRU) plus a fully connected layer outputs the probability distribution of the classification, including the following steps: The cross-modal fusion feature Fcross and the cross-fusion feature Fcross are concatenated to obtain a new feature vector Fcombined = [Fcross; Fcross]; Use GRU to perform temporal modeling on Fcombined to capture the temporal dependencies between features and obtain the hidden state hT of the last time step; Input hT into the fully connected layer and softmax function to obtain the probability distribution of the classification.

9. The target echo classification system based on multimodal data fusion is characterized by: The system implements target echo classification by using the target echo classification method based on multimodal data fusion as described in any one of claims 1 to 8.

Images