Wireless signal and thermal imaging multi-modal collaborative detection and directional interference method and system

Abstract

The present invention provides a method and system for multi-modal collaborative detection and directional interference of wireless signals and thermal imaging, which relates to the field of electronic countermeasure technology, including time domain synchronization and spatial domain registration of wireless signal data and infrared thermal imaging data, building a multi-modal feature fusion network, using attention mechanism and recursive neural network to extract the timing characteristics of the target, and output the position coordinates, motion state and behavior pattern information of the target. Then, based on the target state evaluation model and reinforcement learning algorithm, an intelligent interference strategy is designed, and the interference beam is dynamically adjusted through adaptive beamforming technology, finally realizing precise directional interference of the target. The present invention can effectively improve the target detection accuracy and interference efficiency, reduce the false alarm rate, and enhance the target recognition and interference capabilities in complex environments.

Description

Technical Field

[0001] The present invention relates to electronic countermeasure technology, and in particular to a method and system for multi-modal cooperative detection and directional interference of wireless signals and thermal imaging. Background Art

[0002] Traditional target detection and jamming technologies usually rely on a single sensor, such as a radar or infrared sensor. Radar detection technology has the advantages of all-weather working capability and a long detection distance, but it is easily interfered by complex electromagnetic environments and has difficulty identifying the fine features of the target. Infrared thermal imaging technology can passively detect the thermal radiation of the target and has good concealment and anti-interference capabilities, but it is greatly affected by factors such as ambient temperature and weather conditions, has a limited detection distance, and is difficult to obtain the target's motion state information. The use of these technologies alone has certain limitations and it is difficult to meet the needs of accurate detection and effective jamming of targets in complex environments. The existing technologies mainly have the following defects and shortcomings:

[0003] Single-mode detection has limited information and is difficult to adapt to complex environments: Traditional target detection technology usually relies on a single sensor, obtains limited information, and is easily affected by environmental factors. In complex environments, such as strong electromagnetic interference, rainy and foggy weather, the reliability and accuracy of single-mode detection will decrease.

[0004] Low target recognition accuracy makes it difficult to achieve refined jamming: Most existing jamming technologies are based on the approximate location information of the target, making it difficult to identify and classify the target in a refined manner. This results in poor jamming effects and may even cause accidental damage to friendly targets.

[0005] Interference strategies lack intelligence and adaptability: Traditional interference strategies are usually pre-set and lack the ability to analyze and predict the target's behavior patterns and motion states. This makes it difficult for interference strategies to adapt to the dynamic changes of the target and easy for the target to evade or counter-interference. Summary of the invention

[0006] The embodiments of the present invention provide a method and system for multi-modal collaborative detection and directional interference of wireless signals and thermal imaging, which can solve the problems in the prior art.

[0007] According to a first aspect of the embodiments of the present invention,

[0008] Provides wireless signal and thermal imaging multi-modal collaborative detection and directional interference methods, including:

[0009] The wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor are synchronized in the time domain and registered in the space domain to obtain a multi-source data set aligned in time and space, the wireless signal data of the multi-source data set is feature decomposed by wavelet transform and target feature information is extracted, the infrared thermal imaging data is segmented and feature extracted to obtain multi-modal feature data of the target;

[0010] Constructing a multimodal feature fusion network, inputting the multimodal feature data into the multimodal feature fusion network, using an attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, inputting the weighted features into a recursive neural network for time series modeling to obtain target time series features; inputting the target time series features into a multi-task learning framework, and outputting the position coordinates, motion state, and behavior pattern information of the target;

[0011] A target state assessment model is established based on the position coordinates, motion state and behavior pattern information of the target, and an intelligent interference strategy is designed using a reinforcement learning algorithm. The intelligent interference strategy is input into a waveform generator to generate a corresponding interference signal, and an adaptive beamforming algorithm is used to dynamically adjust the azimuth angle and pitch angle of the interference beam to transmit an interference signal to the target.

[0012] In an optional embodiment,

[0013] The steps of performing time domain synchronization and spatial domain registration on the wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor to obtain a multi-source data set aligned in time and space include:

[0014] The wireless signal data and infrared thermal imaging data are aligned in the time domain based on the sliding alignment algorithm of the time window. The wireless signal data and infrared thermal imaging data at the same time are determined by calculating the timestamp difference of adjacent data frames, and the time alignment label is generated.

[0015] Perform spatial feature extraction on the wireless signal data, construct a signal covariance matrix using the antenna array manifold characteristics, perform feature decomposition on the signal covariance matrix to obtain a signal subspace, estimate the signal arrival angle based on a multiple signal classification algorithm to obtain a wireless signal spatial feature vector; perform spatial feature extraction on the infrared thermal imaging data, extract the target bounding box coordinates using a target detection neural network, calculate the azimuth and pitch angle of the target in the image plane, and perform angle conversion based on camera calibration parameters to obtain an infrared thermal imaging spatial feature vector;

[0016] Based on the time alignment tag, the wireless signal space feature vector and the infrared thermal imaging space feature vector, spatial registration is performed, a spherical coordinate system with the center of the antenna array as the origin is established, the wireless signal space feature vector and the infrared thermal imaging space feature vector are converted to the spherical coordinate system, the azimuth angle difference and the elevation angle difference of the wireless signal space feature vector and the infrared thermal imaging space feature vector are calculated, and the registration is completed when the azimuth angle difference and the elevation angle difference are less than a preset angle threshold, so as to obtain a feature pair aligned in time and space;

[0017] The signal-to-noise ratio value of the wireless signal is calculated, the infrared thermal imaging target detection probability value is evaluated, and the spatiotemporally aligned feature pairs whose signal-to-noise ratio value is greater than the signal-to-noise ratio threshold and whose infrared thermal imaging target detection probability value is greater than the target detection threshold are formed into a multi-source data set.

[0018] In an optional embodiment,

[0019] The steps of constructing a multimodal feature fusion network, inputting the multimodal feature data into the multimodal feature fusion network, using an attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, and inputting the weighted features into a recursive neural network for time series modeling to obtain target time series features include:

[0020] Perform multi-scale decomposition on the multimodal feature data to obtain physical layer features, representation layer features and semantic layer features, wherein the physical layer features include spatial position and orientation information, the representation layer features include signal quality and gradient information, and the semantic layer features include spectrum and depth feature information;

[0021] Construct a multimodal feature fusion network, the multimodal feature fusion network includes a feature projection module and a feature constraint module, wherein the feature projection module includes a wireless signal feature projection unit and an infrared image feature projection unit, the multimodal feature data are respectively input into the corresponding feature projection unit for feature space mapping, the similarity matrix of the mapped features is calculated, and the cross-modal feature weights are generated based on the similarity matrix; the feature constraint module introduces a space transformation network at the physical layer to calculate the spatial consistency constraint, introduces a temporal smoothing factor at the representation layer to calculate the temporal continuity constraint, calculates the state consistency constraint at the semantic layer through adversarial learning, and integrates the spatial consistency constraint, the temporal continuity constraint and the state consistency constraint based on the cross-modal feature weight to generate the initial fusion feature;

[0022] An attention mechanism is used to adaptively assign weights to different modal features in the initial fusion features, wherein the attention mechanism includes a long-range feature extraction branch and a short-range feature extraction branch, wherein the long-range feature extraction branch captures the global dependency of the target motion, and the short-range feature extraction branch captures the local features of the target state, and an adaptive weight is calculated based on the statistics of the long-range features and the short-range features, and a weighted feature is obtained based on the adaptive weight;

[0023] The weighted features are input into a recursive neural network, which includes a feature encoding module and a timing decoding module. The input features are encoded by the feature encoding module, and the timing decoding module is used to extract the timing dependency to obtain the timing features. The timing features are weighted and fused to output the target timing features.

[0024] In an optional embodiment,

[0025] The step of inputting the target time series features into a multi-task learning framework and outputting the target's position coordinates, motion state, and behavior pattern information comprises:

[0026] Input the target temporal features into a shared encoder to extract a universal feature representation, and the shared encoder uses a task-specific decoder to perform feature decoding for position coordinate estimation, motion state discrimination, and behavior pattern recognition, respectively; perform spatiotemporal feature enhancement on the universal feature representation, use a self-attention mechanism in the time dimension to calculate the temporal feature dependency to generate a temporal attention weight, and introduce a spatial attention module in the spatial dimension to capture local spatial correlations to obtain enhanced features;

[0027] The enhanced features are input into a multi-scale feature pyramid network, features of different scales are fused through a residual connection structure, position prediction is optimized using a cascade regression method, and real-time position coordinate information of the target is output;

[0028] A temporal convolutional network is used to extract target speed and acceleration features, and the speed and acceleration features and real-time position coordinate information in a continuous time period are input into a bidirectional gated recurrent unit network, and the motion state information of the target is identified through a multi-layer perceptron;

[0029] A temporal graph structure is constructed based on real-time position coordinate information and motion state information. The nodes of the graph structure represent the state of the target at a certain moment, and the edges represent the temporal conversion relationship of the state. A temporal graph convolutional network is used to extract the behavioral features in the graph structure, and the behavioral pattern information of the target is output through graph pooling and fully connected layers.

[0030] In an optional embodiment,

[0031] The steps of establishing a target state evaluation model based on the position coordinates, motion state and behavior pattern information of the target and using a reinforcement learning algorithm to design an intelligent interference strategy include:

[0032] Based on the position coordinates, motion state and behavior pattern information of the target, construct a multidimensional evaluation index including a spatial activity range index, a motion characteristic index and a behavior characteristic index; construct a target state evaluation model, map the multidimensional evaluation index into a hierarchical Markov decision process, the state space is composed of the multidimensional evaluation index, the action space includes the interference strategy type and the interference parameter configuration, the interference strategy type is selected in the top-level decision process, and the interference parameter configuration is determined in the bottom-level decision process;

[0033] A dual Q network is used for interference strategy learning. The dual Q network includes a current network and a target network. The current network evaluates the state-action value function to predict the interference strategy benefit. The target network generates a target Q value. The state-action transition sequence of the target in the decision-making process is stored through the experience replay mechanism to establish a sample pool. The sampling weight is set based on the temporal difference error to perform sample screening.

[0034] A hierarchical exploration mechanism is constructed based on the state-action value function, an ε-greedy strategy is adopted at the interference strategy type level, Gaussian noise is introduced at the interference parameter configuration level, and the exploration probability is dynamically adjusted according to the statistical variance of the multi-dimensional evaluation index to generate a preliminary interference strategy;

[0035] A multi-objective reward function including jamming effect, resource consumption and jamming concealment is constructed, and the preliminary jamming strategy is optimized by using a gradient projection method to obtain a final intelligent jamming strategy;

[0036] The final intelligent interference strategy is updated in real time by online learning, the interference strategy execution deviation is calculated based on the multi-dimensional evaluation index, and the intelligent interference strategy is optimized by adjusting the weight coefficient of the multi-objective reward function and the network learning parameters.

[0037] In an optional embodiment,

[0038] The steps of constructing a multi-objective reward function including jamming effect, resource consumption and jamming concealment, and optimizing the preliminary jamming strategy by using the gradient projection method to obtain the final intelligent jamming strategy include:

[0039] Construct a multi-objective reward function including an interference effect reward function, a resource consumption penalty term, and an interference concealment reward function, wherein the interference effect reward function is calculated based on the deviation of the spatial activity range index, the motion characteristic index deviation, and the behavior characteristic index deviation of the target before and after the interference, the resource consumption penalty term is calculated based on the transmission power and energy constraint of the interference device, and the interference concealment reward function is calculated based on the Euclidean distance and mutual information between the target signal feature vector and the interference signal feature vector;

[0040] Obtain a reward gradient for the parameters of the preliminary interference strategy using the multi-objective reward function, construct a Lagrangian function based on the reward gradient, the Lagrangian function includes a reward function term and a constraint penalty term, and optimize the preliminary interference strategy using a gradient projection method. In each iteration, firstly calculate the first-order derivative and the second-order derivative of the Lagrangian function for the interference strategy parameters, construct a quadratic approximation model, solve the optimal descent direction based on the quadratic approximation model, update the interference strategy parameters along the optimal descent direction, and project the updated interference strategy parameters to a parameter space that satisfies the transmission power and energy constraints.

[0041] The optimization step size is dynamically adjusted based on the constraint violation degree. The constraint violation degree is calculated by the violation degree vector of the transmission power and energy constraints. The optimization step size decays as the constraint violation degree increases. The final intelligent interference strategy is obtained through iterative optimization.

[0042] In an optional embodiment,

[0043] The intelligent jamming strategy is input into a waveform generator to generate a corresponding jamming signal, and an adaptive beamforming algorithm is used to dynamically adjust the direction angle and the pitch angle of the jamming beam. The steps of transmitting the jamming signal to the target include:

[0044] Mapping the power allocation vector of the intelligent interference strategy into a subcarrier energy allocation matrix, constructing an orthogonal subcarrier set based on the subcarrier energy allocation matrix, and performing energy weighting and pseudo-random phase modulation on the orthogonal subcarrier set to obtain an interference signal complex envelope;

[0045] Constructing a spatial correlation matrix based on the complex envelope of the interference signal, constructing an array manifold vector according to the array element spacing and the carrier wavelength of the antenna array, establishing a weight optimization objective function including a beam pointing constraint term and a spatial correlation matrix term in combination with the spatial correlation matrix and the array manifold vector, and solving the weight optimization objective function to obtain an initial value of the antenna array weight vector;

[0046] The spatial correlation matrix is ​​inverted to obtain an anti-correlation matrix; a recursive least squares algorithm is used to update the antenna array weight vector in real time, an error signal is calculated based on the antenna array input vector and the expected response signal, the anti-correlation matrix is ​​updated according to the error signal, and the anti-correlation matrix is ​​used to dynamically optimize the antenna array weight vector to achieve adaptive adjustment of the directional angle and elevation angle of the interference beam.

[0047] According to a second aspect of the embodiments of the present invention,

[0048] Provide wireless signal and thermal imaging multi-modal collaborative detection and directional jamming system, including:

[0049] The first unit is used to perform time domain synchronization and spatial domain registration on the wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor to obtain a multi-source data set aligned in time and space, perform feature decomposition on the wireless signal data of the multi-source data set through wavelet transform and extract target feature information, perform target segmentation and feature extraction on the infrared thermal imaging data, and obtain multi-modal feature data of the target;

[0050] The second unit is used to construct a multimodal feature fusion network, input the multimodal feature data into the multimodal feature fusion network, use the attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, input the weighted features into a recursive neural network for time series modeling to obtain target time series features; input the target time series features into a multi-task learning framework, and output the position coordinates, motion state and behavior pattern information of the target;

[0051] The third unit is used to establish a target state evaluation model based on the position coordinates, motion state and behavior pattern information of the target, use a reinforcement learning algorithm to design an intelligent interference strategy, input the intelligent interference strategy into a waveform generator to generate a corresponding interference signal, and use an adaptive beamforming algorithm to dynamically adjust the azimuth angle and pitch angle of the interference beam to transmit an interference signal to the target.

[0052] According to a third aspect of the embodiments of the present invention,

[0053] An electronic device is provided, comprising:

[0054] processor;

[0055] a memory for storing processor-executable instructions;

[0056] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0057] A fourth aspect of the embodiments of the present invention is:

[0058] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0059] The present invention integrates the target feature information extracted by wireless signals and wavelet transform with the target segmentation and feature extraction results of infrared thermal imaging to construct a multimodal feature fusion network. Through multimodal information fusion and attention mechanism, the limitation of a single sensor affected by environmental factors can be effectively overcome, and the accuracy and robustness of target detection can be improved, especially in complex environments.

[0060] The present invention uses a recursive neural network to model the target's temporal characteristics, and outputs the target's position coordinates, motion state, and behavior pattern information through a multi-task learning framework. This makes the perception of the target state more comprehensive and in-depth, not only being able to obtain the target's position information, but also understanding the target's motion trend and behavior intention.

[0061] The present invention designs intelligent jamming strategies based on the target state assessment model and reinforcement learning algorithm, and dynamically adjusts the jamming beam through adaptive beamforming technology. This intelligent directional jamming method can adjust the jamming strategy according to the real-time state of the target, improve the effectiveness and pertinence of the jamming, and reduce the consumption of jamming energy. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 It is a flowchart of a method for multi-modal cooperative detection and directional interference of wireless signals and thermal imaging according to an embodiment of the present invention;

[0063] Figure 2 It is a structural schematic diagram of a wireless signal and thermal imaging multi-modal collaborative detection and directional interference system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0064] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0065] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0066] Figure 1 FIG. 1 is a flow chart of a method for multi-modal cooperative detection and directional interference of wireless signals and thermal imaging according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0067] S1. Perform time domain synchronization and spatial domain registration on the wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor to obtain a multi-source data set aligned in time and space, perform feature decomposition on the wireless signal data of the multi-source data set through wavelet transform and extract target feature information, perform target segmentation and feature extraction on the infrared thermal imaging data, and obtain multi-modal feature data of the target;

[0068] S2. Construct a multimodal feature fusion network, input the multimodal feature data into the multimodal feature fusion network, use an attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, input the weighted features into a recursive neural network for time series modeling to obtain target time series features; input the target time series features into a multi-task learning framework, and output the target's position coordinates, motion state, and behavior pattern information;

[0069] S3. A target state assessment model is established based on the position coordinates, motion state and behavior pattern information of the target, and an intelligent interference strategy is designed using a reinforcement learning algorithm. The intelligent interference strategy is input into a waveform generator to generate a corresponding interference signal, and an adaptive beamforming algorithm is used to dynamically adjust the azimuth angle and pitch angle of the interference beam to transmit an interference signal to the target.

[0070] In an optional embodiment,

[0071] The steps of performing time domain synchronization and spatial domain registration on the wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor to obtain a multi-source data set aligned in time and space include:

[0072] The wireless signal data and infrared thermal imaging data are aligned in the time domain based on the sliding alignment algorithm of the time window. The wireless signal data and infrared thermal imaging data at the same time are determined by calculating the timestamp difference of adjacent data frames, and the time alignment label is generated.

[0073] Perform spatial feature extraction on the wireless signal data, construct a signal covariance matrix using the antenna array manifold characteristics, perform feature decomposition on the signal covariance matrix to obtain a signal subspace, estimate the signal arrival angle based on a multiple signal classification algorithm to obtain a wireless signal spatial feature vector; perform spatial feature extraction on the infrared thermal imaging data, extract the target bounding box coordinates using a target detection neural network, calculate the azimuth and pitch angle of the target in the image plane, and perform angle conversion based on camera calibration parameters to obtain an infrared thermal imaging spatial feature vector;

[0074] Based on the time alignment tag, the wireless signal space feature vector and the infrared thermal imaging space feature vector, spatial registration is performed, a spherical coordinate system with the center of the antenna array as the origin is established, the wireless signal space feature vector and the infrared thermal imaging space feature vector are converted to the spherical coordinate system, the azimuth angle difference and the elevation angle difference of the wireless signal space feature vector and the infrared thermal imaging space feature vector are calculated, and the registration is completed when the azimuth angle difference and the elevation angle difference are less than a preset angle threshold, so as to obtain a feature pair aligned in time and space;

[0075] The signal-to-noise ratio value of the wireless signal is calculated, the infrared thermal imaging target detection probability value is evaluated, and the spatiotemporally aligned feature pairs whose signal-to-noise ratio value is greater than the signal-to-noise ratio threshold and whose infrared thermal imaging target detection probability value is greater than the target detection threshold are formed into a multi-source data set.

[0076] Exemplarily, in order to more effectively fuse wireless signals and infrared thermal imaging data, this embodiment provides an accurate spatiotemporal alignment method, and thereby constructs a high-quality multi-source dataset.

[0077] First, time synchronization is performed. A sliding alignment algorithm based on a time window is used. A time window is set, for example, 50 milliseconds. Timestamp information is extracted from the wireless signal data stream and the infrared thermal imaging data stream respectively. For example, the timestamps of the wireless signal data stream are 1000 milliseconds, 1001 milliseconds, 1002 milliseconds, etc., and the timestamps of the infrared thermal imaging data stream are 998 milliseconds, 1003 milliseconds, 1008 milliseconds, etc. Taking the wireless signal data stream as a reference, the corresponding data frames whose timestamp difference falls within the time window are searched in the infrared thermal imaging data stream. For example, taking the 1000 millisecond data frame of the wireless signal as an example, the data frames whose timestamp difference falls within 50 milliseconds are searched in the infrared data stream. The infrared data frame with a timestamp of 998 milliseconds is found, and its time difference is 2 milliseconds, which falls within the time window. Therefore, the 1000 millisecond data frame of the wireless signal and the 998 millisecond data frame of the infrared thermal imaging are marked as time-aligned frame pairs, and the time alignment label "1000-998" is generated. Similarly, all data frames are aligned in time window sliding mode to generate a series of time alignment labels.

[0078] Next, perform spatial feature extraction. For wireless signals, the manifold characteristics of the antenna array are used. First, construct a signal covariance matrix to reflect the distribution of signal energy among different antennas. Then perform eigendecomposition on the covariance matrix to obtain the signal subspace. Based on the multiple signal classification algorithm, estimate the signal arrival angle, for example, the azimuth is 30 degrees and the elevation is 15 degrees, forming a wireless signal spatial feature vector (30, 15). For infrared thermal imaging data, a target detection neural network is used. For example, the YOLOv5 network is used to extract the coordinate information of the target bounding box. Assume that a target is detected and the pixel coordinates of the center of its bounding box are (100, 200). Combined with camera calibration parameters, such as focal length, image principal point coordinates, etc., the pixel coordinates are converted to azimuth and elevation in the camera coordinate system. Assume that the target azimuth is calculated to be 32 degrees and the elevation is 12 degrees, forming an infrared thermal imaging spatial feature vector (32, 12).

[0079] Then, perform spatial alignment. Establish a spherical coordinate system with the center of the antenna array as the origin. Convert the wireless signal spatial feature vector (30, 15) and the infrared thermal imaging spatial feature vector (32, 12) to the spherical coordinate system. Calculate the azimuth difference and elevation difference of the two spatial feature vectors. For example, the azimuth difference is 32-30=2 degrees, and the elevation difference is 15-12=3 degrees. Preset an angle threshold, for example 5 degrees. Since both 2 degrees and 3 degrees are less than the threshold of 5 degrees, it is considered that the two feature vectors have completed the alignment, and the spatiotemporal aligned feature pair (30, 15)-(32, 12) and the corresponding time alignment label "1000-998" are obtained.

[0080] Finally, construct a multi-source dataset. Calculate the signal-to-noise ratio of the wireless signal, for example, the calculated signal-to-noise ratio is 10 dB. Evaluate the infrared thermal imaging target detection probability value, for example, the target detection probability output by the YOLOv5 network is 0.9. Preset the signal-to-noise ratio threshold, for example, 5 dB, and preset the target detection threshold, for example, 0.8. Since 10 dB is greater than 5 dB, and 0.9 is greater than 0.8, the spatiotemporal aligned feature pair (30, 15)-(32, 12) and its corresponding time alignment label "1000-998" are added to the multi-source dataset. Repeat the above steps, process all data frames, and finally obtain a complete spatiotemporal aligned multi-source dataset.

[0081] The present invention ensures the temporal and spatial consistency of wireless signals and infrared thermal imaging data through precise spatiotemporal alignment, thereby improving the accuracy and reliability of subsequent data fusion; combining the arrival angle information of wireless signals and the target visual information of infrared thermal imaging can more comprehensively describe the target characteristics and enhance the target recognition and tracking capabilities; the constructed spatiotemporal aligned multi-source data set can be applied to a wider range of scenarios, such as security monitoring, environmental monitoring, autonomous driving and other fields, to improve system performance and intelligence level.

[0082] In an optional embodiment,

[0083] The steps of constructing a multimodal feature fusion network, inputting the multimodal feature data into the multimodal feature fusion network, using an attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, and inputting the weighted features into a recursive neural network for time series modeling to obtain target time series features include:

[0084] Perform multi-scale decomposition on the multimodal feature data to obtain physical layer features, representation layer features and semantic layer features, wherein the physical layer features include spatial position and orientation information, the representation layer features include signal quality and gradient information, and the semantic layer features include spectrum and depth feature information;

[0085] Construct a multimodal feature fusion network, the multimodal feature fusion network includes a feature projection module and a feature constraint module, wherein the feature projection module includes a wireless signal feature projection unit and an infrared image feature projection unit, the multimodal feature data are respectively input into the corresponding feature projection unit for feature space mapping, the similarity matrix of the mapped features is calculated, and the cross-modal feature weights are generated based on the similarity matrix; the feature constraint module introduces a space transformation network at the physical layer to calculate the spatial consistency constraint, introduces a temporal smoothing factor at the representation layer to calculate the temporal continuity constraint, calculates the state consistency constraint at the semantic layer through adversarial learning, and integrates the spatial consistency constraint, the temporal continuity constraint and the state consistency constraint based on the cross-modal feature weight to generate the initial fusion feature;

[0086] An attention mechanism is used to adaptively assign weights to different modal features in the initial fusion features, wherein the attention mechanism includes a long-range feature extraction branch and a short-range feature extraction branch, wherein the long-range feature extraction branch captures the global dependency of the target motion, and the short-range feature extraction branch captures the local features of the target state, and an adaptive weight is calculated based on the statistics of the long-range features and the short-range features, and a weighted feature is obtained based on the adaptive weight;

[0087] The weighted features are input into a recursive neural network, which includes a feature encoding module and a timing decoding module. The input features are encoded by the feature encoding module, and the timing decoding module is used to extract the timing dependency to obtain the timing features. The timing features are weighted and fused to output the target timing features.

[0088] Exemplarily, first, the data collected by the multimodal sensor is preprocessed, including data cleaning, normalization, and format conversion. Taking wireless signals and infrared images as examples, the wireless signal data may contain received signal strength indication (RSSI), channel state information (CSI), etc., and the infrared image data is a pixel matrix. Here, it is assumed that the wireless signal data contains the RSSI values ​​of three channels, and the infrared image data is a grayscale image of 64x64 pixels. After preprocessing, the wireless signal data is converted into a time series, and the infrared image data is converted into a two-dimensional matrix.

[0089] Next, the preprocessed multimodal feature data is decomposed at multiple scales. The wireless signal data is divided into time windows, and the statistical features (such as mean, variance, maximum value, minimum value, etc.) in each window are extracted as physical layer features to reflect the spatial position and orientation information of the signal. For example, the direction of the signal source can be estimated by calculating the difference in RSSI values ​​of different receivers. The feature changes in adjacent time windows are used as representation layer features to reflect signal quality and gradient information. For example, rapid changes in RSSI values ​​may indicate signal occlusion or environmental changes. The wireless signal data is subjected to spectral analysis, and frequency domain features are extracted as semantic layer features to reflect the spectral information of the signal. For infrared image data, the pixel position information in the image is used as the physical layer feature, the edge gradient information of the image is used as the representation layer feature, and the features extracted by the pre-trained deep convolutional network of the image are used as semantic layer features to reflect the deep feature information of the image.

[0090] Then, a multimodal feature fusion network is constructed. The network includes a feature projection module and a feature constraint module. In the feature projection module, the wireless signal feature projection unit and the infrared image feature projection unit are used to map the multimodal feature data to a common feature space. For example, a multi-layer perceptron (MLP) can be used for feature projection. The similarity matrix of the mapped features is calculated, for example, using cosine similarity. The cross-modal feature weights are generated based on the similarity matrix, for example, the similarity is converted into weights using a softmax function. In the feature constraint module, a spatial transformation network is introduced at the physical layer to calculate the spatial consistency constraint. For example, the infrared image can be spatially aligned according to the spatial position information of the wireless signal. A temporal smoothing factor is introduced at the representation layer to calculate the temporal continuity constraint. For example, the feature can be smoothed using an exponentially weighted moving average. The state consistency constraint is calculated at the semantic layer through adversarial learning. For example, a generative adversarial network (GAN) can be used to learn the common representation of cross-modal features. The initial fusion feature is generated by integrating the spatial consistency constraint, the temporal continuity constraint, and the state consistency constraint based on the cross-modal feature weight.

[0091] After that, the attention mechanism is used to adaptively assign weights to the initial fusion features. The mechanism includes a long-range feature extraction branch and a short-range feature extraction branch. The long-range feature extraction branch uses, for example, a long short-term memory network (LSTM) to capture the global dependencies of the target motion. The short-range feature extraction branch uses, for example, a convolutional neural network (CNN) to capture the local features of the target state. The adaptive weights are calculated based on the statistics (such as mean and variance) of the long-range features and the short-range features, for example, by using a sigmoid function to convert the statistics into weights. The initial fusion features are weighted based on the adaptive weights to obtain weighted features.

[0092] Finally, the weighted features are input into the recursive neural network. The network includes a feature encoding module and a time series decoding module. The feature encoding module uses, for example, a recurrent neural network (RNN) to encode the input features. The time series decoding module uses, for example, LSTM to extract the time series dependency to obtain the time series features. The time series features are weighted fused, for example, using weighted average, and the target time series features are output.

[0093] Through multi-scale decomposition and cross-modal feature fusion, the present invention can more comprehensively capture the feature information of the target and improve the feature representation capability; through feature constraints and attention mechanisms, it can effectively suppress the influence of noise and interference and enhance the robustness of the model; through time series modeling and weighted fusion, it can more accurately extract the time series features of the target and improve the recognition accuracy.

[0094] In an optional embodiment,

[0095] The step of inputting the target time series features into a multi-task learning framework and outputting the target's position coordinates, motion state, and behavior pattern information comprises:

[0096] Input the target temporal features into a shared encoder to extract a universal feature representation, and the shared encoder uses a task-specific decoder to perform feature decoding for position coordinate estimation, motion state discrimination, and behavior pattern recognition, respectively; perform spatiotemporal feature enhancement on the universal feature representation, use a self-attention mechanism in the time dimension to calculate the temporal feature dependency to generate a temporal attention weight, and introduce a spatial attention module in the spatial dimension to capture local spatial correlations to obtain enhanced features;

[0097] The enhanced features are input into a multi-scale feature pyramid network, features of different scales are fused through a residual connection structure, position prediction is optimized using a cascade regression method, and real-time position coordinate information of the target is output;

[0098] A temporal convolutional network is used to extract target speed and acceleration features, and the speed and acceleration features and real-time position coordinate information in a continuous time period are input into a bidirectional gated recurrent unit network, and the motion state information of the target is identified through a multi-layer perceptron;

[0099] A temporal graph structure is constructed based on real-time position coordinate information and motion state information. The nodes of the graph structure represent the state of the target at a certain moment, and the edges represent the temporal conversion relationship of the state. A temporal graph convolutional network is used to extract the behavioral features in the graph structure, and the behavioral pattern information of the target is output through graph pooling and fully connected layers.

[0100] Exemplarily, first, the time series features of the target are input into a shared encoder. This shared encoder can be a convolutional neural network, which is used to extract a universal feature representation of the target. Assume that the target time series feature is a time series of length 100, and the feature dimension of each time step is 512. The shared encoder converts this 100x512 feature sequence into a fixed-length feature vector, such as 1x2048.

[0101] Next, the general feature representation is enhanced with spatiotemporal features. In the temporal dimension, the self-attention mechanism is used to capture the dependencies between time steps. Specifically, the similarity between each time step and other time steps is calculated to generate temporal attention weights. Then, the features are weighted and summed according to these weights to obtain enhanced features in the temporal dimension. For example, for a 1x2048 feature vector, the self-attention mechanism may pay more attention to certain specific time segments, thereby enhancing the feature representation of these segments. In the spatial dimension, a spatial attention module is introduced to capture local spatial correlations. This module calculates the similarity between each position in the feature map and other positions, and weights the features according to the similarity to obtain enhanced features in the spatial dimension. Assuming that the position information of the target in the image is also included in the features, the spatial attention module can enhance the features of the area around the target and suppress the features of irrelevant areas.

[0102] Then, the enhanced features are input into a multi-scale feature pyramid network. This network can be a FPN (Feature Pyramid Network), which fuses features of different scales through a residual connection structure. Position prediction is performed at different scales, and the prediction results are gradually optimized by cascade regression, and the real-time position coordinate information of the target is finally output. For example, FPN can perform position prediction on feature maps with 1 / 4, 1 / 8 and 1 / 16 resolutions respectively, and then these prediction results are fused to obtain more accurate position coordinates. Assuming that the position of the target in the image is (100, 200), the target position coordinates output by this step are (100, 200).

[0103] Next, a temporal convolutional network is used to extract the speed and acceleration features of the target. For example, a network containing multiple one-dimensional convolutional layers and pooling layers can be used to extract these features. The extracted speed and acceleration features and the real-time position coordinate information in a continuous time period are input into a bidirectional gated recurrent unit network (Bi-GRU). Bi-GRU can learn the temporal laws of target motion and identify the target's motion state information, such as stillness, uniform motion, accelerated motion, etc., through a multi-layer perceptron. Assuming that the target moves to the right at a uniform speed over a period of time, the motion state information output by this step is "uniform motion".

[0104] Finally, a time-series graph structure is constructed based on the real-time position coordinate information and motion state information. The nodes of the graph structure represent the states of the target at different times, and the edges represent the temporal conversion relationship between states. For example, the target position and motion state of each time step can be taken as a node, and the nodes of adjacent time steps can be connected to form an edge. Then, a temporal graph convolutional network (TGCN) is used to extract behavioral features in the graph structure. TGCN can learn the relationship between nodes and between nodes and edges to capture the pattern of target behavior. The graph structure is converted into a fixed-size feature vector through a graph pooling operation, and the target's behavior pattern information, such as walking, running, jumping, etc., is output through a fully connected layer. Assuming that the target walks at a constant speed for a period of time and then accelerates to run, the behavior pattern information output by this step may be "walking first and then running".

[0105] The present invention can effectively improve the accuracy of target position, motion state and behavior pattern prediction through a multi-task learning framework and spatiotemporal feature enhancement. The shared encoder and multi-scale feature pyramid network can learn richer feature representations, while the self-attention mechanism and spatial attention module can capture finer-grained spatiotemporal dependencies, thereby improving prediction accuracy; the multi-task learning framework can enable the model to learn common features between different tasks, thereby improving the generalization ability of the model. Even in the case of insufficient training data, the model can better adapt to new scenarios and targets; the target's position, motion state and behavior pattern can be learned simultaneously, avoiding redundant calculations and information loss caused by learning each task separately. Through shared encoders and multi-level feature fusion, the correlation between different tasks can be effectively utilized to improve learning efficiency and prediction performance.

[0106] In an optional embodiment,

[0107] The steps of establishing a target state evaluation model based on the position coordinates, motion state and behavior pattern information of the target and using a reinforcement learning algorithm to design an intelligent interference strategy include:

[0108] Based on the position coordinates, motion state and behavior pattern information of the target, construct a multidimensional evaluation index including a spatial activity range index, a motion characteristic index and a behavior characteristic index; construct a target state evaluation model, map the multidimensional evaluation index into a hierarchical Markov decision process, the state space is composed of the multidimensional evaluation index, the action space includes the interference strategy type and the interference parameter configuration, the interference strategy type is selected in the top-level decision process, and the interference parameter configuration is determined in the bottom-level decision process;

[0109] A dual Q network is used for interference strategy learning. The dual Q network includes a current network and a target network. The current network evaluates the state-action value function to predict the interference strategy benefit. The target network generates a target Q value. The state-action transition sequence of the target in the decision-making process is stored through the experience replay mechanism to establish a sample pool. The sampling weight is set based on the temporal difference error to perform sample screening.

[0110] A hierarchical exploration mechanism is constructed based on the state-action value function, an ε-greedy strategy is adopted at the interference strategy type level, Gaussian noise is introduced at the interference parameter configuration level, and the exploration probability is dynamically adjusted according to the statistical variance of the multi-dimensional evaluation index to generate a preliminary interference strategy;

[0111] A multi-objective reward function including jamming effect, resource consumption and jamming concealment is constructed, and the preliminary jamming strategy is optimized by using a gradient projection method to obtain a final intelligent jamming strategy;

[0112] The final intelligent interference strategy is updated in real time by online learning, the interference strategy execution deviation is calculated based on the multi-dimensional evaluation index, and the intelligent interference strategy is optimized by adjusting the weight coefficient of the multi-objective reward function and the network learning parameters.

[0113] For example, first, the location coordinates, motion state, and behavior pattern information of the target are collected. For example, the location coordinates of the target can be obtained through GPS or other positioning systems, the motion state can include speed, acceleration, direction, etc., and the behavior pattern can include residence time, activity frequency, trajectory regularity, etc. Suppose we collect the activity data of a target within a week, including the location coordinates, movement speed, and residence time in different areas at different time periods every day.

[0114] Next, based on the collected information, construct multi-dimensional evaluation indicators. Spatial activity range indicators can include the target's activity area, the coordinates of the center point of the activity range, etc. Motion characteristic indicators can include the target's average speed, maximum speed, acceleration range, etc. Behavioral characteristic indicators can include the target's stay time ratio in different areas, the regularity of activities, etc. For example, based on the target's activity data for one week, it is calculated that its activity area is 10 square kilometers, the average speed is 5 kilometers per hour, and the stay time in area A accounts for 50%.

[0115] Then, a target state evaluation model is constructed. The above multi-dimensional evaluation indicators are mapped into a hierarchical Markov decision process. The state space is composed of multi-dimensional evaluation indicators. For example, a state can be expressed as {activity area: 10, average speed: 5, A area stay time ratio: 50%}. The action space includes interference strategy types and interference parameter configurations. Interference strategy types can include guidance, delay, obstruction, etc. Interference parameter configurations can include the intensity, duration, and range of interference. The top-level decision process selects the interference strategy type, such as selecting the "guidance" strategy. The bottom-level decision process determines the interference parameter configuration, such as determining that the intensity of guidance is level 3, the duration is 1 hour, and the range is 1 km around the current location of the target.

[0116] Next, a dual Q network is used to learn the interference strategy. The dual Q network includes the current network and the target network. The current network evaluates the state-action value function to predict the interference strategy benefit. The target network generates the target Q value. The state-action transition sequence of the target in the decision-making process is stored through the experience replay mechanism to establish a sample pool. The sampling weight is set based on the temporal difference error to perform sample screening. For example, the various states encountered by the target in the past week, the interference strategies adopted, and the benefits obtained are stored in the sample pool, and different sampling weights are assigned according to the size of the benefits.

[0117] Then, a hierarchical exploration mechanism is constructed based on the state-action value function. The ε-greedy strategy is adopted at the interference strategy type level, for example, the interference strategy type is randomly selected with a certain probability. Gaussian noise is introduced at the interference parameter configuration level, for example, random noise is added based on the determined interference parameters. The exploration probability is dynamically adjusted according to the statistical variance of the multi-dimensional evaluation indicators. For example, if the target's behavior pattern is relatively stable, the exploration probability is reduced; otherwise, the exploration probability is increased. Finally, a preliminary interference strategy is generated.

[0118] Next, a multi-objective reward function including interference effect, resource consumption and interference concealment is constructed. The interference effect can measure the degree of influence of the interference strategy on the target behavior. Resource consumption can measure the cost required for the interference strategy. Interference concealment can measure the probability that the interference strategy is detected by the target. For example, if a interference strategy successfully guides the target out of a specific area, the interference effect is high; if the strategy consumes a lot of resources, the resource consumption is high; if the strategy is not easily detected by the target, the interference concealment is high. The gradient projection method is used to optimize the preliminary interference strategy to obtain the final intelligent interference strategy.

[0119] Finally, the final intelligent jamming strategy is updated in real time using online learning. The execution deviation of the jamming strategy is calculated based on multi-dimensional evaluation indicators. For example, the difference between the actual behavior of the target and the expected behavior is compared. The intelligent jamming strategy is optimized by adjusting the weight coefficients of the multi-objective reward function and the network learning parameters. For example, if the concealment of the jamming strategy is found to be insufficient, the weight of concealment in the reward function is increased.

[0120] By constructing multi-dimensional evaluation indicators and a hierarchical Markov decision process, the present invention can more accurately evaluate the target state and select a more effective interference strategy, thereby improving the interference efficiency; by optimizing the multi-objective reward function, the resource consumption of the interference strategy can be reduced, and the cost can be reduced as much as possible while ensuring the interference effect; the hierarchical exploration mechanism and online learning method can make the interference strategy more flexible and adaptable, thereby enhancing the concealment of the interference and reducing the probability of being detected by the target.

[0121] In an optional embodiment,

[0122] The steps of constructing a multi-objective reward function including jamming effect, resource consumption and jamming concealment, and optimizing the preliminary jamming strategy by using the gradient projection method to obtain the final intelligent jamming strategy include:

[0123] Construct a multi-objective reward function including an interference effect reward function, a resource consumption penalty term, and an interference concealment reward function, wherein the interference effect reward function is calculated based on the deviation of the spatial activity range index, the motion characteristic index deviation, and the behavior characteristic index deviation of the target before and after the interference, the resource consumption penalty term is calculated based on the transmission power and energy constraint of the interference device, and the interference concealment reward function is calculated based on the Euclidean distance and mutual information between the target signal feature vector and the interference signal feature vector;

[0124] Obtain a reward gradient for the parameters of the preliminary interference strategy using the multi-objective reward function, construct a Lagrangian function based on the reward gradient, the Lagrangian function includes a reward function term and a constraint penalty term, and optimize the preliminary interference strategy using a gradient projection method. In each iteration, firstly calculate the first-order derivative and the second-order derivative of the Lagrangian function for the interference strategy parameters, construct a quadratic approximation model, solve the optimal descent direction based on the quadratic approximation model, update the interference strategy parameters along the optimal descent direction, and project the updated interference strategy parameters to a parameter space that satisfies the transmission power and energy constraints.

[0125] The optimization step size is dynamically adjusted based on the constraint violation degree. The constraint violation degree is calculated by the violation degree vector of the transmission power and energy constraints. The optimization step size decays as the constraint violation degree increases. The final intelligent interference strategy is obtained through iterative optimization.

[0126] Exemplarily, first, a multi-objective reward function is constructed. This function aims to comprehensively evaluate the effectiveness, resource consumption and concealment of the interference strategy. In the interference effect reward function part, the interference effect is measured by comparing the changes in the spatial activity range, motion characteristics and behavioral characteristics of the target before and after the interference. For example, the reduction of the target's activity range, the reduction of the movement speed, and the change of the behavior pattern all indicate that the interference effect is good. The spatial activity range can be quantified by the size of the target's activity area, the motion characteristics can be represented by the average speed or acceleration of the target, and the behavioral characteristics can be described by the entropy value of the target's behavior pattern. Assuming that the target's activity range before the interference is 10 square kilometers and is reduced to 5 square kilometers after the interference, it can be considered that the interference effect is good. In the resource consumption penalty term part, the transmission power and energy constraints of the interference device are considered. The greater the transmission power, the more energy consumption, and the greater the value of the penalty term. For example, when the transmission power is 100 watts, the value of the penalty term is 10, and when the transmission power is 50 watts, the value of the penalty term is 5. In the interference concealment reward function part, the concealment of the interference is measured by calculating the Euclidean distance and mutual information between the target signal feature vector and the interference signal feature vector. The larger the Euclidean distance, the smaller the mutual information, indicating that the difference between the interference signal and the target signal is greater and the interference is more hidden. For example, when the Euclidean distance is 10 and the mutual information is 0.1, the interference is relatively hidden; when the Euclidean distance is 1 and the mutual information is 0.9, the interference is easily detected.

[0127] Next, the gradient projection method is used to optimize the preliminary interference strategy. First, the reward gradient is calculated based on the multi-objective reward function for the parameters of the preliminary interference strategy, and the gradient indicates the direction in which the reward function value increases. Then, a Lagrangian function containing reward function terms and constraint penalty terms is constructed. The constraint penalty term is used to limit the interference strategy parameters within the allowed range, such as the transmission power and energy constraints. In each iterative optimization, the first-order and second-order derivatives of the Lagrangian function for the interference strategy parameters are first calculated to construct a quadratic approximation model. The first-order derivative represents the rate of change of the function value, and the second-order derivative represents the rate of change of the rate of change. Based on the quadratic approximation model, the optimal descent direction is solved, which makes the Lagrangian function value decrease the fastest. Then, the interference strategy parameters are updated along the optimal descent direction. Finally, the updated interference strategy parameters are projected into the parameter space that meets the transmission power and energy constraints to ensure the feasibility of the interference strategy. For example, if the updated transmission power exceeds the maximum limit, it is adjusted to the maximum limit value.

[0128] Finally, the optimization step size is dynamically adjusted based on the constraint violation degree. The constraint violation degree is calculated by the violation degree vector of the transmit power and energy constraints, reflecting the degree to which the current interference strategy parameters violate the constraints. The optimization step size decays as the constraint violation degree increases to avoid constraint violations. For example, if the constraint violation degree is large, the optimization step size is reduced to update the parameters with a smaller amplitude to prevent the parameters from exceeding the constraint range. Through iterative optimization, an intelligent interference strategy that satisfies the constraints and maximizes the value of the multi-objective reward function is finally obtained.

[0129] The present invention comprehensively considers the interference effect, resource consumption and concealment through the design of a multi-objective reward function, and can effectively improve the interference effect, such as reducing the target activity range, reducing the target movement speed, changing the target behavior pattern, etc.; by considering the transmission power and energy constraints, it can effectively reduce the resource consumption in the interference process, such as reducing the transmission power and energy consumption; by calculating the Euclidean distance and mutual information between the target signal feature vector and the interference signal feature vector, it can enhance the concealment of the interference and reduce the probability of being detected by the target.

[0130] In an optional embodiment,

[0131] The intelligent jamming strategy is input into a waveform generator to generate a corresponding jamming signal, and an adaptive beamforming algorithm is used to dynamically adjust the direction angle and the pitch angle of the jamming beam. The steps of transmitting the jamming signal to the target include:

[0132] Mapping the power allocation vector of the intelligent interference strategy into a subcarrier energy allocation matrix, constructing an orthogonal subcarrier set based on the subcarrier energy allocation matrix, and performing energy weighting and pseudo-random phase modulation on the orthogonal subcarrier set to obtain an interference signal complex envelope;

[0133] Constructing a spatial correlation matrix based on the complex envelope of the interference signal, constructing an array manifold vector according to the array element spacing and the carrier wavelength of the antenna array, establishing a weight optimization objective function including a beam pointing constraint term and a spatial correlation matrix term in combination with the spatial correlation matrix and the array manifold vector, and solving the weight optimization objective function to obtain an initial value of the antenna array weight vector;

[0134] The spatial correlation matrix is ​​inverted to obtain an anti-correlation matrix; a recursive least squares algorithm is used to update the antenna array weight vector in real time, an error signal is calculated based on the antenna array input vector and the expected response signal, the anti-correlation matrix is ​​updated according to the error signal, and the anti-correlation matrix is ​​used to dynamically optimize the antenna array weight vector to achieve adaptive adjustment of the directional angle and elevation angle of the interference beam.

[0135] Exemplarily, first, the intelligent interference strategy is converted into a signal that can be used for waveform generation. The intelligent interference strategy is usually given in the form of a power allocation vector, which indicates the interference power allocation on different frequency bands. The power allocation vector is mapped to a subcarrier energy allocation matrix, which represents the energy size on each subcarrier. For example, for a system containing 128 subcarriers, the power allocation vector can be [0.1, 0.2, ..., 0.05], and the mapped subcarrier energy allocation matrix is ​​a 128-dimensional diagonal matrix, and the diagonal elements correspond to the values ​​of the power allocation vector. Then, a set of orthogonal subcarriers is constructed based on the subcarrier energy allocation matrix. For example, orthogonal frequency division multiplexing (OFDM) technology can be used to generate an orthogonal subcarrier set. The orthogonal subcarrier set is energy-weighted, that is, each subcarrier is weighted according to the corresponding value in the subcarrier energy allocation matrix. Afterwards, the weighted subcarriers are pseudo-randomly phase modulated, for example, each subcarrier can be phase modulated using a uniformly distributed random number between 0 and 2π. Finally, these modulated subcarriers are superimposed to obtain the complex envelope of the interference signal.

[0136] Next, the initial value of the antenna array weight vector is calculated based on the generated interference signal and antenna array parameters. First, a spatial correlation matrix is ​​constructed based on the complex envelope of the interference signal. The spatial correlation matrix describes the correlation between the interference signals received by different array elements. For example, the covariance matrix of the complex envelope of the interference signal on different array elements can be calculated as the spatial correlation matrix. Then, an array manifold vector is constructed based on the array element spacing and carrier wavelength of the antenna array. The array manifold vector describes the phase difference when the signal arrives at different array elements. For example, for a uniform linear array, the array manifold vector can be expressed as a complex exponential vector, in which the phase of each element is related to the array element position and the signal arrival angle. After that, a weight optimization objective function is established by combining the spatial correlation matrix and the array manifold vector. The objective function contains two items: beam pointing constraint term and spatial correlation matrix term. The beam pointing constraint term is used to ensure that the interference beam points to the target direction, and the spatial correlation matrix term is used to minimize the energy of the interference signal in other directions. Finally, by solving the objective function, the initial value of the antenna array weight vector is obtained. For example, the minimum variance distortionless response (MVDR) algorithm can be used to solve the objective function.

[0137] Finally, the antenna array weight vector is updated in real time to achieve adaptive adjustment of the interference beam. First, the spatial correlation matrix is ​​inverted to obtain the anti-correlation matrix. Then, the recursive least squares (RLS) algorithm is used to update the antenna array weight vector in real time. The RLS algorithm is an iterative algorithm that can continuously update the weight vector based on new observation data. In each iteration, the error signal is first calculated based on the antenna array input vector and the expected response signal. The expected response signal represents the ideal interference signal. For example, the signal in the target direction can be set to 1 and the signal in other directions can be set to 0. Then, the anti-correlation matrix is ​​updated according to the error signal. Finally, the antenna array weight vector is dynamically optimized using the updated anti-correlation matrix. Through continuous iterative updates, the directional angle and elevation angle of the interference beam can be adaptively adjusted so that it is always aligned with the target.

[0138] The present invention uses adaptive beamforming technology to concentrate interference energy in the target direction, reduce the waste of interference energy, and thus improve the interference efficiency; the adaptive beamforming technology can adjust the direction of the interference beam in real time according to the change of the target position, thereby enhancing the robustness of the interference; the intelligent interference strategy is directly converted into waveform generation parameters, which simplifies the system design and reduces the difficulty of implementation.

[0139] Figure 2 FIG. 1 is a schematic diagram of the structure of a wireless signal and thermal imaging multi-modal cooperative detection and directional interference system according to an embodiment of the present invention. Figure 2 As shown, the system comprises:

[0140] The first unit is used to perform time domain synchronization and spatial domain registration on the wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor to obtain a multi-source data set aligned in time and space, perform feature decomposition on the wireless signal data of the multi-source data set through wavelet transform and extract target feature information, perform target segmentation and feature extraction on the infrared thermal imaging data, and obtain multi-modal feature data of the target;

[0141] The second unit is used to construct a multimodal feature fusion network, input the multimodal feature data into the multimodal feature fusion network, use the attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, input the weighted features into a recursive neural network for time series modeling to obtain target time series features; input the target time series features into a multi-task learning framework, and output the position coordinates, motion state and behavior pattern information of the target;

[0142] The third unit is used to establish a target state evaluation model based on the position coordinates, motion state and behavior pattern information of the target, use a reinforcement learning algorithm to design an intelligent interference strategy, input the intelligent interference strategy into a waveform generator to generate a corresponding interference signal, and use an adaptive beamforming algorithm to dynamically adjust the azimuth angle and pitch angle of the interference beam to transmit an interference signal to the target.

[0143] According to a third aspect of the embodiments of the present invention,

[0144] An electronic device is provided, comprising:

[0145] processor;

[0146] a memory for storing processor-executable instructions;

[0147] The processor is configured to call the instructions stored in the memory to execute the aforementioned method.

[0148] A fourth aspect of the embodiments of the present invention is:

[0149] A computer-readable storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the aforementioned method is implemented.

[0150] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A wireless signal and thermal imaging multi-modal collaborative detection and directional interference method, characterized in that: include: The wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor are synchronized in the time domain and registered in the space domain to obtain a multi-source data set aligned in time and space, the wireless signal data of the multi-source data set is feature decomposed by wavelet transform and target feature information is extracted, the infrared thermal imaging data is segmented and feature extracted to obtain multi-modal feature data of the target; Constructing a multimodal feature fusion network, inputting the multimodal feature data into the multimodal feature fusion network, using an attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, and inputting the weighted features into a recursive neural network for time series modeling to obtain target time series features; Input the target time series features into a multi-task learning framework, and output the target's position coordinates, motion state, and behavior pattern information; A target state assessment model is established based on the position coordinates, motion state and behavior pattern information of the target, and an intelligent interference strategy is designed using a reinforcement learning algorithm. The intelligent interference strategy is input into a waveform generator to generate a corresponding interference signal, and an adaptive beamforming algorithm is used to dynamically adjust the azimuth angle and pitch angle of the interference beam to transmit an interference signal to the target.

2. The method according to claim 1, characterized in that The steps of performing time domain synchronization and spatial domain registration on the wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor to obtain a multi-source data set aligned in time and space include: The wireless signal data and infrared thermal imaging data are aligned in the time domain based on the sliding alignment algorithm of the time window. The wireless signal data and infrared thermal imaging data at the same time are determined by calculating the timestamp difference of adjacent data frames, and the time alignment label is generated. Perform spatial feature extraction on the wireless signal data, construct a signal covariance matrix using the antenna array manifold characteristics, perform feature decomposition on the signal covariance matrix to obtain a signal subspace, estimate the signal arrival angle based on a multiple signal classification algorithm to obtain a wireless signal spatial feature vector; perform spatial feature extraction on the infrared thermal imaging data, extract the target bounding box coordinates using a target detection neural network, calculate the azimuth and pitch angle of the target in the image plane, and perform angle conversion based on camera calibration parameters to obtain an infrared thermal imaging spatial feature vector; Based on the time alignment tag, the wireless signal space feature vector and the infrared thermal imaging space feature vector, spatial registration is performed, a spherical coordinate system with the center of the antenna array as the origin is established, the wireless signal space feature vector and the infrared thermal imaging space feature vector are converted to the spherical coordinate system, the azimuth angle difference and the elevation angle difference of the wireless signal space feature vector and the infrared thermal imaging space feature vector are calculated, and the registration is completed when the azimuth angle difference and the elevation angle difference are less than a preset angle threshold, so as to obtain a feature pair aligned in time and space; The signal-to-noise ratio value of the wireless signal is calculated, the infrared thermal imaging target detection probability value is evaluated, and the spatiotemporally aligned feature pairs whose signal-to-noise ratio value is greater than the signal-to-noise ratio threshold and whose infrared thermal imaging target detection probability value is greater than the target detection threshold are formed into a multi-source data set.

3. The method according to claim 1, characterized in that The steps of constructing a multimodal feature fusion network, inputting the multimodal feature data into the multimodal feature fusion network, using an attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, and inputting the weighted features into a recursive neural network for time series modeling to obtain target time series features include: Perform multi-scale decomposition on the multimodal feature data to obtain physical layer features, representation layer features and semantic layer features, wherein the physical layer features include spatial position and orientation information, the representation layer features include signal quality and gradient information, and the semantic layer features include spectrum and depth feature information; Construct a multimodal feature fusion network, the multimodal feature fusion network includes a feature projection module and a feature constraint module, wherein the feature projection module includes a wireless signal feature projection unit and an infrared image feature projection unit, the multimodal feature data are respectively input into the corresponding feature projection unit for feature space mapping, the similarity matrix of the mapped features is calculated, and the cross-modal feature weights are generated based on the similarity matrix; the feature constraint module introduces a space transformation network at the physical layer to calculate the spatial consistency constraint, introduces a temporal smoothing factor at the representation layer to calculate the temporal continuity constraint, calculates the state consistency constraint at the semantic layer through adversarial learning, and integrates the spatial consistency constraint, the temporal continuity constraint and the state consistency constraint based on the cross-modal feature weight to generate the initial fusion feature; An attention mechanism is used to adaptively assign weights to different modal features in the initial fusion features, wherein the attention mechanism includes a long-range feature extraction branch and a short-range feature extraction branch, wherein the long-range feature extraction branch captures the global dependency of the target motion, and the short-range feature extraction branch captures the local features of the target state, and an adaptive weight is calculated based on the statistics of the long-range features and the short-range features, and a weighted feature is obtained based on the adaptive weight; The weighted features are input into a recursive neural network, which includes a feature encoding module and a timing decoding module. The input features are encoded by the feature encoding module, and the timing decoding module is used to extract the timing dependency to obtain the timing features. The timing features are weighted and fused to output the target timing features.

4. The method according to claim 1, characterized in that: The step of inputting the target time series features into a multi-task learning framework and outputting the target's position coordinates, motion state, and behavior pattern information comprises: Input the target temporal features into a shared encoder to extract a universal feature representation, and the shared encoder uses a task-specific decoder to perform feature decoding for position coordinate estimation, motion state discrimination, and behavior pattern recognition, respectively; perform spatiotemporal feature enhancement on the universal feature representation, use a self-attention mechanism in the time dimension to calculate the temporal feature dependency to generate a temporal attention weight, and introduce a spatial attention module in the spatial dimension to capture local spatial correlations to obtain enhanced features; The enhanced features are input into a multi-scale feature pyramid network, features of different scales are fused through a residual connection structure, position prediction is optimized using a cascade regression method, and real-time position coordinate information of the target is output; A temporal convolutional network is used to extract target speed and acceleration features, and the speed and acceleration features and real-time position coordinate information in a continuous time period are input into a bidirectional gated recurrent unit network, and the motion state information of the target is identified through a multi-layer perceptron; A temporal graph structure is constructed based on real-time position coordinate information and motion state information. The nodes of the graph structure represent the state of the target at a certain moment, and the edges represent the temporal conversion relationship of the state. A temporal graph convolutional network is used to extract the behavioral features in the graph structure, and the behavioral pattern information of the target is output through graph pooling and fully connected layers.

5. The method according to claim 1, characterized in that The steps of establishing a target state evaluation model based on the position coordinates, motion state and behavior pattern information of the target and using a reinforcement learning algorithm to design an intelligent interference strategy include: Based on the position coordinates, motion state and behavior pattern information of the target, construct a multidimensional evaluation index including a spatial activity range index, a motion characteristic index and a behavior characteristic index; construct a target state evaluation model, map the multidimensional evaluation index into a hierarchical Markov decision process, the state space is composed of the multidimensional evaluation index, the action space includes the interference strategy type and the interference parameter configuration, the interference strategy type is selected in the top-level decision process, and the interference parameter configuration is determined in the bottom-level decision process; A dual Q network is used for interference strategy learning. The dual Q network includes a current network and a target network. The current network evaluates the state-action value function to predict the interference strategy benefit. The target network generates a target Q value. The state-action transition sequence of the target in the decision-making process is stored through the experience replay mechanism to establish a sample pool. The sampling weight is set based on the temporal difference error to perform sample screening. A hierarchical exploration mechanism is constructed based on the state-action value function, an ε-greedy strategy is adopted at the interference strategy type level, Gaussian noise is introduced at the interference parameter configuration level, the exploration probability is dynamically adjusted according to the statistical variance of the multi-dimensional evaluation index, and a preliminary interference strategy is generated; A multi-objective reward function including jamming effect, resource consumption and jamming concealment is constructed, and the preliminary jamming strategy is optimized by using a gradient projection method to obtain a final intelligent jamming strategy; The final intelligent interference strategy is updated in real time by online learning, the interference strategy execution deviation is calculated based on the multi-dimensional evaluation index, and the intelligent interference strategy is optimized by adjusting the weight coefficient of the multi-objective reward function and the network learning parameters.

6. The method according to claim 5, characterized in that The steps of constructing a multi-objective reward function including jamming effect, resource consumption and jamming concealment, and optimizing the preliminary jamming strategy by using the gradient projection method to obtain the final intelligent jamming strategy include: Construct a multi-objective reward function including an interference effect reward function, a resource consumption penalty term, and an interference concealment reward function, wherein the interference effect reward function is calculated based on the deviation of the spatial activity range index, the motion characteristic index deviation, and the behavior characteristic index deviation of the target before and after the interference, the resource consumption penalty term is calculated based on the transmission power and energy constraint of the interference device, and the interference concealment reward function is calculated based on the Euclidean distance and mutual information between the target signal feature vector and the interference signal feature vector; Obtain a reward gradient for the parameters of the preliminary interference strategy using the multi-objective reward function, construct a Lagrangian function based on the reward gradient, the Lagrangian function includes a reward function term and a constraint penalty term, and optimize the preliminary interference strategy using a gradient projection method. In each iteration, firstly calculate the first-order derivative and the second-order derivative of the Lagrangian function for the interference strategy parameters, construct a quadratic approximation model, solve the optimal descent direction based on the quadratic approximation model, update the interference strategy parameters along the optimal descent direction, and project the updated interference strategy parameters to a parameter space that satisfies the transmission power and energy constraints. The optimization step size is dynamically adjusted based on the constraint violation degree. The constraint violation degree is calculated by the violation degree vector of the transmission power and energy constraints. The optimization step size decays as the constraint violation degree increases. The final intelligent interference strategy is obtained through iterative optimization.

7. The method according to claim 1, characterized in that The intelligent jamming strategy is input into a waveform generator to generate a corresponding jamming signal, and an adaptive beamforming algorithm is used to dynamically adjust the direction angle and the pitch angle of the jamming beam. The steps of transmitting the jamming signal to the target include: Mapping the power allocation vector of the intelligent interference strategy into a subcarrier energy allocation matrix, constructing an orthogonal subcarrier set based on the subcarrier energy allocation matrix, and performing energy weighting and pseudo-random phase modulation on the orthogonal subcarrier set to obtain an interference signal complex envelope; Constructing a spatial correlation matrix based on the complex envelope of the interference signal, constructing an array manifold vector according to the array element spacing and the carrier wavelength of the antenna array, establishing a weight optimization objective function including a beam pointing constraint term and a spatial correlation matrix term in combination with the spatial correlation matrix and the array manifold vector, and solving the weight optimization objective function to obtain an initial value of the antenna array weight vector; The spatial correlation matrix is ​​inverted to obtain an anti-correlation matrix; a recursive least squares algorithm is used to update the antenna array weight vector in real time, an error signal is calculated based on the antenna array input vector and the expected response signal, the anti-correlation matrix is ​​updated according to the error signal, and the anti-correlation matrix is ​​used to dynamically optimize the antenna array weight vector to achieve adaptive adjustment of the directional angle and elevation angle of the interference beam.

8. A wireless signal and thermal imaging multi-modal cooperative detection and directional interference system, used to implement the method described in any one of claims 1 to 7, characterized in that: include: The first unit is used to perform time domain synchronization and spatial domain registration on the wireless signal data collected by the wireless signal receiving antenna array and the infrared thermal imaging data collected by the thermal imaging sensor to obtain a multi-source data set aligned in time and space, perform feature decomposition on the wireless signal data of the multi-source data set through wavelet transform and extract target feature information, perform target segmentation and feature extraction on the infrared thermal imaging data, and obtain multi-modal feature data of the target; The second unit is used to construct a multimodal feature fusion network, input the multimodal feature data into the multimodal feature fusion network, use an attention mechanism to adaptively assign weights to different modal features in the multimodal feature data to obtain weighted features, and input the weighted features into a recursive neural network for time series modeling to obtain target time series features; Input the target time series features into a multi-task learning framework, and output the target's position coordinates, motion state, and behavior pattern information; The third unit is used to establish a target state evaluation model based on the position coordinates, motion state and behavior pattern information of the target, use a reinforcement learning algorithm to design an intelligent interference strategy, input the intelligent interference strategy into a waveform generator to generate a corresponding interference signal, and use an adaptive beamforming algorithm to dynamically adjust the azimuth angle and pitch angle of the interference beam to transmit an interference signal to the target.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images