Radiation source signal behavior identification method and device, equipment and storage medium

Abstract

The invention relates to the technical field of signal processing, and discloses a radiation source signal behavior identification method, device and equipment and a storage medium, and the method comprises the steps: receiving radiation source signals of various modes, carrying out the preprocessing of the signals of different modes, and carrying out the alignment of the preprocessed signals through a timestamp, and obtaining a multi-mode data signal; inputting the multi-modal data signal into a multi-modal large model, separating the multi-modal data signal in an input layer, and performing feature extraction on the multi-modal data signal through a feature extraction layer of the multi-modal large model to obtain target feature information; transmitting the target feature information to a behavior decoding layer of a multi-modal large model to predict and classify the behavior of the radiation source signal, and introducing a knowledge graph to generate behavior information of the radiation source signal; evaluating the radiation source signal behavior information, and performing model optimization on the multi-modal large model according to an evaluation result; according to the invention, unmanned intervention automation of the whole process from signal processing to behavior identification is realized, and the operation efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of signal processing technology, and in particular to a radiation source signal behavior recognition method, device, equipment and storage medium. Background Art

[0002] With the rapid development of wireless communication and electronic warfare technology, radiation source identification in complex electromagnetic environments has become a key task in electronic countermeasures and spectrum management. Radiation source identification refers to identifying specific signal sources and their behavioral attributes through signal feature extraction and pattern analysis. Traditional methods mainly rely on signal processing and classification algorithms, but with the increase in signal complexity, especially in multimodal signals, traditional methods are difficult to meet the requirements of real-time, accuracy and intelligence. Summary of the invention

[0003] The purpose of the present invention is to solve the above-mentioned problem and to design a radiation source signal behavior recognition method, device, equipment and storage medium.

[0004] A first aspect of the present invention provides a method for identifying radiation source signal behavior, the method comprising:

[0005] Receive radiation source signals of multiple modes, pre-process the signals of different modes, and align the pre-processed signals using time stamps to obtain multi-modal data signals;

[0006] Inputting the multimodal data signal into the multimodal large model, separating the multimodal data signal using a dynamic separation technology at the input layer, extracting features from the multimodal data signal through a feature extraction layer of the multimodal large model, and obtaining target feature information;

[0007] The target feature information is transmitted to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, and the knowledge graph is introduced to combine the historical behavior, geographical location and mission scenario of the radiation source to generate the behavior information of the radiation source signal;

[0008] The generated radiation source signal behavior information is evaluated, and the multimodal large model is optimized based on the evaluation results.

[0009] Optionally, in a first implementation of the first aspect of the present invention, receiving radiation source signals of multiple modes, preprocessing signals of different modes, and obtaining multimodal data signals by aligning the preprocessed signals by timestamps, includes:

[0010] After receiving radiation source signals of multiple modes, the radiation source signals are preliminarily classified into slow frequency signals and fast frequency signals. The slow frequency signals are filtered using a low-pass filter to extract the slow frequency components. The transfer function is:

[0011]

[0012] Where N represents the filter order, a k represents the filter coefficient, s represents the complex frequency variable;

[0013] The dynamic filtering window is determined according to the frequency of different radiation source signals, and the slow frequency signal is modeled in time series using an autoregressive model to generate an audio signal:

[0014]

[0015] In the formula, X t represents the signal value at the current moment, p represents the model order, represents the model coefficient, ∈ t represents the random error term;

[0016] For fast frequency signals, a two-dimensional time-frequency spectrum is generated through short-time Fourier transform processing:

[0017]

[0018] Where STFT(t,f) represents the signal strength at time t and frequency f, x(τ) represents the input signal, τ represents the time variable, w(τ-t) represents the window function, t represents time, f represents frequency, and e represents the window function. -j2πfτ represents the contribution of the calculated signal at frequency f, j represents the imaginary unit and j 2 = -1;

[0019] Use gamma correction to improve the contrast and resolution of 2D time-spectrograms:

[0020]

[0021] In the formula, I in Represents the pixel value of the input image, I out Represents the pixel value of the output image, and γ represents the gamma value;

[0022] The preprocessed slow-frequency audio signal and fast-frequency time-frequency video signal are aligned according to the timestamps of the signals to obtain a multimodal data signal.

[0023] Optionally, in a second implementation of the first aspect of the present invention, the multimodal data signal is input into a multimodal large model, a dynamic separation technology is used at the input layer to separate the multimodal data signal, and features are extracted from the multimodal data signal through a feature extraction layer of the multimodal large model to obtain target feature information, including:

[0024] The multimodal data signal is input into the multimodal large model, and the multimodal data signal is preliminarily standardized at the input layer. Then, the standardized multimodal data signal is separated by dynamic separation technology based on the frequency of the signal. The filter group is set to be F = {F1, F2, ..., F m}, the signal after filtering is Y={Y1,Y2,…,Y m}, Where X i represents the standardized multimodal data signal, and * represents the convolution operation;

[0025] CNN-GRU is used to extract local features of time series signals from slow-frequency audio signals in the separated multimodal data signals and form slow-frequency feature representations. 3D-CNN is used to extract spatial and temporal features of video signals from fast-frequency time-frequency video signals in the separated multimodal data signals and generate deep representations of fast-frequency signals.

[0026] A multimodal self-attention mechanism is used for feature fusion, and the high-dimensional features of different modalities are unified into the same spatial dimension through a multi-layer perceptron to obtain the target feature information.

[0027] Optionally, in a third implementation of the first aspect of the present invention, the CNN-GRU is used to extract local features of the time series signal from the slow frequency audio signal in the separated multimodal data signal and form a slow frequency feature representation, and the 3D-CNN is used to extract spatial and temporal features of the video signal from the fast frequency time-frequency video signal in the separated multimodal data signal, and generate a deep representation of the fast frequency signal, including:

[0028] CNN is used to extract the local features of the time series signal from the slow-frequency audio signal in the separated multimodal data signal. CNN consists of multiple convolutional layers and pooling layers. The convolution operation is:

[0029]

[0030] In the formula, C i,j represents the value of the convolution layer output at position (i, j), K represents the size of the convolution kernel, and W k represents the weight of the convolution kernel at position k, A i+k,j+k Represents the value of the input signal at position (i+k,j+k);

[0031] Suppose the output after convolutional layer processing is C = {c1, c2, ..., c n}, c1 is passed as the initial input to the first time step of GRU. At each time step, GRU receives the current input feature vector and the hidden state of the previous time step, merges the forget gate and the input gate into an update gate, and merges the cell state and the hidden state to calculate the hidden state of the current time step. Repeat the calculation until all inputs are processed, and obtain a series of hidden states H = {h1,h,…,h n}, the slow frequency feature representation is obtained through the hidden state H;

[0032] 3D-CNN is used to extract the spatial and temporal features of the video signal from the fast-frequency time-frequency video signal in the separated multimodal data signal. The convolution operation is:

[0033]

[0034] Where D i,j,k represents the value of the output feature map at position (i, j, k), L, M, N represent the size of the 3D convolution kernel in three dimensions, V i+l,j+m,k+n represents the value of the input video signal at position (i+l,j+m,k+n, Wl,m,n represents the weight of the 3D convolution kernel at position l,,m,n;

[0035] The time-frequency domain features of the video signal are fused with the spatial and temporal features extracted by 3D-CNN by splicing to generate a deep representation of the fast-frequency signal.

[0036] Optionally, in a fourth implementation of the first aspect of the present invention, transmitting the target feature information to a behavior decoding layer of a multimodal large model to predict and classify the behavior of the radiation source signal includes:

[0037] Input the target feature information into the behavior decoding layer of the multimodal large model, where the behavior decoding layer is built based on Transformer. Transformer consists of an encoder and a decoder. The encoder is used to encode the input features, and the decoder is used to generate prediction results.

[0038] Use the Transformer encoder to encode the sequence of target feature information, obtain the encoded feature sequence by stacking multiple encoder layers, and perform feature fusion on the encoded feature sequence and the original target feature information. The encoder consists of multiple layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network.

[0039] The Transformer decoder is used to predict the behavior of the fused features. The output sequence of the decoder is obtained by stacking multiple decoder layers. The category with the highest probability in the output sequence is used as the final prediction result to obtain the behavior category of the radiation source signal. The decoder consists of multiple layers, each of which includes a multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network.

[0040] Optionally, in a fifth implementation of the first aspect of the present invention, the introduction of the knowledge graph combines the historical behavior, geographical location and task scenario of the radiation source to generate the behavior information of the radiation source signal, including:

[0041] Collect historical behavior data, geographic location information, and task scenario descriptions of radiation sources, and construct a knowledge graph, where nodes include at least radiation source entities, behavior category entities, geographic location entities, and task scenario entities, and edges represent the relationships between different entities;

[0042] By querying the knowledge graph, we can find the historical behaviors, geographic locations, and mission scenario information related to the current radiation source, and fuse the behavior category features output by the multimodal large model with the information features in the knowledge graph to obtain the fused feature vector.

[0043] Based on the fused feature vector, RNN is used to perform sequence modeling on the fused feature vector to generate the behavior information of the radiation source signal through RNN.

[0044] Optionally, in a sixth implementation of the first aspect of the present invention, the step of evaluating the generated radiation source signal behavior information and optimizing the multimodal large model according to the evaluation result includes:

[0045] Collecting a data set for evaluating the behavior information of radiation source signals, wherein the data set at least includes the actual radiation source signal behavior annotation and the behavior information generated by the multimodal large model;

[0046] A set of parameters of the multimodal large model is randomly generated as sparrow individuals to form an initial population. For each sparrow individual, its corresponding parameters are applied to the multimodal large model, and the fitness value is calculated using the data set.

[0047] Update the positions of the discoverer, joiner, and scout. The discoverer is responsible for finding food sources, the joiner follows the discoverer to find food, and the scout performs random searches at the edge of the population. The population size is 30, the parameter dimension is 10, the maximum number of iterations is 100, the warning value is 0.6, the safety value is 0.7, and the step control parameter is 0.8;

[0048] Repeat the iteration until the maximum number of iterations is reached. When the sparrow search algorithm ends, the optimal multimodal large model parameters are output.

[0049] A second aspect of the present invention provides a radiation source signal behavior identification device, the radiation source signal behavior identification device comprising:

[0050] A signal processing module is used to receive radiation source signals of multiple modes, pre-process the signals of different modes, and align the pre-processed signals using timestamps to obtain multi-modal data signals;

[0051] A feature extraction module, used to input the multimodal data signal into the multimodal large model, separate the multimodal data signal using a dynamic separation technology at the input layer, and extract features from the multimodal data signal through the feature extraction layer of the multimodal large model to obtain target feature information;

[0052] A behavior decoding module is used to transmit the target feature information to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, and introduce the knowledge graph to combine the historical behavior, geographical location and mission scenario of the radiation source to generate the behavior information of the radiation source signal;

[0053] The model optimization module is used to evaluate the generated radiation source signal behavior information and optimize the multi-modal large model based on the evaluation results.

[0054] The third aspect of the present invention provides a radiation source signal behavior identification device, which includes a memory and at least one processor, wherein the memory stores instructions; the at least one processor calls the instructions in the memory to enable the radiation source signal behavior identification device to perform each step of the radiation source signal behavior identification method as described in any one of the above items.

[0055] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and when the instructions are executed by a processor, the various steps of the radiation source signal behavior identification method as described in any of the above items are implemented.

[0056] In the technical solution provided by the present invention, by receiving radiation source signals of multiple modes, preprocessing the signals of different modes, and aligning the preprocessed signals with timestamps, a multimodal data signal is obtained; the multimodal data signal is input into a multimodal large model, the multimodal data signal is separated by a dynamic separation technology at the input layer, and the multimodal data signal is feature extracted by a feature extraction layer of the multimodal large model to obtain target feature information; the target feature information is transmitted to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, and the knowledge graph is introduced to combine the historical behavior, geographical location and task scenario of the radiation source to generate the behavior information of the radiation source signal; the generated radiation source signal behavior information is evaluated, and the multimodal large model is optimized according to the evaluation result; the present invention realizes unmanned automation of the entire process from signal processing to behavior recognition, thereby improving operation efficiency; and effectively copes with signal interference and characteristic changes in complex electromagnetic environments through multimodal information fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the following detailed description of the preferred embodiment.The drawings are only for the purpose of illustrating the preferred embodiments and are not to be construed as limiting the invention.

[0058] Figure 1 A schematic diagram of a first embodiment of a method for identifying radiation source signal behavior provided by an embodiment of the present invention;

[0059] Figure 2 A schematic diagram of a second embodiment of a method for identifying radiation source signal behavior provided by an embodiment of the present invention;

[0060] Figure 3 A schematic diagram of a third embodiment of a method for identifying radiation source signal behavior provided by an embodiment of the present invention;

[0061] Figure 4 A schematic diagram of the structure of a radiation source signal behavior identification device provided by an embodiment of the present invention;

[0062] Figure 5 A schematic diagram of the structure of a radiation source signal behavior recognition device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0063] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, device, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0064] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 A schematic diagram of a first embodiment of a method for identifying radiation source signal behavior provided by an embodiment of the present invention, the method specifically comprises the following steps:

[0065] Step 101: receiving radiation source signals of multiple modes, preprocessing the signals of different modes, and aligning the preprocessed signals by using time stamps to obtain multi-modal data signals;

[0066] In this embodiment, the radiation source signal can have multiple modes, such as electromagnetic wave signals, sound wave signals, infrared signals, etc. Signals of different modes have different characteristics and advantages, and can provide more comprehensive radiation source information; receiving radiation source signals can be achieved through various sensors and monitoring equipment, such as antennas, microphones, infrared detectors, etc. These devices can be installed in different locations and environments to obtain radiation source signals of different angles and ranges; in the military field, radiation source signals can be received through various platforms such as satellites, aircraft, and ground radars, including enemy radar signals, communication signals, missile launch signals, etc. In the field of environmental monitoring, radiation source signals from different pollution sources, such as factory exhaust and automobile exhaust, can be received through sensor networks; radiation source signals of different modes may have different formats, frequency ranges, noise levels, etc., and need to be preprocessed to improve signal quality and availability. Preprocessing can include operations such as signal amplification, filtering, noise reduction, and digitization; for electromagnetic wave signals, RF amplification, bandpass filtering, noise removal, etc. can be performed to improve signal strength and clarity. For sound wave signals, microphone amplification, low-pass filtering, background noise removal and other processing can be performed to improve the audibility and recognition of the signal. For infrared signals, infrared detector signal amplification, thermal noise removal and other processing can be performed to improve the sensitivity and accuracy of the signal; radiation source signals of different modes may differ in time, and timestamp alignment is required to ensure the synchronization and consistency of multi-modal data signals. Timestamp alignment can be achieved by adding timestamp information to the signal or using synchronous clocks. For signals with timestamp information, alignment can be performed directly based on the timestamp. For signals without timestamp information, synchronous clocks or other synchronization mechanisms can be used to ensure synchronous acquisition of signals. For example, a global positioning system clock or a high-precision atomic clock can be used as a synchronous clock to ensure that the signals collected by different sensors are consistent in time.

[0067] Step 102: input the multimodal data signal into the multimodal large model, separate the multimodal data signal using a dynamic separation technology at the input layer, extract features from the multimodal data signal through the feature extraction layer of the multimodal large model, and obtain target feature information;

[0068] Step 103: Transmit the target feature information to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, and introduce the knowledge graph to combine the historical behavior, geographical location and mission scenario of the radiation source to generate the behavior information of the radiation source signal;

[0069] In this embodiment, the target feature information is input into the behavior decoding layer of the multimodal large model, wherein the behavior decoding layer is constructed based on Transformer, and Transformer is composed of an encoder and a decoder, the encoder is used to encode the input features, and the decoder is used to generate prediction results; the encoder of Transformer is used to encode the sequence of target feature information, and the encoded feature sequence is obtained by stacking multiple encoder layers, and the encoded feature sequence is fused with the original target feature information, wherein the encoder is composed of multiple layers, each layer including a multi-head self-attention mechanism and a feedforward neural network; the decoder of Transformer is used to perform behavior prediction on the fused features, and the output sequence of the decoder is obtained by stacking multiple decoder layers, and the category with the highest probability in the output sequence is used as the final prediction result to obtain the behavior category of the radiation source signal, wherein the decoder is composed of multiple layers, each layer including a multi-head self-attention mechanism, an encoder-decoder attention mechanism and a feedforward neural network.

[0070] In this embodiment, historical behavior data, geographic location information, and task scenario descriptions of the radiation source are collected to construct a knowledge graph, in which nodes include at least radiation source entities, behavior category entities, geographic location entities, and task scenario entities, and edges represent relationships between different entities; by querying the knowledge graph, historical behavior, geographic location, and task scenario information related to the current radiation source are found, and the behavior category features output by the multimodal large model are fused with the information features in the knowledge graph to obtain a fused feature vector; based on the fused feature vector, RNN is used to perform sequence modeling on the fused feature vector to generate behavior information of the radiation source signal through RNN.

[0071] Step 104: Evaluate the generated radiation source signal behavior information, and optimize the multi-modal large model according to the evaluation result.

[0072] In this embodiment, a data set for evaluating the behavior information of radiation source signals is collected, wherein the data set includes at least the real radiation source signal behavior annotation and the behavior information generated by the multimodal large model; a set of multimodal large model parameters are randomly generated as sparrow individuals to form an initial population, and for each sparrow individual, its corresponding parameters are applied to the multimodal large model, and the fitness value is calculated using the data set; the positions of the discoverer, the joiner, and the scout are updated, the discoverer is responsible for finding food sources, the joiner follows the discoverer to find food, and the scout performs random searches at the edge of the population, wherein the population size is 30, the parameter dimension is 10, the maximum number of iterations is 100, the warning value is 0.6, the safety value is 0.7, and the step size control parameter is 0.8; the iteration is repeated until the maximum number of iterations is reached, and when the sparrow search algorithm ends, the optimal multimodal large model parameters are output.

[0073] In the embodiment of the present invention, through the end-to-end characteristics of unified modeling and deep learning, the model can more accurately capture the multimodal characteristics of the signal and significantly improve the accuracy of behavior recognition. At the same time, the simplified process greatly improves the recognition efficiency and is suitable for scenarios with high real-time requirements. The comprehensive processing of multimodal input by a single large model effectively avoids the redundancy and bottleneck of traditional step-by-step processing, ensures the full integration of information between different modal features, and improves the applicability in complex electromagnetic environments. By utilizing the generalization ability and knowledge reasoning mechanism of the large model, it can adapt to more types of signal characteristics and behavior types, support applications in dynamic electromagnetic environments, and improve the response capability to unknown signals. The overall solution covers the entire process from signal input to behavior derivation, reduces manual intervention and rule dependence, and realizes a fully intelligent signal recognition system. The model combines the extracted parameters and scenario information to complete the intelligent behavior derivation of the radiation source based on a large-scale knowledge base, thereby supporting more advanced applications, such as interference source identification and threat assessment.

[0074] See also Figure 2 , a schematic diagram of a second embodiment of a method for identifying radiation source signal behavior provided by an embodiment of the present invention, the method comprising:

[0075] Step 201: After receiving radiation source signals of multiple modes, the radiation source signals are preliminarily classified into slow frequency signals and fast frequency signals;

[0076] In this embodiment, the radiation source signal can come from a variety of different devices and systems, including communication equipment, radar systems, electronic countermeasure equipment, etc. These signals have different frequency ranges, modulation methods and signal characteristics; for example, communication signals may include voice communication, data communication, etc., and their frequency range is usually between several thousand hertz and tens of megahertz, and the frequency range of radar signals can range from hundreds of megahertz to tens of megahertz. The classification of slow-frequency signals and fast-frequency signals can be based on the frequency characteristics of the signal. Signals with lower frequencies can be defined as slow-frequency signals, and signals with higher frequencies can be defined as fast-frequency signals. The specific classification threshold can be determined based on actual application requirements and signal characteristics. For example, signals with frequencies below 100kHz can be regarded as slow-frequency signals, and signals with frequencies above 100kHz can be regarded as fast-frequency signals.

[0077] Step 202: filtering the slow frequency signal using a low-pass filter to extract the slow frequency component;

[0078] In this embodiment, the transfer function is:

[0079]

[0080] Where N represents the filter order, a k represents the filter coefficient, s represents the complex frequency variable;

[0081] In this embodiment, the low-pass filter is a filter that can pass low-frequency signals and block high-frequency signals. It can achieve the filtering function by performing weighted summation on the input signal. Common low-pass filters include Butterworth filters, Chebyshev filters, etc.; first, determine the cutoff frequency of the filter, that is, the frequency point at which the filter starts to block high-frequency signals. The cutoff frequency is determined based on the frequency range and requirements of the slow-frequency signal. Then, the input slow-frequency signal is processed through a low-pass filter to obtain a filtered signal. During the filtering process, the parameters of the filter are adjusted as needed. The purpose of extracting the slow-frequency component through a low-pass filter is to remove high-frequency noise and interference in the signal and retain the low-frequency components in the signal.

[0082] Step 203: determine a dynamic filtering window according to the frequencies of signals from different radiation sources, use an autoregressive model to perform time series modeling on slow frequency signals, and generate an audio signal;

[0083] In this embodiment, the calculation formula is:

[0084]

[0085] In the formula, X t represents the signal value at the current moment, p represents the model order, represents the model coefficient, ∈ t represents the random error term;

[0086] Step 204: For the fast frequency signal, generate a two-dimensional time-frequency spectrum through short-time Fourier transform processing;

[0087] In this embodiment, the calculation formula is:

[0088]

[0089] Where STFT(t,f) represents the signal strength at time t and frequency f, x(τ) represents the input signal, τ represents the time variable, w(τ-t) represents the window function, t represents time, f represents frequency, and e represents the window function. -j2πfτ represents the contribution of the calculated signal at frequency f, j represents the imaginary unit and j 2 = -1;

[0090] In this embodiment, for fast frequency signals, the process of using short-time Fourier transform for processing may include the following steps: first, determine the window length and overlap rate of the short-time Fourier transform, the window length determines the length of each time period, and the overlap rate determines the degree of overlap between adjacent time periods, then divide the input fast frequency signal into several short time periods, and perform Fourier transform on each time period. The Fourier transform result obtained can be expressed as a complex matrix, in which each element represents the amplitude and phase of the signal at a specific time and frequency, and finally, the amplitude part of the complex matrix is ​​modulo to obtain a two-dimensional time-frequency spectrum of the signal. During the processing, adjust the parameters such as window length and overlap rate as needed to obtain better time-frequency resolution and accuracy.

[0091] Step 205: using gamma correction to improve the contrast and resolution of the two-dimensional time-frequency spectrum;

[0092] In this embodiment, the calculation formula is:

[0093]

[0094] In the formula, I in Represents the pixel value of the input image, I out Represents the pixel value of the output image, and γ represents the gamma value;

[0095] Step 206: align the preprocessed slow-frequency audio signal and the fast-frequency time-frequency video signal according to the timestamps of the signals to obtain a multimodal data signal.

[0096] See also Figure 3 , a schematic diagram of a third embodiment of a method for identifying radiation source signal behavior provided by an embodiment of the present invention, the method comprising:

[0097] Step 301: Input the multimodal data signal into the multimodal large model, perform preliminary standardization processing on the multimodal data signal at the input layer, and then separate the standardized multimodal data signal using dynamic separation technology based on the frequency of the signal;

[0098] In this embodiment, the filter bank is assumed to be F = {F1, F2, ..., F m}, the signal after filtering is Y={Y1,Y2,…,Y m}, Where X i represents the standardized multimodal data signal, and * represents the convolution operation;

[0099] Step 302: using CNN-GRU to extract local features of the time series signal from the slow frequency audio signal in the separated multimodal data signal and form a slow frequency feature representation, using 3D-CNN to extract spatial and temporal features of the video signal from the fast frequency time-frequency video signal in the separated multimodal data signal, and generate a deep representation of the fast frequency signal;

[0100] In this embodiment, CNN is used to extract local features of the time series signal from the slow frequency audio signal in the separated multimodal data signal. The CNN consists of multiple convolutional layers and pooling layers. The convolution operation is:

[0101]

[0102] In the formula, C i,j represents the value of the convolution layer output at position (i, j), K represents the size of the convolution kernel, and W k represents the weight of the convolution kernel at position k, A i+k,j+k Represents the value of the input signal at position (i+k,j+k);

[0103] Suppose the output after convolutional layer processing is C = {c1, c2, ..., c n}, c1 is passed as the initial input to the first time step of GRU. At each time step, GRU receives the current input feature vector and the hidden state of the previous time step, merges the forget gate and the input gate into an update gate, and merges the cell state and the hidden state to calculate the hidden state of the current time step. Repeat the calculation until all inputs are processed, and obtain a series of hidden states H = {h1,h,…,h n}, the slow frequency feature representation is obtained through the hidden state H;

[0104] 3D-CNN is used to extract the spatial and temporal features of the video signal from the fast-frequency time-frequency video signal in the separated multimodal data signal. The convolution operation is:

[0105]

[0106] Where D i,j,k represents the value of the output feature map at position (i, j, k), L, M, N represent the size of the 3D convolution kernel in three dimensions, V i+l,j+m,k+n represents the value of the input video signal at position (i+l,j+m,k+n, Wl,,m,n represents the weight of the 3D convolution kernel at position l,,m,,n;

[0107] The time-frequency domain features of the video signal are fused with the spatial and temporal features extracted by 3D-CNN by splicing to generate a deep representation of the fast-frequency signal.

[0108] Step 303: Use a multimodal self-attention mechanism to perform feature fusion, and unify high-dimensional features of different modalities into the same spatial dimension through a multi-layer perceptron to obtain target feature information.

[0109] See also Figure 4 , a schematic diagram of the structure of a radiation source signal behavior recognition device provided by an embodiment of the present invention, the device comprises:

[0110] The signal processing module 401 is used to receive radiation source signals of multiple modes, pre-process the signals of different modes, and align the pre-processed signals using timestamps to obtain multi-modal data signals;

[0111] The feature extraction module 402 is used to input the multimodal data signal into the multimodal large model, separate the multimodal data signal using a dynamic separation technology at the input layer, and extract features from the multimodal data signal through the feature extraction layer of the multimodal large model to obtain target feature information;

[0112] The behavior decoding module 403 is used to transmit the target feature information to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, and introduce the knowledge graph to combine the historical behavior, geographical location and mission scenario of the radiation source to generate the behavior information of the radiation source signal;

[0113] The model optimization module 404 is used to evaluate the generated radiation source signal behavior information and optimize the multi-modal large model according to the evaluation results.

[0114] In the embodiments of the present invention, the whole process from signal processing to behavior recognition is automated without human intervention, thereby improving operational efficiency. Through multimodal information fusion, signal interference and characteristic changes in complex electromagnetic environments can be effectively dealt with. Slow-frequency features provide behavior trend predictions, and fast-frequency features analyze behavior details, and the combination of the two improves the overall recognition effect. The framework supports new modal inputs, such as spatial position information, radiation source task labels, etc., which enhances the application scope of the model.

[0115] above Figure 4 The radiation source signal behavior identification device in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The radiation source signal behavior identification device in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0116] Figure 56 is a schematic diagram of the structure of a radiation source signal behavior recognition device provided by an embodiment of the present invention. The radiation source signal behavior recognition device 600 may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 610 (for example, one or more processors) and a memory 620, and one or more storage media 630 (for example, one or more mass storage devices) storing application programs 633 or data 632. Among them, the memory 620 and the storage medium 630 can be short-term storage or permanent storage. The program stored in the storage medium 630 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the radiation source signal behavior recognition device 600. Furthermore, the processor 610 can be configured to communicate with the storage medium 630, and execute a series of instruction operations in the storage medium 630 on the radiation source signal behavior recognition device 600 to implement the method provided by the above implementation.

[0117] The radiation source signal behavior recognition device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input and output interfaces 660, and / or one or more operating devices 631, such as Windows Serve, Mac OS X, Unix, Linux, FreeBSD, etc. It will be understood by those skilled in the art that Figure 5 The structure of the radiation source signal behavior recognition device shown does not constitute a limitation on the computer device provided by the present invention, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0118] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes each step of the radiation source signal behavior identification method provided in the above embodiments.

[0119] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described equipment, devices, and units can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0120] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0121] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying radiation source signal behavior, characterized in that: The radiation source signal behavior identification method comprises: Receive radiation source signals of multiple modes, pre-process the signals of different modes, and align the pre-processed signals using time stamps to obtain multi-modal data signals; Inputting the multimodal data signal into the multimodal large model, separating the multimodal data signal using a dynamic separation technology at the input layer, extracting features from the multimodal data signal through a feature extraction layer of the multimodal large model, and obtaining target feature information; The target feature information is transmitted to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, and the knowledge graph is introduced to combine the historical behavior, geographical location and mission scenario of the radiation source to generate the behavior information of the radiation source signal; The generated radiation source signal behavior information is evaluated, and the multimodal large model is optimized based on the evaluation results.

2. A method for identifying radiation source signal behavior according to claim 1, characterized in that: The receiving of radiation source signals of multiple modes, preprocessing the signals of different modes, and obtaining a multi-modal data signal by aligning the preprocessed signals by time stamps, includes: After receiving radiation source signals of multiple modes, the radiation source signals are preliminarily classified into slow frequency signals and fast frequency signals. The slow frequency signals are filtered using a low-pass filter to extract the slow frequency components. The transfer function is: Where N represents the filter order, a k represents the filter coefficient, s represents the complex frequency variable; The dynamic filtering window is determined according to the frequency of different radiation source signals, and the slow frequency signal is modeled in time series using an autoregressive model to generate an audio signal: Where, X t represents the signal value at the current moment, p represents the model order, represents the model coefficient, ∈ t represents the random error term; For fast frequency signals, a two-dimensional time-frequency spectrum is generated through short-time Fourier transform processing: Where STFT(t, f) represents the signal strength at time t and frequency f, x(τ) represents the input signal, τ represents the time variable, w(τ-t) represents the window function, t represents time, f represents frequency, and e represents the window function. -j2πfτ represents the contribution of the calculated signal at frequency f, j represents the imaginary unit and j 2 = -1; Use gamma correction to improve the contrast and resolution of 2D time-spectrograms: In the formula, I in Represents the pixel value of the input image, I out Represents the pixel value of the output image, and γ represents the gamma value; The preprocessed slow-frequency audio signal and fast-frequency time-frequency video signal are aligned according to the timestamps of the signals to obtain a multimodal data signal.

3. A method for identifying radiation source signal behavior according to claim 1, characterized in that: Inputting the multimodal data signal into the multimodal large model, separating the multimodal data signal by using a dynamic separation technology at the input layer, extracting features from the multimodal data signal by using a feature extraction layer of the multimodal large model, and obtaining target feature information, including: The multimodal data signal is input into the multimodal large model, and the multimodal data signal is preliminarily standardized at the input layer. Then, the standardized multimodal data signal is separated by dynamic separation technology based on the frequency of the signal. The filter group is set to be F = {F1, F2, ..., F m }, the signal after filtering is Y = {Y1, Y2, ..., Y m }, Where X i represents the standardized multimodal data signal, and * represents the convolution operation; CNN-GRU is used to extract local features of time series signals from slow-frequency audio signals in the separated multimodal data signals and form slow-frequency feature representations. 3D-CNN is used to extract spatial and temporal features of video signals from fast-frequency time-frequency video signals in the separated multimodal data signals and generate deep representations of fast-frequency signals. A multimodal self-attention mechanism is used for feature fusion, and the high-dimensional features of different modalities are unified into the same spatial dimension through a multi-layer perceptron to obtain the target feature information.

4. A method for identifying radiation source signal behavior according to claim 3, characterized in that: The method uses CNN-GRU to extract local features of the time series signal from the slow frequency audio signal in the separated multimodal data signal and forms a slow frequency feature representation, and uses 3D-CNN to extract spatial and temporal features of the video signal from the fast frequency time-frequency video signal in the separated multimodal data signal, and generates a deep representation of the fast frequency signal, including: CNN is used to extract the local features of the time series signal from the slow-frequency audio signal in the separated multimodal data signal. CNN consists of multiple convolutional layers and pooling layers. The convolution operation is: In the formula, C i，j represents the value of the convolution layer output at position (i, j), K represents the size of the convolution kernel, and W k represents the weight of the convolution kernel at position k, A i+k，j+k Represents the value of the input signal at position (i+k, j+k); Suppose the output after convolutional layer processing is C = {c1, c2, ..., c n }, c1 is passed as the initial input to the first time step of GRU. At each time step, GRU receives the current input feature vector and the hidden state of the previous time step, merges the forget gate and the input gate into an update gate, and merges the cell state and the hidden state to calculate the hidden state of the current time step. Repeat the calculation until all inputs are processed, and obtain a series of hidden states H = {h1, h, ..., h n }, the slow frequency feature representation is obtained through the hidden state H: 3D-CNN is used to extract the spatial and temporal features of the video signal from the fast-frequency time-frequency video signal in the separated multimodal data signal. The convolution operation is: Where D i,j,k represents the value of the output feature map at position (i, j, k), L, M, N represent the size of the 3D convolution kernel in three dimensions, V i+l,j+m,k+n represents the value of the input video signal at position (i+l,j+m,k+n, Wl,m,n represents the weight of the 3D convolution kernel at position l,m,n; The time-frequency domain features of the video signal are fused with the spatial and temporal features extracted by 3D-CNN by splicing to generate a deep representation of the fast-frequency signal.

5. A method for identifying radiation source signal behavior according to claim 1, characterized in that: The target feature information is transmitted to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, including: Input the target feature information into the behavior decoding layer of the multimodal large model, where the behavior decoding layer is built based on Transformer. Transformer consists of an encoder and a decoder. The encoder is used to encode the input features, and the decoder is used to generate prediction results. Use the Transformer encoder to encode the sequence of target feature information, obtain the encoded feature sequence by stacking multiple encoder layers, and perform feature fusion on the encoded feature sequence and the original target feature information. The encoder consists of multiple layers, each of which includes a multi-head self-attention mechanism and a feedforward neural network. The Transformer decoder is used to predict the behavior of the fused features. The output sequence of the decoder is obtained by stacking multiple decoder layers. The category with the highest probability in the output sequence is used as the final prediction result to obtain the behavior category of the radiation source signal. The decoder consists of multiple layers, each of which includes a multi-head self-attention mechanism, an encoder-decoder attention mechanism, and a feedforward neural network.

6. A method for identifying radiation source signal behavior according to claim 1, characterized in that: The introduced knowledge graph combines the historical behavior, geographical location and task scenario of the radiation source to generate the behavior information of the radiation source signal, including: Collect historical behavior data, geographic location information, and task scenario descriptions of radiation sources, and construct a knowledge graph, where nodes include at least radiation source entities, behavior category entities, geographic location entities, and task scenario entities, and edges represent the relationships between different entities; By querying the knowledge graph, we can find the historical behaviors, geographic locations, and mission scenario information related to the current radiation source, and fuse the behavior category features output by the multimodal large model with the information features in the knowledge graph to obtain the fused feature vector. Based on the fused feature vector, RNN is used to perform sequence modeling on the fused feature vector to generate the behavior information of the radiation source signal through RNN.

7. A method for identifying radiation source signal behavior according to claim 1, characterized in that: The step of evaluating the generated radiation source signal behavior information and optimizing the multi-modal large model according to the evaluation result includes: Collecting a data set for evaluating the behavior information of radiation source signals, wherein the data set at least includes the actual radiation source signal behavior annotation and the behavior information generated by the multimodal large model; A set of parameters of the multimodal large model is randomly generated as sparrow individuals to form an initial population. For each sparrow individual, its corresponding parameters are applied to the multimodal large model, and the fitness value is calculated using the data set. Update the positions of the discoverer, joiner, and scout. The discoverer is responsible for finding food sources, the joiner follows the discoverer to find food, and the scout performs random searches at the edge of the population. The population size is 30, the parameter dimension is 10, the maximum number of iterations is 100, the warning value is 0.6, the safety value is 0.7, and the step control parameter is 0.8; Repeat the iteration until the maximum number of iterations is reached. When the sparrow search algorithm ends, the optimal multimodal large model parameters are output.

8. A radiation source signal behavior recognition device, characterized in that: The radiation source signal behavior identification device comprises: A signal processing module is used to receive radiation source signals of multiple modes, pre-process the signals of different modes, and align the pre-processed signals using time stamps to obtain multi-modal data signals; A feature extraction module, used to input the multimodal data signal into the multimodal large model, separate the multimodal data signal using a dynamic separation technology at the input layer, and extract features from the multimodal data signal through the feature extraction layer of the multimodal large model to obtain target feature information; A behavior decoding module is used to transmit the target feature information to the behavior decoding layer of the multimodal large model to predict and classify the behavior of the radiation source signal, and introduce the knowledge graph to combine the historical behavior, geographical location and mission scenario of the radiation source to generate the behavior information of the radiation source signal; The model optimization module is used to evaluate the generated radiation source signal behavior information and optimize the multi-modal large model based on the evaluation results.

9. A radiation source signal behavior recognition device, characterized in that: The radiation source signal behavior identification device includes a memory and at least one processor, wherein instructions are stored in the memory; the at least one processor calls the instructions in the memory so that the radiation source signal behavior identification device performs each step of the radiation source signal behavior identification method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the various steps of the radiation source signal behavior identification method as described in any one of claims 1 to 7 are implemented.

Images