A multi-modal heterogeneous deep learning radiation source individual identification method fusing knowledge features

Abstract

The application discloses a multi-modal heterogeneous deep learning radiation source individual identification method fusing knowledge features, and belongs to the technical field of wireless communication security and signal processing, and solves the problems of insufficient feature extraction capability and weak anti-interference capability existing in the prior radiation source individual identification method. The method comprises the following steps: acquiring a radio frequency signal of a wireless transmitting device; constructing four kinds of heterogeneous modal inputs of the radio frequency signal, wherein the four kinds of heterogeneous modal inputs comprise a time domain sequence modal, a time-frequency image modal, a graph structure modal and an expert knowledge feature modal; inputting the inputs of the respective heterogeneous modal into corresponding neural networks respectively, and extracting high-order semantic features of the corresponding modal through the corresponding neural networks; normalizing the high-order semantic features extracted by the respective heterogeneous modal, and splicing the high-order semantic features in the channel dimension to obtain a multi-modal fusion feature vector; and identifying the category of the radiation source individual based on the multi-modal fusion feature vector by using a first classifier.

Description

Technical Field

[0001] This invention belongs to the field of wireless communication security and signal processing technology, specifically relating to a multimodal heterogeneous deep learning method for identifying individual radiation sources that integrates knowledge features. Background Technology

[0002] With the development of wireless communication technology and the Internet of Things (IoT), billions of wireless devices are widely deployed in urban, home, and industrial environments, making device authentication and security management increasingly important. Traditional authentication mechanisms rely heavily on encryption algorithms and device identifiers (such as Media Access Control (Media Access Control) addresses). However, these mechanisms struggle to guarantee system security and usability due to difficulties in key management, spoofing of addresses, and limited computing resources. Specific Emitter Identification (SEI) technology, a method based on physical layer radio frequency fingerprints generated by differences in wireless device hardware, eliminates the need for device authentication and offers advantages such as unforgeability, no pairing required, and low computational complexity, making it an effective solution to the wireless device identification problem.

[0003] Existing SEI methods are mainly divided into statistical methods based on human-made features and automatic feature extraction methods based on deep learning. The former relies on expert experience and has poor robustness; the latter, although it can automatically extract features, mostly depends on a single structural input and is difficult to comprehensively represent the heterogeneous features of signals in time-frequency, structural and statistical dimensions; highly parameterized models are sensitive to the number of training samples, and their recognition performance drops sharply in scenarios with few samples or low signal-to-noise ratio; fully data-driven models lack domain prior guidance and have insufficient generalization ability and interpretability.

[0004] Therefore, there is an urgent need for a knowledge-driven, hybrid heterogeneous SEI method to improve the accuracy and robustness of device identification. Summary of the Invention

[0005] Based on the above analysis, the present invention aims to provide a multimodal heterogeneous deep learning method for identifying radiation sources by integrating knowledge features, so as to solve the problems of insufficient feature extraction capability and weak anti-interference capability in existing radiation source identification methods.

[0006] The present invention provides a multimodal heterogeneous deep learning-based individual radiation source identification method that integrates knowledge features, the method comprising: Acquire radio frequency signals from wireless transmitting devices; The inputs for the radio frequency signal are constructed in four heterogeneous modes, including time-domain sequence mode, time-frequency image mode, graph structure mode, and expert knowledge feature mode. The inputs of each heterogeneous mode are input into the corresponding neural network, and the high-order semantic features of the corresponding mode are extracted through the corresponding neural network. The neural networks corresponding to the time-domain sequence mode, time-frequency image mode, graph structure mode, and expert knowledge feature mode are respectively a hybrid convolutional-Transformer network, a convolutional neural network, a graph neural network, and a deep knowledge feature fusion network. After normalizing the high-order semantic features extracted from each heterogeneous modality, they are concatenated in the channel dimension to obtain a multimodal fusion feature vector; Based on the multimodal fusion feature vector, the first classifier is used to identify the category of the radiation source individual.

[0007] Based on the above solution, this solution also makes the following improvements: Furthermore, the high-order semantic features of each heterogeneous modality are normalized and then concatenated along the channel dimension to obtain fused features, and then the following steps are performed: Norm normalization is performed on the high-order semantic features extracted from each heterogeneous modality; The higher-order semantic features of the four heterogeneous modalities after norm normalization are concatenated sequentially along the channel dimension to obtain a multimodal fusion feature vector.

[0008] Furthermore, the input to the time-domain sequence mode is the orthogonal IQ time-domain signal sequence obtained by demodulating the radio frequency signal; The neural network corresponding to the time-domain sequence mode is used to extract high-order time-domain semantic features of the orthogonal IQ time-domain signal sequence.

[0009] Furthermore, the input of the time-frequency image mode is constructed by performing discrete wavelet transform on the orthogonal IQ signals of the radio frequency signal, and then mapping the I / Q channels into a dual-channel image through multi-scale selective reconstruction; The neural network corresponding to the time-frequency image mode is used to extract the time-frequency features of the dual-channel image.

[0010] Furthermore, the input of the graph structure mode is constructed by mapping the sampling points in the IQ sequence of the radio frequency signal to graph nodes through linear fixed graph mapping, and constructing edges based on the temporal neighborhood between sampling points to form a graph structure with a unified topology. The neural network corresponding to the graph structure mode is used to extract the topological features of the graph structure.

[0011] Furthermore, the input of the expert knowledge feature mode is constructed by extracting the RF-DNA statistical features, Hilbert-Huang transform spectrum statistical features, and wavelet domain multi-scale statistical features of the radio frequency signal and combining them into an expert feature vector; The neural network corresponding to the expert knowledge feature modality is used to extract the physical statistical features of the expert feature vector.

[0012] Furthermore, the deep knowledge feature fusion network includes: three sets of structurally consistent feature encoding subnetworks, a feature connection layer, and a second classifier; among which, Three sets of structurally consistent feature coding subnetworks are used to encode RF-DNA statistical features, Hilbert-Huang transform spectral statistical features, and wavelet domain multi-scale statistical features, respectively. The feature connection layer is used to concatenate the feature vectors output by all feature encoding subnetworks along the channel dimension to obtain a high-dimensional knowledge feature fusion vector. The second classifier is used to extract the physical statistical features of the expert feature vector from the high-dimensional knowledge feature fusion vector.

[0013] Furthermore, the hybrid convolutional-Transformer network includes: The head convolution module is used to perform shallow mapping on the orthogonal IQ time-domain signal sequence and output an RF fingerprint feature map; The time-frequency residual dimensionality reduction module is used to perform multi-scale time-frequency joint dynamic feature extraction on the radio frequency fingerprint feature map to obtain the synergistic fusion features of time-domain dynamics and frequency-domain characteristics; A Transformer encoder is used to aggregate the global dependency information of the collaboratively fused features through a self-attention mechanism and output the temporal features of the orthogonal IQ time-domain signal sequence using classification labels.

[0014] Furthermore, the method also includes training a neural network with four heterogeneous modalities and a first classifier, wherein the training is performed as follows: Receive radio frequency signal samples from wireless transmitting devices, summarize the categories of each radio frequency signal sample and its corresponding radiation source, and construct a radio frequency signal training set; The neural networks of four heterogeneous modes were trained independently using a radio frequency signal training set; Remove the classification layer of each trained heterogeneous modality neural network, and use the feature vector of the input classification layer of each heterogeneous modality neural network as the high-order semantic feature of the corresponding heterogeneous modality; normalize the high-order semantic features extracted from each heterogeneous modality and concatenate them in the channel dimension to obtain the multimodal fusion feature vector. The multimodal fusion feature vectors of each radio frequency signal sample are input into the first classifier, and the category of the radiation source individual corresponding to each radio frequency signal sample is used as the sample label. The parameters of the first classifier and the neural network of each heterogeneous mode are updated and optimized by the backpropagation algorithm to obtain the trained neural network of each heterogeneous mode and the first classifier.

[0015] Furthermore, the output dimension of the first classifier is equal to the number of individual categories of the radiation source.

[0016] Compared with the prior art, the present invention can achieve at least the following beneficial effects: The multimodal heterogeneous deep learning radiation source individual identification method that integrates knowledge features provided by this invention has at least the following beneficial effects: (1) This invention integrates four types of input modes: sequence, image, graph structure and expert features, builds a multi-branch neural network architecture, completes the deep modeling and fusion of heterogeneous features of radio frequency signals, fully explores and utilizes the features extracted from four different types of input data, and effectively improves the recognition accuracy and system robustness of the radiation source individual identification (SEI) system.

[0017] (2) This invention introduces a knowledge-driven expert feature pathway. By integrating RF-DNA features, Hilbert-Huang transform features and multi-scale wavelet statistical features, a deep neural network is built to complete the modeling. This achieves the complementary advantages of domain knowledge and data-driven methods, and improves the stability of the system in scenarios with few samples and non-cooperative scenarios.

[0018] (3) The sequence network of the present invention replaces the original IQ signal with amplitude-phase sequence as input. By directly extracting the energy and angle intrinsic physical quantities of the signal, it can effectively focus on stable fingerprint features generated by factors such as transmitter hardware nonlinearity and phase noise, and improve the robustness of the features to various interferences such as carrier frequency offset.

[0019] In summary, this invention, through deep fusion of multimodal heterogeneous features, fully mines the discriminative information of signals in the time, frequency, structural, and statistical domains, significantly improving the accuracy and robustness of individual radiation source identification. It can adapt to the needs of device identity authentication and security management in complex electromagnetic environments and can be applied to scenarios such as physical layer device authentication, radio frequency fingerprint recognition, and wireless network security management.

[0020] Other features and advantages of the invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of the invention may be realized and obtained from what is particularly pointed out in the description and the drawings. Attached Figure Description

[0021] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0022] Figure 1 A flowchart of a multimodal heterogeneous deep learning radiation source individual identification method that integrates knowledge features, provided in an embodiment of the present invention; Figure 2 A hybrid heterogeneous framework for integrating knowledge features for individual radiation source identification; Figure 3 A diagram of the MixCNN-Transformer architecture used in sequence network construction; Figure 4 To construct a hierarchical structure for the DWT decomposition signal in image networks; Figure 5 The reconstruction process of channel I in image network construction; Figure 6 The residual block structure of the feature extraction network used by the image network; Figure 7 The DiffPool structure graph in the construction of graph-structured networks; Figure 8 The process of dual-tree complex wavelet transform in knowledge feature networks; Figure 9 A deep knowledge fusion network used for knowledge feature networks. Detailed Implementation

[0023] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0024] A specific embodiment of the present invention provides a multimodal heterogeneous deep learning method for identifying individual radiation sources by fusing knowledge features. The flowchart of this method is as follows: Figure 1 As shown, a hybrid heterogeneous framework for fusing knowledge features for individual radiation source identification is as follows: Figure 2 As shown. Specific details are as follows.

[0025] Step S1: Acquire the radio frequency signal of the wireless transmitting device; Step S2: Construct the inputs for the four heterogeneous modes of the radio frequency signal. The four heterogeneous modes include time-domain sequence mode, time-frequency image mode, graph structure mode, and expert knowledge feature mode. Input the inputs of each heterogeneous mode into the corresponding neural network, and extract the high-order semantic features of the corresponding heterogeneous mode through the corresponding neural network. Step S3: Normalize the high-order semantic features extracted from each heterogeneous modality and then concatenate them along the channel dimension to obtain a multimodal fusion feature vector; Step S4: Based on the multimodal fusion feature vector, use the first classifier to identify the category of the radiation source individual.

[0026] The following is a detailed explanation of the inputs to the four heterogeneous modalities in step S2 and their corresponding neural networks.

[0027] (1) Input of time-domain sequence mode and its corresponding neural network (also known as "sequence network") The input to the time-domain sequence mode is the orthogonal IQ time-domain signal sequence obtained by demodulating the radio frequency signal. The neural network corresponding to the time-domain sequence mode is a hybrid convolutional-transformer network, used to extract high-order time-domain semantic features of the orthogonal IQ time-domain signal sequence.

[0028] The hybrid convolutional-Transformer network includes: a head convolutional module for shallow mapping of the orthogonal IQ time-domain signal sequence and outputting an RF fingerprint feature map; a time-frequency residual dimensionality reduction module for multi-scale time-frequency joint dynamic feature extraction of the RF fingerprint feature map to obtain a synergistic fusion feature of time-domain dynamics and frequency-domain characteristics; and a Transformer encoder for aggregating the global dependency information of the synergistic fusion feature through a self-attention mechanism and outputting the temporal features of the orthogonal IQ time-domain signal sequence using classification labels.

[0029] In practice, the input to the time-domain sequence mode directly uses the orthogonal IQ time-domain signal sequence obtained by demodulating the radio frequency signal, without relying on any manual feature transformation. This preserves the original physical information of the signal and avoids micro-fingerprint distortion that may result from pre-feature transformation. Specifically, the real and imaginary parts of the radio frequency signal are extracted. virtual part They are represented as follows:

[0030]

[0031] in, Representative extraction Operators of the real part Representative extraction Operators with the imaginary part These are the received discrete sampling points, forming a two-channel matrix. This reflects the time-domain characteristics of the original signal.

[0032] Existing technologies generally employ multi-layer stacked LSTM networks to process the original IQ sequences. However, LSTM can only extract a single-dimensional temporal dependency. On the one hand, it is difficult to effectively capture multi-scale local micro-fingerprints introduced by hardware component mutations (such as transient power-on spikes and short-term hardware defects). On the other hand, when the sequence is long, it is prone to gradient vanishing, resulting in insufficient ability to model long-distance physical characteristics of the entire sequence (such as steady-state phase drift and global frequency shift).

[0033] To address the aforementioned shortcomings, this embodiment improves and constructs a hybrid neural network architecture combining convolutional and Transformer encoders (MixCNN-Transformer), i.e., a hybrid convolutional-Transformer network, the network architecture of which is as follows: Figure 3 As shown, this architecture utilizes the expanded receptive field of dilated convolution to extract multi-scale local micro-temporal patterns, while combining the self-attention mechanism of the Transformer encoder to adaptively capture global long-range physical dependencies, thus achieving efficient complementary extraction of local micro-fingerprints and global fingerprint semantics.

[0034] The hybrid convolutional-Transformer network gradually maps the input IQ orthogonal signal sequence into a higher-order semantic vector (i.e., higher-order temporal semantic features). The specific processing steps are as follows.

[0035] Dimension is After the original orthogonal IQ signal is input into the head convolutional layer, the head convolutional layer performs shallow mapping on the orthogonal IQ time-domain signal sequence by performing the following operations: The shallow mapping specifically includes two layers. The first layer mapping uses... convolution kernel span Lu He Two-dimensional convolution is performed on the two channels of the path to extract the data without aggressive downsampling. The initial spatiotemporal coupling characteristics between the two signals are preserved, maintaining the physical fingerprint distribution of the original signals; the second layer mapping employs... The convolution kernel performs feature extraction (i.e., fine-grained feature refinement) in the time domain and outputs an RF fingerprint feature map, which can preserve the complete micro fingerprint.

[0036] The time-frequency residual dimensionality reduction module (ResBlock_TF) performs multi-scale time-frequency joint dynamic feature extraction on the RF fingerprint feature map, performing the following operations: First, a two-stage downsampling dimensionality reduction is performed on the RF fingerprint feature map. Specifically, in each stage, the RF fingerprint feature map passes in parallel through a time-frequency residual block containing two paths: the first path uses dilated one-dimensional convolution to expand the receptive field, specifically capturing local abrupt changes and dynamic responses of the IQ signal at different time spans; the second path utilizes... Convolution is used to perform channel mixing and extract the frequency domain distribution characteristics of the signal across each channel. Then, residual connections are used to synergistically fuse the time-domain dynamics and frequency domain characteristics to obtain the synergistic fusion features of the time-domain dynamics and frequency domain characteristics, thereby achieving in-depth mining of the time-frequency evolution law of the IQ signal.

[0037] The Transformer encoder aggregates the global dependencies of the entire IQ signal sequence as follows: The collaboratively fused features are linearly projected onto a fixed dimension and input into a Transformer encoder containing a 2-layer structure and 8 attention heads. A learnable classification token (CLS Token) is appended to the sequence header. The correlation between any two temporal positions in the entire sequence is calculated using a self-attention mechanism, thereby adaptively aggregating the global phase drift and frequency shift information in the entire IQ signal sequence into the CLS Token. Finally, the output vector corresponding to the CLS Token is extracted as the high-order temporal semantic feature of the sequence input network, represented as follows: .

[0038] (2) Input of time-frequency image mode and its corresponding neural network (also known as "image network") The input to the time-frequency image mode is constructed by performing discrete wavelet transform on the orthogonal IQ signals of the radio frequency signal, and then mapping the I / Q channels into a dual-channel image through multi-scale selective reconstruction. The neural network corresponding to the time-frequency image mode is a convolutional neural network, used to extract the time-frequency features of the dual-channel image.

[0039] In the specific implementation process, in order to further explore the multi-scale discrimination features of IQ signals in the time-frequency domain, in the time-frequency image module, a dual-channel image construction method based on discrete wavelet transform (DWT) multi-scale selective reconstruction is adopted for the orthogonal IQ signals of radio frequency signals: taking the orthogonal IQ signals of radio frequency signals as input, unlike the extraction of one-dimensional instantaneous changes in the time domain sequence mode, the time-frequency image mode converts the one-dimensional sequence into a two-dimensional dual-channel narrow-side image.

[0040] In terms of specific implementation, compared with the existing technology that directly generates single-channel spectrograms based on short-time Fourier transform (STFT) or continuous wavelet transform (CWT), the improvement of this step is mainly reflected in two aspects: First, it avoids the loss of physical layer hardware fingerprint information caused by the traditional method of simply synthesizing the I / Q channels or discarding the orthogonal phase relationship; second, it overcomes the problems of unclear high-frequency micro fingerprints and reduced time domain resolution in traditional time-frequency maps.

[0041] Specifically, firstly, DWT decomposition is performed on the I / Q channels of the orthogonal IQ signal of the radio frequency signal, and the hierarchical structure of the decomposed signal using DWT is as follows: Figure 3 As shown, the signal can be Decomposed into low-frequency approximate components and high-frequency detail components , No. The layer decomposition operation is as follows:

[0042]

[0043] in, and These represent low-pass and high-pass filters, respectively, with lengths... This depends on the wavelet function used (e.g., Haar, Daubechies, Coiflets, etc.). Each level of decomposition is accompanied by a 2x downsampling, so a signal of length N can be processed at most... Layer decomposition. Then, a selective reconstruction method is used to construct the image, that is, each time only the wavelet components at a specific scale are retained, and the rest are set to zero before reconstruction, highlighting the feature response at that scale. The corresponding [number]th [level]. The layer reconstruction operation is as follows:

[0044] And so on, after The reconstruction can restore the features at each scale to their original length. The sequence. Finally, by performing separate operations on the I / Q channels. This reconstruction operation yields the construction matrices for both the I and Q channels. and The reconstruction process of channel I is as follows Figure 4 As shown, the features of the two channels are then concatenated along the channel dimension to form the final dual-channel image representation. .

[0045] In practice, ResNet18 can be used as the convolutional neural network. Figure 5 The residual block structure allows information to be directly passed across layers while preserving low-level features, improving training efficiency and model expressive power. ResNet18 achieves a good balance between shallow depth and high recognition accuracy, making it suitable for processing the multi-channel wavelet images constructed in this embodiment.

[0046] (3) Inputs to graph structure modes and their corresponding neural networks (also known as "graph structure networks") The input to the graph structure mode is constructed by mapping the sampling points in the IQ sequence of the radio frequency signal to graph nodes through a linear fixed graph mapping, and building edges based on the temporal neighborhood between sampling points to form a graph structure with a unified topology. The neural network corresponding to the graph structure mode is a graph neural network, used to extract the topological features of the graph structure.

[0047] In the specific implementation process, in order to further explore the nonlinear topological correlation characteristics of IQ signals in the structural domain, a method combining linear fixed graph mapping (LFGM) and hierarchical graph pooling (DiffPool) is used to construct the input of the graph structure mode. The aim is to map one-dimensional sampling points into graph structure data and extract the temporal topological correlation and global structural features between signal sampling points.

[0048] In terms of specific implementation, compared with the shortcomings of traditional graph neural networks, which use dynamic graph construction leading to high computational complexity and inconsistent topological structures of different samples making batch training difficult, and the existing technology of traditional graph pooling directly discarding nodes and destroying signal continuity, the improvement of this process is mainly reflected in: First, by using linear fixed graph mapping (LFGM) linked to the oversampling rate, the uniformity of graph topology and significant reduction of graph construction complexity are achieved while preserving temporal topological continuity; Second, a differentiable hierarchical graph pooling (DiffPool) module is introduced to achieve adaptive top-down aggregation from node-level micro-perturbations to graph-level global structural semantics.

[0049] This invention uses LFGM as the graph mapping method, and the specific implementation is as follows: First, an IQ signal graph is constructed, assuming a given IQ signal... The length of is N, which is expressed as , Map it to a graph structure ,in Represents a set of nodes. This represents the set of edges. The graph contains N nodes, each node... For a sampling point in the IQ sequence, its node characteristics are:

[0050] That is, from the first The feature vector is composed of the IQ values ​​of each sampling point. The set of edges in the graph. Distance based on time location: If the interval between two sampling points does not exceed the hyperparameter Then add an unbounded line between them, that is... ,in Called the connectivity radius, it controls the density of the graph structure. This invention sets it as the oversampling rate of the signal to better preserve local continuity. Since the graph structure corresponding to each signal is topologically identical, the LFGM method only reflects signal differences in node attributes, which significantly simplifies graph construction complexity and unifies graph structure input.

[0051] After completing the graph mapping, this embodiment uses a graph neural network for graph-level classification. Unlike traditional GNN models, this embodiment introduces DiffPool as a graph pooling module, a differentiable graph pooling method that can achieve hierarchical graph representation while maintaining end-to-end trainability. DiffPool can learn the clustering assignment matrix of nodes, constructing a hierarchical representation through clustering. Figure 6 This approach extracts graph features with more global semantics by first encoding the graph nodes using a graph convolutional network, then generating a clustering assignment matrix to learn the mapping from nodes to clusters, and finally constructing a new graph and iterating to complete multi-layer graph abstraction.

[0052] (4) The input of the expert knowledge feature modality and its corresponding neural network (also known as the "knowledge feature network") The input of the expert knowledge feature mode is constructed by extracting the RF-DNA statistical features, Hilbert-Huang transform spectrum statistical features, and dual-tree complex wavelet multi-scale statistical features of the radio frequency signal and combining them into an expert feature vector.

[0053] The neural network corresponding to the expert knowledge feature modality is a deep knowledge feature fusion network, used to extract the physical statistical features of the expert feature vector.

[0054] To further incorporate domain prior knowledge to compensate for the generalization shortcomings of purely data-driven networks in scenarios with small samples and high noise, this embodiment also constructs a knowledge feature network based on the physical statistical properties of the signal. The input of the expert knowledge feature modality extracts RF-DNA statistical features, Hilbert-Huang transform spectrum statistical features, and wavelet domain multi-scale statistical features of the radio frequency signal's IQ signal, aiming to extract physical layer fingerprint features that are validly verified by expert experience.

[0055] In terms of specific implementation, compared to existing technologies that rely solely on single statistical features and easily lead to feature interference and redundancy when fusing multiple features (early fusion), traditional physical layer identification methods are improved in the following ways: First, a multi-dimensional physical fingerprint database covering time-domain segmentation, non-stationary time-frequency energy distribution, and multi-scale translation invariance is comprehensively constructed. Second, a deep mid-level fusion architecture based on multi-branch sub-networks is proposed, which explicitly preserves the specificity of each physical mode while eliminating dimensional differences, significantly improving the discriminative power and robustness of expert features in complex electromagnetic environments. The specific feature extraction and network construction process is as follows.

[0056] 1) Statistical characteristics based on RF-DNA First, receive the radio frequency signal. Converted into three types of instantaneous attribute sequences , and Then, each attribute category is divided into equal-length segments of length N, resulting in R non-overlapping fields, with each segment having a length of L = N / R. Next, four statistical features are extracted from each sub-segment, with instantaneous amplitude as the primary feature. Let its sub-segment be Then take the average. Standard deviation skewness kurtosis Repeat the above process for each attribute (amplitude, phase, frequency) to obtain the following feature dimensions:

[0057] In this embodiment, R=8 was chosen, which ultimately yields a 96-dimensional RF-DNA feature vector. .

[0058] 2) Statistical characteristics constructed based on Hilbert-Huang transform First, the I / Q channels of the orthogonal IQ signals of the radio frequency signal are decomposed into several intrinsic mode function (IMF) components using EMD:

[0059] in Indicates the first One modal component, For the residual term, Then, a Hilbert transform is performed on each IMF component to generate the corresponding analytic signal:

[0060] in This represents the Hilbert transform operator. Represents the imaginary unit. Instantaneous amplitude, The instantaneous phase is used to obtain the instantaneous frequency. Then, a Hilbert spectrum was constructed, and the energy corresponding to each IMF component on the time-frequency plane was plotted. The distribution is mapped onto a two-dimensional grid to form a Hilbert spectrum:

[0061] The obtained spectral dimension is Where M is the number of IMFs and N is the signal length. This spectrum is the core time-frequency representation in the Hilbert-Huang transform, possessing a three-dimensional description capability of time-frequency-energy, and can reflect the energy evolution process of each frequency component within different time periods. After obtaining the spectrum, this invention employs two feature extraction algorithms: an algorithm based on entropy and first- and second-order moments (EM) and a correlation-based (CB) algorithm, extracting expert features from two perspectives: the uniformity of energy distribution and the correlation between spectra.

[0062] For the EM algorithm, this method will The following statistics are calculated as if the image were a two-dimensional grayscale image: Energy entropy: To calculate the uniformity of energy distribution in a spectrum, firstly... Uniformly divided along the time dimension (n) and the frequency dimension (m) into There are three rectangular grid regions. The energy within the i-th and j-th grids is denoted as . Its energy percentage is:

[0063] Energy entropy is defined as:

[0064] First moment:

[0065] Second moment:

[0066] Finally, they are combined into the EM feature vector. .

[0067] Next, we continue calculating the CB features, assuming the spectra of the i-th and j-th samples are... , Then its correlation coefficient is defined as:

[0068] in Representation of spectral matrix The mean, Similarly, the more similar the distribution structures of two spectra, the closer the correlation is to 1; otherwise, it is closer to 0 or a negative value. This invention calculates the correlation coefficient between the spectra of each class's samples and the training samples of other classes, and statistically analyzes its maximum value, average value, and standard deviation to form the final CB feature vector.

[0069] 3) Wavelet domain multi-scale statistical features (i.e., multi-scale statistical features based on wavelet transform) The specific process is as follows: Figure 7 As shown, the DT-CWT structure consists of two parallel filter banks, Tree1 and Tree2. Each tree independently performs a traditional discrete wavelet decomposition process, but outputs the wavelet real and imaginary coefficients of the signal respectively. In each decomposition stage, Tree1 uses a pair of orthogonal filters. , After low-pass and high-pass filtering, downsampling is performed to obtain the first... l Real wavelet coefficients of the layer Similarly, the imaginary wavelet coefficients can be obtained for Tree2. Therefore, the first l The complex wavelet coefficients of the layer are constructed as follows:

[0070] in, It represents the imaginary unit.

[0071] Then calculate instantaneous amplitude Phase With frequency Then, for these signal feature entropies at each scale level, the following statistical features are extracted: variance skewness With kurtosis The calculations for skewness and kurtosis are the same as those for the statistical characteristics of RF-DNA, and the calculation of variance is as follows:

[0072] in express , and Any feature in, This represents the mean. This invention selects three typical sub-regions, performs a 5-level DT-CWT decomposition on each region, and extracts three signal features (amplitude, phase, and frequency) and three statistics for each feature at each level. The final wavelet domain RF fingerprint dimension is: .

[0073] After extracting three types of traditional RF fingerprint features, this embodiment further constructs a deep fusion framework to achieve unified modeling and efficient classification of multimodal expert features. Unlike early fusion methods that directly concatenate all features into a single neural network, this embodiment adopts an intermediate fusion strategy. By constructing a multi-branch structure to model each type of feature separately, extracting their high-level semantic representations, and then fusing them, this effectively improves the complementary expressive power between different features and the discriminative performance of the model.

[0074] Preferably, in this embodiment, the deep knowledge feature fusion network includes: three sets of structurally consistent feature encoding sub-networks, a feature connection layer, and a second classifier; wherein, the three sets of structurally consistent feature encoding sub-networks are used to encode RF-DNA statistical features, Hilbert-Huang transform spectral statistical features, and wavelet domain multi-scale statistical features, respectively; the feature connection layer is used to concatenate the feature vectors output by all feature encoding sub-networks in the channel dimension to obtain a high-dimensional knowledge feature fusion vector; and the second classifier is used to extract the physical statistical features of the expert feature vector from the high-dimensional knowledge feature fusion vector.

[0075] Specifically, RF-DNA statistical features, Hilbert-Huang time-spectrum statistical features, and wavelet domain multi-scale statistical features are input into three sets of structurally consistent feature encoding sub-networks (also known as "fully connected sub-networks") for feature encoding. Each feature encoding sub-network contains two fully connected layers and uses the ReLU activation function. This process allows the three heterogeneous features to extract 64-dimensional deep representations in their respective subspaces, preserving their modality specificity. Subsequently, in the feature connection layer, the three feature vectors are concatenated in the semantic space to form a unified fused representation, called the high-dimensional knowledge feature fusion vector, which serves as the input to the first classifier. The second classifier includes a fully connected layer and an output layer. In the second classifier, the fusion vector is first compressed to 128 dimensions through a fully connected layer, and then a K-dimensional class score vector is generated through the output layer. This output, along with the ground truth labels, is used for supervised optimization of the cross-entropy loss function. The main structure of the deep knowledge feature fusion network is as follows: Figure 8 As shown.

[0076] This concludes the introduction to the four heterogeneous modes of radio frequency signal input and their corresponding neural networks.

[0077] In step S3, the high-order semantic features of each heterogeneous modality are normalized and then concatenated in the channel dimension to obtain fused features. The following steps are performed: norm normalization is applied to the high-order semantic features extracted from each heterogeneous modality; the norm-normalized high-order semantic features of the four heterogeneous modalities are concatenated in the channel dimension to obtain a multimodal fused feature vector.

[0078] Specifically, the higher-order semantic features (i.e., feature vectors) of the time-domain sequence mode, time-frequency image mode, graph structure mode, and expert knowledge feature mode are respectively represented as follows: , , , .right , , , Norm normalization is performed on each of the four heterogeneous modalities. Then, the norm-normalized high-order semantic features of the four modalities are concatenated sequentially along the channel dimension to obtain the multimodal fusion feature vector.

[0079] In this embodiment, the multimodal fusion feature vector is represented as:

[0080] in, The norm normalization operation is as follows:

[0081]

[0082] in, The feature vector to be normalized, Representing vectors Length, represent The One element, Indicates to conduct Norm operations It is the exponent value in the norm formula. This is for absolute value operations.

[0083] As explained above, the multimodal fusion feature vector encompasses complementary information from four modalities: time domain, time-frequency domain, graph structure, and expert knowledge. It preserves the discriminative characteristics of the original signal in different representation spaces and achieves cross-modal feature alignment and fusion through normalization and concatenation operations. This fusion vector is not only high-dimensional and information-density, but also exhibits good scale consistency and semantic separability after nonlinear transformation and normalization of each sub-network, laying a solid foundation for accurate decision-making by the subsequent classifier. Therefore, the category of individual radiation sources can be identified based on the multimodal fusion feature vector. The specific implementation method is explained below.

[0084] In step S4, the multimodal fusion feature vector is input into the first classifier. The output dimension of the first classifier is equal to the number of radiation source individual categories. The first classifier identifies the category of the radiation source individual.

[0085] The first classifier consists of a fully connected layer. In practice, the fused multimodal feature vector is input into this fully connected layer, and the output dimension equals the number of individual categories of the radiation source. This layer directly outputs the raw score for each class, representing the model's confidence in each class. The predicted class is the index corresponding to the maximum value in the output score, i.e. ,in For the first The output score corresponding to the class This is an index for the final predicted radiation source category.

[0086] It should be noted that the prerequisite for implementing the above-mentioned multimodal heterogeneous deep learning radiation source individual identification method that integrates knowledge features is to train the neural network and the first classifier involved in the method so that their combination can achieve the radiation source individual identification function. The specific implementation method is explained below.

[0087] Step C1: Receive radio frequency signal samples from the wireless transmitting device, summarize the categories of each radio frequency signal sample and its corresponding radiation source, and construct a radio frequency signal training set.

[0088] Specifically, the receiving device collects radio frequency (RF) signals radiated from multiple wireless transmitters of the same model but different models. Due to the microscopic physical tolerances of the internal hardware components of each wireless transmitter (such as power amplifier nonlinearity, phase noise, and IQ imbalance), the signals radiated by different devices of the same model each possess inherent individual characteristics. The process involves collecting RF signals from multiple different wireless transmitters. That is, a time-domain signal containing the hardware fingerprint features of the corresponding radiation source individual, with each sample signal having a length of [length missing]. The sampling rate is a fixed value. .

[0089] Step C2: Train the neural networks of the four heterogeneous modes independently using the radio frequency signal training set.

[0090] By independently training four heterogeneous modal neural networks, the independent feature extraction capabilities and classification performance of each heterogeneous modality neural network can be obtained.

[0091] Step C3: Remove the classification layer of each trained heterogeneous modality neural network, and use the feature vector of the input classification layer of each heterogeneous modality neural network as the high-order semantic feature of the corresponding heterogeneous modality; normalize the high-order semantic features extracted from each heterogeneous modality and concatenate them in the channel dimension to obtain the multimodal fusion feature vector.

[0092] This invention further designs a unified fusion training strategy, fusing and modeling the high-order semantic features extracted from the intermediate layers of each model to improve overall recognition performance and model generalization ability. Specifically, the trained MixCNN-Transformer network, ResNet18 image network, DiffPool graph neural network, and deep expert feature fusion network are used as the backbone models for four modalities, respectively. After removing the final classification layer of each network, their feature extraction modules are retained. The hidden state features of the last time step after the MixCNN-Transformer is expanded in step S2, the deep image features output after the global average pooling layer of ResNet18 in step S3, the graph-level structure features obtained by DiffPool through multi-layer graph abstraction and readout function aggregation in step S4, and the knowledge features after dimensionality reduction by concatenating the three feature codes of the deep expert network in steps S5-4 are used as the high-order semantic representations of each intermediate layer, thereby obtaining the high-order semantic features corresponding to IQ sequences, images, graph structures, and expert features as inputs, denoted as:

[0093] in , , , These represent trained sequence networks, image networks, graph neural networks, and expert feature networks, respectively, excluding the final classification layer; , , , These represent the amplitude-phase of the IQ signal, the dual-channel narrow-side image, the graph structure, and the traditional feature input, respectively.

[0094] Step C4: Input the multimodal fusion feature vector of each radio frequency signal sample into the first classifier, and use the category of the radiation source individual corresponding to each radio frequency signal sample as the sample label. Update and optimize the parameters of the first classifier and the neural network of each heterogeneous mode through the backpropagation algorithm to obtain the trained neural network of each heterogeneous mode and the first classifier.

[0095] As described above, in this embodiment, the method utilizes a dataset of labeled individual radiation source signal samples to perform end-to-end training on the constructed network model. During actual training, this embodiment loads the backbone networks of each modality as pre-trained models and fine-tunes their parameters using a small learning rate; simultaneously, a larger learning rate is set for the newly added fusion fully connected layer to accelerate model convergence and improve the model's fusion discrimination ability. Training uses the cross-entropy loss function as the objective function, and the network parameters are optimized using the backpropagation algorithm. The optimizer uses the Adam algorithm, with an initial learning rate set to 0.001. After every 5 training epochs, the learning rate is adjusted to 80% of the current value. Furthermore, when performing fusion training on the DeepSIG2 dataset, the initial learning rate for all network layers except the final classification layer is set to 0.00001. The training batch size is set to 128, and the number of training epochs is set to 50.

[0096] It's also important to note that the first classifier is not fixed but participates in training as part of the network. Under the supervision of a labeled RF signal sample dataset, the classifier parameters and the front-end feature extraction network parameters are updated and optimized together through backpropagation to ensure that the model's final output matches the true category. The labels used for training are the radiation source categories corresponding to each RF signal. This supervision signal provides the model with clear discrimination criteria, enabling the classifier to effectively learn the distinguishing boundaries between different emission sources.

[0097] In practical applications, after acquiring the received signal samples from the device to be identified, the multimodal features are first constructed according to the above process, resulting in corresponding temporal features (amplitude-phase sequence), image features (dual-channel wavelet image), graph structure features (LFGM mapping), and expert statistical feature vectors. These modal features are then input into the trained sequence network, image network, graph network, and expert feature network, respectively. After passing through the feature extraction modules of each network, high-level semantic representations are obtained, followed by normalization and concatenation fusion to form a unified fused feature vector. This vector is then input into the first classifier, and the network outputs predicted scores for each category through forward propagation. Finally, based on the category index corresponding to the maximum value in the output score, the specific radiation source to which the signal sample belongs is determined.

[0098] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0099] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal heterogeneous deep learning method for identifying individual radiation sources by integrating knowledge features, characterized in that, The method includes: Acquire radio frequency signals from wireless transmitting devices; The inputs for the radio frequency signal are constructed in four heterogeneous modes, including time-domain sequence mode, time-frequency image mode, graph structure mode, and expert knowledge feature mode. The inputs of each heterogeneous mode are input into the corresponding neural network, and the high-order semantic features of the corresponding mode are extracted through the corresponding neural network. The neural networks corresponding to the time-domain sequence mode, time-frequency image mode, graph structure mode, and expert knowledge feature mode are respectively a hybrid convolutional-Transformer network, a convolutional neural network, a graph neural network, and a deep knowledge feature fusion network. After normalizing the high-order semantic features extracted from each heterogeneous modality, they are concatenated in the channel dimension to obtain a multimodal fusion feature vector; Based on the multimodal fusion feature vector, the first classifier is used to identify the category of the radiation source individual.

2. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 1, characterized in that, The process involves normalizing the high-order semantic features of each heterogeneous modality and then concatenating them along the channel dimension to obtain fused features, followed by the execution of: Norm normalization is performed on the high-order semantic features extracted from each heterogeneous modality; The higher-order semantic features of the four heterogeneous modalities after norm normalization are concatenated sequentially along the channel dimension to obtain a multimodal fusion feature vector.

3. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 2, characterized in that, The input to the time-domain sequence mode is the orthogonal IQ time-domain signal sequence obtained by demodulating the radio frequency signal; The neural network corresponding to the time-domain sequence mode is used to extract high-order time-domain semantic features of the orthogonal IQ time-domain signal sequence.

4. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 3, characterized in that, The input of the time-frequency image mode is constructed by performing discrete wavelet transform on the orthogonal IQ signals of the radio frequency signal, and then mapping the I / Q channels into a dual-channel image through multi-scale selective reconstruction. The neural network corresponding to the time-frequency image mode is used to extract the time-frequency features of the dual-channel image.

5. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 4, characterized in that, The input of the graph structure mode is constructed by mapping the sampling points in the IQ sequence of the radio frequency signal to graph nodes through linear fixed graph mapping, and constructing edges based on the time neighborhood between sampling points to form a graph structure with a unified topology. The neural network corresponding to the graph structure mode is used to extract the topological features of the graph structure.

6. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 5, characterized in that, The input of the expert knowledge feature modality is constructed by extracting the RF-DNA statistical features, Hilbert-Huang transform spectrum statistical features, and wavelet domain multi-scale statistical features of the radio frequency signal and combining them into an expert feature vector. The neural network corresponding to the expert knowledge feature modality is used to extract the physical statistical features of the expert feature vector.

7. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 6, characterized in that, The deep knowledge feature fusion network comprises: three sets of structurally consistent feature encoding subnetworks, a feature connection layer, and a second classifier; among which, Three sets of structurally consistent feature coding subnetworks are used to encode RF-DNA statistical features, Hilbert-Huang transform spectral statistical features, and wavelet domain multi-scale statistical features, respectively. The feature connection layer is used to concatenate the feature vectors output by all feature encoding subnetworks along the channel dimension to obtain a high-dimensional knowledge feature fusion vector. The second classifier is used to extract the physical statistical features of the expert feature vector from the high-dimensional knowledge feature fusion vector.

8. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 3, characterized in that, The hybrid convolutional-Transformer network includes: The head convolution module is used to perform shallow mapping on the orthogonal IQ time-domain signal sequence and output an RF fingerprint feature map; The time-frequency residual dimensionality reduction module is used to perform multi-scale time-frequency joint dynamic feature extraction on the radio frequency fingerprint feature map to obtain the synergistic fusion features of time-domain dynamics and frequency-domain characteristics; A Transformer encoder is used to aggregate the global dependency information of the collaboratively fused features through a self-attention mechanism and output the temporal features of the orthogonal IQ time-domain signal sequence using classification labels.

9. The multimodal heterogeneous deep learning radiation source individual identification method based on the fusion of knowledge features according to any one of claims 1-8, characterized in that, The method further includes training a neural network with four heterogeneous modalities and a first classifier, wherein the training is performed as follows: Receive radio frequency signal samples from wireless transmitting devices, summarize the categories of each radio frequency signal sample and its corresponding radiation source, and construct a radio frequency signal training set; The neural networks of four heterogeneous modes were trained independently using a radio frequency signal training set; Remove the classification layer of each trained heterogeneous modality neural network, and use the feature vector of the input classification layer of each heterogeneous modality neural network as the high-order semantic feature of the corresponding heterogeneous modality; normalize the high-order semantic features extracted from each heterogeneous modality and concatenate them in the channel dimension to obtain the multimodal fusion feature vector. The multimodal fusion feature vectors of each radio frequency signal sample are input into the first classifier, and the category of the radiation source individual corresponding to each radio frequency signal sample is used as the sample label. The parameters of the first classifier and the neural network of each heterogeneous mode are updated and optimized by the backpropagation algorithm to obtain the trained neural network of each heterogeneous mode and the first classifier.

10. The multimodal heterogeneous deep learning radiation source individual identification method integrating knowledge features according to claim 9, characterized in that, The output dimension of the first classifier is equal to the number of individual categories of the radiation source.

Images