A communication radiation source cross-mode identification method based on multi-modal information fusion
By constructing a multimodal information fusion network and utilizing convolutional neural networks, graph convolution, and graph shrinkage techniques, the problem of insufficient single-modal data in cross-modal recognition of communication radiation sources is solved, achieving higher cross-modal recognition accuracy and robustness.
Patent Information
- Application Number
- CN202210750915.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-06-28
- Publication Date
- 2025-10-24
- Estimated Expiration
- 2042-06-28
AI Technical Summary
In the existing technology of cross-modal recognition of communication radiation sources, the recognition ability of the method based on single-modal data is insufficient, especially in the absence of data, the recognition ability is significantly reduced.
By constructing a multimodal information fusion network, using convolutional neural networks to extract multimodal features, and performing information fusion through inter-modal relationship graphs and graph convolution and graph shrinkage techniques, cross-modal recognition is achieved.
While maintaining the same-modal recognition capability, the cross-modal recognition accuracy is significantly improved, and a high recognition accuracy can still be maintained when some modal information is missing, showing modal robustness.
Smart Images
Figure CN115952466B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the fields of communications and artificial intelligence, and in particular to a cross-modal recognition technology for communication radiation sources based on multimodal information fusion. Background Art
[0002] The mode of a communication emitter refers to the combination of channel access methods, fixed frequencies, transmission rates, and modulation schemes used to achieve communication. A communication emitter often has multiple modes. Under normal circumstances, a communication emitter will operate in a few fixed modes; however, in emergency situations, such as during wartime, other modes may be used. Cross-mode identification of communication emitters aims to identify the type of communication emitter in emergency mode data using data from the source's normal mode.
[0003] Typical communication radiation source identification methods include expert system-based communication radiation source identification methods and data-driven communication radiation source identification methods.
[0004] An expert system-based communication emitter identification method: This method establishes a specialized communication emitter database and uses domain knowledge to extract communication emitter characteristics, enabling same-mode and cross-modal recognition of communication emitters. This method relies on the expertise of domain experts, and the system's performance largely depends on manually extracted expert knowledge. However, manually extracting knowledge to identify communication emitter sources is difficult for highly variable cross-modal recognition.
[0005] Data-driven communication emitter identification methods use known communication emitter signal data to train a neural network, enabling it to classify known communication emitter signals, perform same-mode identification, and perform cross-mode identification. This method requires a large amount of electromagnetic data on known communication emitter patterns. While it excels at identifying same-mode data, it suffers from significant loss of recognition for cross-mode data where data is lacking. Summary of the Invention
[0006] The technical solution adopted by the present invention to solve the above-mentioned technical problems is to solve the problem of insufficient cross-modal recognition ability of communication radiation sources in existing methods based on single-modal data. A method is proposed by introducing multimodal information of communication radiation sources, constructing inter-modal relationship graph structure data, and fusing modal information with stable cross-modal characteristics with traditional communication data for multimodal information, thereby forming a new cross-modal recognition method for communication radiation sources.
[0007] The technical solution adopted by the present invention to solve the above technical problems is to provide a cross-modal identification method for communication radiation sources based on multimodal information fusion, comprising the following steps:
[0008] Step one, a convolutional neural network is constructed and trained as a multi-modal information fusion network to realize multi-modal feature extraction and cross-mode sample radiation source class determination; during training, the same mode communication radiation source sample set is used as the training sample set, and the training sample set includes multi-modal and single-modal samples; the multi-modal at least includes communication data information and modulation information, and selectively includes image information and text information;
[0009] Step two, extract each modal feature: the modulation feature is composed of modulation parameters; for text information, a text space embedding method is used to map to text features; communication data features and image features are extracted by communication data information and image information through the intermediate layer of the convolutional neural network;
[0010] Step three, use the prior model to construct the inter-modal relationship graph, each feature extracted in the inter-modal relationship graph is a node, and all feature nodes have edges with the modulation feature node; when there are more than two text features, there are edges between different text feature nodes;
[0011] Step four, sort the nodes from large to small according to the degree in the inter-modal relationship graph, the more edges a node has, the higher its degree; the feature nodes with the same degree are divided into a subgraph area, and the multi-modal information fusion network finally shrinks a node as the final cross-mode data vector through multi-layer graph convolution and multiple graph contraction in the inter-modal relationship graph; each layer of graph convolution is used to complete the feature information exchange of the node, and each time of graph contraction is used to contract the node with the highest degree and the subgraph area with the second highest degree into a node; finally, a full connection layer is used to reduce the dimension of the final cross-mode data vector;
[0012] Step five, the cross-mode data vector in the multi-modal information fusion network is input into the softmax layer, and the softmax layer outputs the decision vector to realize the cross-mode recognition of the communication radiation source.
[0013] The application first collects multi-modal information of the communication radiation source, and uses a feature extraction algorithm to extract features from the corresponding modal information. The multi-modal features are represented as nodes in the inter-modal relationship graph, and the edge relationship between the modal nodes is constructed by the prior model. The information between the modal nodes is transmitted through the graph convolutional neural network, and the information of the modal nodes is fused through the graph contraction technology, which avoids the problem of excessive dependence on electromagnetic data features in traditional algorithms, so that multi-modal information can effectively participate in the cross-mode recognition of the communication radiation source. While ensuring that the same mode recognition ability remains basically unchanged, the cross-mode recognition ability of the communication radiation source is greatly improved compared with the traditional algorithm.
[0014] The inter-modal relationship graph of the application is composed of traditional communication data features, modulation features, image features, text features and other multi-modal features. The modal features are used as nodes of the inter-modal relationship graph structure data, and the prior model between the modes is used as the edge of the inter-modal relationship graph structure data. Graph contraction refers to gradually merging the two ends of the edge in the graph into a node through node information exchange fusion. The application uses graph contraction and multi-modal information of communication radiation sources to contract the inter-modal relationship graph structure data into a feature vector, realizing cross-mode recognition of communication radiation sources based on multi-modal information fusion.
[0015] The beneficial effects of the application are:
[0016] 1. Under the condition of only using the same mode electromagnetic data of communication radiation sources and other modal information, not only excellent recognition accuracy of communication radiation sources in the same mode is obtained, but also excellent recognition accuracy of communication radiation sources in the cross mode is obtained.
[0017] 2. It has modal robustness. The identified samples can lack part of the modal information, and better same mode and cross mode recognition accuracy can still be achieved in the case of partial or complete loss of text information and image information.
[0018] 3. Strong expansion capability. Under the prior model, not only the modal information mentioned above can be introduced, but also other multi-modal information can be introduced to construct the inter-modal relationship graph. More modal information fusion is realized through more frequent graph convolution and graph contraction technology. BRIEF DESCRIPTION OF DRAWINGS
[0019] Figure 1 The multi-modal interrelationship graph is for an embodiment.
[0020] Figure 2 The method flowchart is for an embodiment. DETAILED DESCRIPTION
[0021] The embodiment can be divided into five steps, as shown in Figure 2
[0022] Step one, through a convolutional neural network as a multi-modal information fusion network to extract the features of communication data information, modulation information, image information and text information, and the multi-modal features composed of communication data features, modulation features, image features, text features and other multi-modal features realize the compression of modal information, and the multi-modal features are represented as a plurality of one-dimensional vectors, and finally the radiation source category determination of cross-mode samples is completed. During training, the same mode communication radiation source sample set is used as the training sample set, and the training sample set includes multi-modal and single-modal samples; the multi-modal at least includes communication data information and modulation information, and selectively includes image information and text information; the same mode communication radiation source sample set is used for training, and the multi-modal and single-modal samples in the training sample set each account for half.
[0023] Setting the sample center high-dimensional vector g of each mode type using prior knowledge l , l represents the serial number variable of mode type. A convolutional neural network is used, and the weights in each network hidden layer are represented as M i , the bias coefficient is b i , the activation function is activate i , i is the serial number variable of the hidden layer, and the value range is 1 to last, and then the iteration training process of the convolutional neural network is as follows:
[0024] 1) Randomly initialize the hidden layer weights M i and the bias coefficient b i ;
[0025] 2) Calculate the hidden layer output output i = activate i (M i-1 ×M i +b i ) layer by layer, M 0 =X, until the output output last of the last hidden layer; X is the input training sample;
[0026] 3) Calculate the loss loss(output last ,g l ) of the network;
[0027] 4) Update the hidden layer weights M i and the bias coefficient b i according to the loss;
[0028] Repeat 2) to 4) until the loss reaches the preset threshold, the network completes convergence, the training of the convolutional neural network is completed, and the network parameters are fixed. The trained convolutional neural network can determine the radiation source category of cross-mode samples.
[0029] Step two, for text information, it can be mapped into a feature vector using a time text space embedding method. The modulation feature is composed of modulation parameters. Both the communication data feature and the image feature are extracted using a convolutional neural network. For image information G, the output of the middle layer of the trained convolutional neural network is used as the image feature vector of the image information G.
[0030] For communication data information s, the i-th information segment s i is obtained by splicing and cropping. i The short-time Fourier transform is used to obtain a time-frequency matrix X i (m, k), where m and k are the row number and column number of the time-frequency matrix, respectively, k ranges from 0 to N-1, and m ranges from 0 to M-1. Let X M-1 (m, k) = [T0, T1, T2,..., T i ], and represents transposition. Then let (m, k) = [P0, P1, P2,..., P , and S M-1 (m, k) = [P0, P1, P2,..., P i ]. The middle layer of the trained convolutional neural network is used to extract the communication data feature vector of S
[0031] Step three, the knowledge about the relationship between modalities in the prior model and the multi-modal feature extraction are used to construct a modal relationship graph x.
[0032] Through the feature extraction of multi-modal information, the complexity of multi-modal information is reduced. According to the relationship between modalities in the prior model, the modal features are filled into the corresponding nodes of the modal relationship graph, and the nodes are connected by edges: each feature is a node, and all the nodes of the features have edges with the modulation feature node. When there are more than two text features, the nodes of the text features also have edges between each other.
[0033] As Figure 1 described, the embodiment multi-modal information includes modulation features, communication data features, image features, and two text features. The text feature 1 node, the text feature 2 node, the communication data feature node, and the image feature node all have edges with the modulation feature node, and the text feature 1 node and the text feature 2 node have edges between each other.
[0034] In the construction of the modal relationship graph x, in addition to the modulation features and the communication data features, other modalities are not necessary and can be missing.
[0035] Step four, the same mode data features are fused into a vector as a cross-mode data sample using graph convolution and graph contraction of the convolutional neural network, realizing the same mode recognition of the communication radiation source:
[0036] After the inter-modal relation graph x is constructed, the first layer graph convolution h θ The feature information exchange of the modal nodes is completed, that is, where σ is an activation function, and U is an orthogonal matrix of the Laplacian transformation matrix. According to the ordering of the node degrees from large to small: x = (x m ,x w1 ,x w2 ,x c ,x p ), since all nodes have edges with the modulation feature node, the degree of the modulation feature node x m is the highest; the text feature node has edges with the modulation feature node and other text feature nodes, so the degree of the text feature 1 node x w1 is the same as that of the text feature 2 node x w2 , and the second and third are randomly ordered, x w1 ,x w2 form a subgraph region, which is the second highest subgraph region; the communication data feature node x c and the image feature node x p only have edges with the modulation feature node, and the fourth and fifth are ordered, x c ,x p form a subgraph region.
[0037] Each time the current highest degree node and the second highest degree subgraph region are graph contracted to finally contract into a node: first, the highest degree modulation feature node x m and the second highest degree subgraph region x w1 ,x w2 are graph contracted, and the convolutional neural network outputs y = (y i ,y c ,y p ), y i is the output of x m , x w1 , x w2 after the i-th layer graph convolution and graph contraction of the convolutional neural network, and then the current highest degree modulation feature node y i and the second highest degree subgraph region y c ,y p are graph contracted and graph contracted, and multiple layers of graph convolution and multiple times of graph contraction make the network output graph y i finally contract into a node. Then, multiple fully connected layers are used to output the feature vector y iDimensionality reduction is performed, so that it can be used as a cross-mode data vector which is judged by a softmax layer and the loss of sample labels is calculated. Embodiments are constructed in this way, and the sample robustness is strong, and the same mode sample participating in the convolutional neural network training does not require to have all the modes, and the loss of part of the mode information is allowed.
[0038] Step five, the convolutional neural network inputs the cross-mode data vector into the softmax layer to output a judgment vector, and realizes cross-mode recognition of the communication radiation source.
[0039] The application utilizes the cross-mode feature stability of the non-electromagnetic mode, the same mode cross-mode recognition ability, and realizes the communication radiation source multi-mode information fusion technology under the guidance of the prior model through the graph convolution and graph contraction technology. Not only the cross-mode recognition ability is greatly improved compared with the traditional convolutional neural network, but also the same mode recognition ability of the multi-mode information fusion network is kept at the same level as the traditional network.
Claims
1. A communication emitter cross-mode identification method based on multi-modal information fusion, characterized in that, Comprise the following steps: Step one, build and train a convolutional neural network as a multi-modal information fusion network to realize multi-modal feature extraction and cross-mode sample radiation source class determination; During training, use the same mode of communication radiation source sample set as the training sample set, and the training sample set includes multi-modal and single-modal samples; Multi-modal at least includes communication data information and modulation information, and selectively includes image information and text information; Step two, extract each modal feature: modulation feature is composed of modulation parameters; For text information, use text space embedding method to map to text feature; Communication data feature and image feature are extracted by communication data information and image information through the intermediate layer of convolutional neural network; Step three, use prior model to build inter-modal relationship graph, each feature extracted in the inter-modal relationship graph is a node, all feature nodes have edges with modulation feature node, when there are more than 2 text features, there are edges between different text feature nodes; Step four, sort the nodes from large to small according to the degree in the inter-modal relationship graph, the more edges the node has, the higher its degree; The degree of the same feature node is divided into a subgraph area, and the multi-modal information fusion network shrinks each node in the inter-modal relationship graph to a node as the final cross-mode data vector through multi-layer graph convolution and multiple graph contraction, each layer of graph convolution is used to complete the feature information exchange of the node, and each time of graph contraction is used to contract the node with the highest degree and the subgraph area with the second highest degree into a node; Finally, use the full connection layer to reduce the dimension of the final cross-mode data vector; Step five, input the cross-mode data vector in the multi-modal information fusion network into the softmax layer, and the softmax layer outputs the decision vector to realize the cross-mode recognition of the communication radiation source.
2. The method of claim 1, wherein, The training process of the multi-modal information fusion network is: Setting the sample center high-dimensional vector g of each communication radiation source mode type with prior knowledge l , l represents the serial number variable of the mode type, and the weight in each network hidden layer of the neural network is represented as M i , the bias coefficient is b i , the activation function is activate i , i is the serial number variable of the hidden layer, and the value range is 1 to last, and then the iteration process is: 1) Randomly initialize the hidden layer weights M i and bias coefficients b i ; 2) Calculate the hidden layer output output layer by layer i = activate i (M i-1 × M i + b i ), M 0 = X, until the output of the last layer of the hidden layer output last ; X as the training sample input; 3) compute the loss of the network loss(output last , g l ); 4) Update each hidden layer weights M by backpropagation of the loss i and bias coefficients b i ; Repeat 2) to 4) until the loss reaches the preset threshold, and the training of the convolutional neural network is completed.
3. The method of claim 1, wherein, The training sample set includes half of the multi-modal and single-modal samples.
4. The method of claim 1, wherein, The communication data information is spliced and cropped multiple information segments before being input into the convolutional neural network, and a short-time Fourier transform is used to obtain a time-frequency matrix X of each information segment i (m, k), m and k are respectively the column number and row number of the time-frequency matrix, k is in the range of 0 to N-1, and m is in the range of 0 to M-1; each column in the time-frequency matrix X i (m, k) is T m , represents transposition, a k,m represents the kth element in each column T m , and a transformed column P m , The transformed matrix S m is composed of M-1 columns P i (m, k) = [P0, P1, P2, …, P M-1 ], and the transformed matrix S i (m, k) is input into the convolutional neural network.
5. The method of claim 1, wherein, The method for the feature information exchange of the graph convolution completed node is: x is the input inter-modal relation graph, h θ is the graph convolution parameter, σ is an activation function, U is an orthogonal matrix of a Laplace transform matrix, represents transposition, and y is the graph convolution output.