Multimodal data fusion method and system for modulation signal classification and recognition

Abstract

The present invention relates to the field of multimodal technology, and specifically discloses a multimodal data fusion method and system for modulation signal classification and recognition, wherein the method comprises: obtaining an original IQ signal; performing maximum value normalization and minimum value normalization processing on the amplitude of the original IQ signal, and generating an IQ sequence signal according to the normalized IQ signal; calculating the amplitude and phase of an AP sequence signal according to the original IQ signal, and generating an AP sequence signal; converting the original IQ signal into a set of points on a complex plane, and generating an IQ color density constellation diagram; projecting the original IQ signal in the amplitude-phase dimension, and generating an AP scatter plot; extracting sequence modal signal features; extracting image modal signal features; performing feature fusion according to sequence modal signal features and image modal signal features, and obtaining joint features; mapping the joint features to each modulation category, and outputting the modulation category corresponding to the maximum probability as a modulation signal classification and recognition result.

Description

Technical Field

[0001] The present invention relates to the field of multimodal technology, and in particular to a multimodal data fusion method and system for modulation signal classification and recognition. Background Art

[0002] 5G communication technology can generate a large amount of communication data, and deep learning technology in artificial intelligence can learn the features in a large amount of data. Therefore, the joint development of artificial intelligence technology and wireless communication technology is the current mainstream development direction. This patent aims to study the relevant algorithms that combine automatic modulation classification (AMC) technology with deep learning technology in the communication physical layer. AMC refers to the automatic identification of the modulation mode of the signal without or with partial prior knowledge. This process is an intermediate step between signal detection and signal demodulation. It is a prerequisite for demodulating the signal at the receiving end and a key link in wireless communication. AMC plays a key role in both civil and military fields.

[0003] Deep learning is a powerful artificial intelligence technology that can automatically learn features from massive amounts of data and fit nonlinear networks. It has achieved great success in research fields such as computer vision and speech recognition. Automatic modulation recognition based on deep learning has therefore received widespread attention. Currently, automatic modulation recognition methods based on deep learning are mainly divided into two categories.

[0004] The first type is to convert the signal into an image, and then use the deep learning algorithm in the field of image recognition to classify the modulated signal. For example, the signal is converted into a time-frequency diagram, constellation diagram, amplitude histogram, cyclic spectrum diagram, eye diagram, etc., and then classified using a neural network; the second type is to directly use a neural network to extract features from the signal sequence and complete the classification. For example, directly use the in-phase orthogonal signal sequence (IQ), amplitude phase signal sequence (AP), frequency sequence, etc. as the input of the neural network for classification.

[0005] In these two types of AMC methods, the signal of the image modality has spatial distribution characteristics, and the signal of the sequence modality has temporal distribution characteristics. Therefore, this patent extracts more effective features by fusing the data of the two modalities into the same network to obtain better classification performance. Summary of the invention

[0006] In view of the above problems, the purpose of the present invention is to provide a multimodal data fusion method and system for modulation signal classification and recognition, which improves the recognition effect compared with the method using single modal data as input.

[0007] The present invention provides a multimodal data fusion method for modulation signal classification and recognition, comprising:

[0008] Step S101, obtaining an original IQ signal;

[0009] Step S102, performing maximum value normalization and minimum value normalization processing on the amplitude of the original IQ signal, and generating an IQ sequence signal according to the normalized IQ signal;

[0010] Step S103, calculating the amplitude and phase of the AP sequence signal according to the original IQ signal, and generating the AP sequence signal according to the amplitude and phase;

[0011] Step S104, converting the original IQ signal into a set of points on a complex plane to generate an IQ color density constellation diagram;

[0012] Step S105, projecting the original IQ signal in the amplitude-phase dimension to generate an AP scatter plot;

[0013] Step S106, extracting sequence modal signal features according to the IQ sequence signal and the AP sequence signal;

[0014] Step S107, extracting image modality signal features according to the IQ color density constellation diagram and the AP scatter diagram;

[0015] Step S108, performing feature fusion according to the sequence modality signal feature and the image modality signal feature to obtain a joint feature;

[0016] Step S109: Map the joint feature to each modulation category, and output the modulation category corresponding to the maximum probability as a modulation signal classification and recognition result.

[0017] In a possible implementation, step S102 includes:

[0018] The received original IQ signal (r I , r Q ) The amplitude is normalized to the maximum value and the minimum value, the range is (-1, 1), the formula is as follows:

[0019]

[0020] The IQ sequence signal samples of length n are organized in the matrix S IQ In the above example, an IQ sequence signal S is generated. IQ , the formula is as follows:

[0021]

[0022] In the formula, (r I , r Q ) is the original IQ signal, r I is the in-phase component, r Q is the orthogonal component, S IQis the IQ sequence signal, n is the length, r I (n) is the IQ signal after normalization of the minimum amplitude, r Q (n) is the IQ signal after normalization of the maximum amplitude.

[0023] In a possible implementation, step S103 includes:

[0024] The amplitude is normalized by L2 norm, and the normalized amplitude range is (0, 1). The phase is normalized between (-1, 1). The formula is as follows:

[0025]

[0026]

[0027] Organize the AP sequence signal samples of length n in the matrix S AP In the process, AP sequence signal S is generated. AP , the formula is as follows:

[0028]

[0029] In the formula, (r I , r Q ) is the original IQ signal, r I is the in-phase component, r Q is the orthogonal component, S AP is the AP sequence signal, n is the length, r A (n) is the amplitude, r Q (n) is the phase.

[0030] In a possible implementation, step S104 includes:

[0031] The original IQ signal (r I , r Q ) is converted to a set of points on the complex plane, the formula is as follows:

[0032] I IQ =[r I (n), r Q (n)]=|(r I (0), r Q (0)), (r I (1), r Q (1)),…,(r I (n-1), r Q (n-1))|

[0033] In the formula, (r I , r Q ) is the original IQ signal, r Iis the in-phase component, r Q is the orthogonal component, I IQ is the set of points in the IQ color density constellation diagram, and n is the length.

[0034] In a possible implementation, step S105 includes:

[0035] Project the original IQ signal in the amplitude-phase dimension, the formula is as follows:

[0036] I AP =[r A (n), r P (n)]=[(r A (0), r P (0)), (r A (1), r P (1)),…,(r A (n-1), r P (n-1))]

[0037] In the formula, r A (n) is the amplitude, r Q (n) is the phase, I AP is the set of points in the IQ color density constellation diagram, and n is the length.

[0038] In a possible implementation, step S106 includes:

[0039] Performing convolution operations with convolution kernel sizes of 1×2, 3×1, and 3×1 on the IQ sequence signal in sequence, and extracting channel features of different scales and dimensions to obtain a first feature;

[0040] The AP sequence signal is sequentially subjected to convolution operations with convolution kernel sizes of 1×2, 3×1, and 3×1, and channel features of different scales and dimensions are extracted to obtain a second feature;

[0041] vertically splicing the first feature and the second feature together to obtain a sequence signal fusion feature;

[0042] The sequence signal fusion features are input into the LSTM network to extract the time series features of the sequence signal to obtain the sequence modal signal features.

[0043] In a possible implementation, step S107 includes:

[0044] Perform convolution operations with convolution kernel sizes of 7×7, 5×5, and 3×3 on the IQ color density constellation map in sequence, and extract spatial features of different depths of the image to obtain a third feature;

[0045] Convolution operations with kernel sizes of 7×7, 5×5, and 3×3 are performed on the AP scatter plot in sequence, and spatial features of different depths of the image are extracted to obtain the fourth feature.

[0046] vertically splicing the third feature and the fourth feature together to obtain an image signal fusion feature;

[0047] The image signal fusion feature is input into the fully connected layer, and after nonlinear transformation, it is mapped to the sequence modal signal feature dimensional space to obtain the image modal signal feature.

[0048] In a possible implementation, step S108 includes:

[0049] In the network feature layer, the sequence modality signal features and the image modality signal features are fused through a connection function to obtain a joint feature.

[0050] In a possible implementation, step S109 includes:

[0051] The joint feature is mapped to each modulation category according to the cross entropy loss function, and the modulation category corresponding to the maximum probability is output as the modulation signal classification recognition result.

[0052] The present invention also provides a multimodal data fusion system for modulated signal classification and recognition, which is used to execute any of the above multimodal data fusion methods, including:

[0053] A signal acquisition module, used to acquire the original IQ signal;

[0054] An IQ sequence signal generating module, used for performing maximum value normalization and minimum value normalization processing on the amplitude of the original IQ signal, and generating an IQ sequence signal according to the normalized IQ signal;

[0055] An AP sequence signal generating module, configured to calculate the amplitude and phase of the AP sequence signal according to the original IQ, and generate the AP sequence signal according to the amplitude and phase;

[0056] An IQ color density constellation diagram generating module, used for converting the original IQ signal into a set of points on a complex plane to generate an IQ color density constellation diagram;

[0057] An AP scatter plot generation module, configured to project the original IQ signal in an amplitude-phase dimension to generate an AP scatter plot;

[0058] A sequence mode signal feature extraction module, used to extract sequence mode signal features according to the IQ sequence signal and the AP sequence signal;

[0059] An image modality signal feature extraction module, used to extract image modality signal features according to the IQ color density constellation diagram and the AP scatter diagram;

[0060] A feature fusion module, used for performing feature fusion according to the sequence modality signal features and the image modality signal features to obtain a joint feature;

[0061] The output module is used to map the joint feature to each modulation category, and output the modulation category corresponding to the maximum probability as the modulation signal classification and recognition result.

[0062] The purpose of the present invention is to provide a multimodal data fusion method and system for modulation signal classification and recognition, which generates multimodal data through the original IQ signal, then fuses the multimodal data in the feature layer of the neural network, and finally trains the network model under the supervision of the loss function to complete the classification of the modulated signal. Compared with the method of using single modal data as input for modulation recognition, the recognition effect of this method is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A flowchart of a multimodal data fusion method provided by an embodiment of the present invention;

[0064] Figure 2 A visualization diagram of IQ sequence signal samples provided by an embodiment of the present invention;

[0065] Figure 3 A sample visualization diagram of an AP sequence signal provided by an embodiment of the present invention;

[0066] Figure 4 A common density constellation diagram provided by an embodiment of the present invention;

[0067] Figure 5 A color density constellation diagram provided for an embodiment of the present invention;

[0068] Figure 6 A three-dimensional scatter plot provided by an embodiment of the present invention;

[0069] Figure 7 Amplitude-phase dimension projection diagram provided by an embodiment of the present invention;

[0070] Figure 8 The overall architecture diagram of the AMFN network provided by the embodiment of the present invention. DETAILED DESCRIPTION

[0071] The following detailed description of the embodiments of the present invention is further described in detail in conjunction with the accompanying drawings and examples. The detailed description of the following embodiments and the accompanying drawings are used to exemplarily illustrate the principles of the present invention, but cannot be used to limit the scope of the present invention, that is, the present invention is not limited to the preferred embodiments described, and the scope of the present invention is defined by the claims.

[0072] In the description of the present invention, it should be noted that, unless otherwise specified, “plurality” means two or more than two; the terms “first”, “second”, etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance; for ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to the specific circumstances.

[0073] After the signal is sent by the transmitter, it reaches the receiving end through the wireless channel and is received as the original IQ signal. I , r Q ) to calculate the multimodal input, and obtain 4 input vectors S for 2 modes in total IQ , S AP , I IQ , I AP . Where S represents the sequence modality and I represents the image modality.

[0074] Assume that the set of modulation categories to be classified is M={1, 2, ...m}, where m is the number of modulation categories. The present invention aims to design a neural network f, which can classify modulation signals based on sequence modal signals and image modal signals.

[0075] M=f(S IQ , S AP , I IQ , I AP )

[0076] f is a nonlinear mapping function from multimodal data input to modulation classification output. The parameters of the function are obtained by neural network training and optimized through iterative training. I , r Q ) to compute multimodal input.

[0077] Figure 1 A flowchart of a multimodal data fusion method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the present invention provides a multimodal data fusion method for modulation signal classification and recognition, comprising:

[0078] Step S101, obtaining an original IQ signal;

[0079] In a possible implementation, the original IQ signal consists of two data vectors, the in-phase component rI and the orthogonal component r Q .

[0080] Step S102, performing maximum value normalization and minimum value normalization processing on the amplitude of the original IQ signal, and generating an IQ sequence signal according to the normalized IQ signal;

[0081] In a possible implementation, the received original IQ signal (r I , r Q ) The amplitude is normalized to the maximum value and the minimum value, the range is (-1, 1), the formula is as follows:

[0082]

[0083] The IQ sequence signal samples of length n are organized in the matrix S IQ In the above example, an IQ sequence signal S is generated. IQ , the formula is as follows:

[0084]

[0085] In the formula, (r I , r Q ) is the original IQ signal, r I is the in-phase component, r Q is the orthogonal component, S IQ is the IQ sequence signal, n is the length, r I (n) is the IQ signal after normalization of the minimum amplitude, r Q (n) is the IQ signal after normalization of the maximum amplitude.

[0086] Figure 2 The IQ sequence signal sample visualization diagram provided by the embodiment of the present invention has a horizontal axis representing the time series (sampling points) and a vertical axis representing the I-channel data and the Q-channel data.

[0087] Step S103, calculating the amplitude and phase of the AP sequence signal according to the original IQ signal, and generating the AP sequence signal according to the amplitude and phase;

[0088] In a possible implementation, the amplitude is normalized by L2 norm, and the normalized amplitude range is (0, 1), and the phase is normalized between (-1, 1), and the formula is as follows:

[0089]

[0090]

[0091] Organize the AP sequence signal samples of length n in the matrix S AP In the process, AP sequence signal S is generated.AP , the formula is as follows:

[0092]

[0093] In the formula, (r I , r Q ) is the original IQ signal, r I is the in-phase component, r Q is the orthogonal component, S AP is the AP sequence signal, n is the length, r A (n) is the amplitude, r Q (n) is the phase.

[0094] Figure 3 The AP sequence signal sample visualization diagram provided in the embodiment of the present invention has a horizontal axis representing a time series (sampling points) and a vertical axis representing A (amplitude) and P (phase).

[0095] Step S104, converting the original IQ signal into a set of points on a complex plane to generate an IQ color density constellation diagram;

[0096] In a possible implementation, step S104 includes:

[0097] The original IQ signal (r I , r Q ) is converted to a set of points on the complex plane, the formula is as follows:

[0098] I IQ =[r I (n), r Q (n)]=[(r I (0), r Q (0)), (r I (1), r Q (1)),…,(r I (n-1), r Q (n-1))]

[0099] In the formula, (r I , r Q ) is the original IQ signal, r I is the in-phase component, r Q is the orthogonal component, I IQ is the set of points in the IQ color density constellation diagram, and n is the length.

[0100] Draw the constellation diagram of the original IQ signal. Each signal sample point corresponds to a constellation point drawn on the complex plane. Then draw a 10×10 grid to divide the constellation diagram, such as Figure 4As shown in the figure, the density of the constellation points in each grid is counted. Because the constellation diagrams of different modulation signals have different distribution characteristics, we divide the density of the constellation points in the grid into 4 levels, and use different colors to color the constellation points in grids of different levels. The constellation points in the grids with a density less than 5 are regarded as noise points and abandoned. The resulting colored density constellation diagram is as follows: Figure 5 As shown, it has richer features than ordinary constellation diagrams.

[0101] Figure 4 The ordinary density constellation diagram provided in the embodiment of the present invention has the I-channel data on the horizontal axis and the Q-channel data on the vertical axis, and then covers a 10×10 grid to count the number of points in each grid (calculate the density), and then according to the size of the density, the points in the grids of different densities are colored differently.

[0102] Figure 5 The color density constellation diagram provided by the embodiment of the present invention colors the points in different density grids with different colors according to the density, thereby forming a color density constellation diagram.

[0103] In an example, red: >45 means that if the number of points in a grid is greater than 45, all the points in the grid are colored red. Purple: (30,45) means that if the number of points in a grid is greater than 30 and less than or equal to 45, all the points in the grid are colored purple. Blue: (20,30) means that if the number of points in a grid is greater than 20 and less than or equal to 30, all the points in the grid are colored blue. Green: (10,20) means that if the number of points in a grid is greater than or equal to 10 and less than or equal to 20, all the points in the grid are colored green. Discard: <10 means that if the number of points in a grid is less than 10, the points in the grid are discarded.

[0104] Step S105, projecting the original IQ signal in the amplitude-phase dimension to generate an AP scatter plot;

[0105] In a possible implementation, step S105 includes:

[0106] Project the original IQ signal in the amplitude-phase dimension, the formula is as follows:

[0107] I AP =[r A (n), r P (n)]=[(r A (0), r P (0)), (r A (1), r P (1)),…,(rA (n-1), r P (n-1))]

[0108] In the formula, r A (n) is the amplitude, r Q (n) is the phase, I AP is the set of points in the IQ color density constellation diagram, and n is the length.

[0109] The original IQ signal (such as Figure 6 ) is projected in the amplitude-phase dimension to obtain an AP scatter plot (as shown in Figure 7 In the 3D graph of the original IQ signal, the feature expression is not obvious. After projecting to the amplitude-phase dimension, the amplitude and phase of the original IQ signal have obvious distribution characteristics, and it can be seen that the amplitude is concentrated near a certain phase.

[0110] Figure 6 The three-dimensional scatter plot provided in the embodiment of the present invention has a time series on the X-axis, A (amplitude) on the Y-axis, and P (phase) on the Z-axis, and then a three-dimensional graph is drawn. The different colors in this graph represent different types of modulation signals. There are 11 categories in total, so there are 11 colors in total. In the three-dimensional graph, the points of 11 colors are mixed together. It can be clearly seen in the AP projection that the distribution of the 11 colors presents different characteristics. Therefore, for the network, the features for identifying the 11 types of signals can be extracted through the AP graph.

[0111] Figure 7 The amplitude-phase dimension projection diagram provided by the embodiment of the present invention is Figure 6 The three-dimensional graph is rotated to form projection graphs in the A (amplitude) and P (phase) dimensions, from which the distribution characteristics of A and P can be found.

[0112] S IQ , S AP , I IQ , I AP These four types of signals constitute a joint data set for modulation identification, and the data sets share one set of labels. It should be noted that the four types of signals in the joint data set are all generated based on the original IQ signals and are not independent of each other. Therefore, although the four types of signals have different characteristics, their characteristics are inherently related, which is the basis for extracting complementary features between different modal signals.

[0113] This paper proposes a multimodal feature fusion network model AMFN for automatic modulation recognition. The entire model is divided into two parts according to the modality of the input signal, one is the sequence modality signal part, and the other is the image modality signal part. The network architecture is as follows Figure 8As shown. The inputs of the sequence modality signal are IQ sequence signal and AP sequence signal. The data format of the signal is N×2, where N is the time series and 2 is the two columns of data stored. The inputs of the image modality signal are IQ color density constellation diagram and AP scatter diagram, and the image size is 255×192×3. In order to ensure the synchronization of different modality data in the network, the sampling rate of the image signal data and the sequence signal data is consistent in the time series.

[0114] Figure 8 The overall architecture diagram of the AMFN network provided in an embodiment of the present invention, Conv: 7×7 represents a convolution kernel of size 7×7, and the others are similar.

[0115] Different colors in LSTM represent different structures, and the shapes can also be distinguished, which will not be described in detail in this invention. The circle in Dense represents the neurons that need to be discarded (Dropout) to prevent overfitting. Similarly, the dotted line connection can also express this meaning, which will not be described in detail in this invention.

[0116] The entire model is divided into two parts according to the modality of the input signal, one is the sequence modality signal part, and the other is the image modality signal part.

[0117] The inputs of the sequence modal signal are IQ sequence signal (IQ sequence) and AP sequence signal (APsequence). For the feature extraction of the sequence modal signal, convolution operations with convolution kernel sizes of 1×2, 3×1, and 3×1 are performed in sequence to extract channel features of different scales and dimensions. After the channel features of the IQ sequence signal and the AP sequence signal are extracted in parallel, they are spliced ​​together vertically to obtain the sequence signal fusion feature S = [S1; S2]. Since the convolution operation and channel attention mechanism extract channel features, the extraction of temporal features is inefficient, especially it is difficult to capture long-term temporal features. Therefore, the output S after convolution is input into the LSTM network to extract the temporal features of the sequence signal, and the sequence modal signal feature S′ is obtained.

[0118] The inputs of the image modality signal are the IQ density constellation and AP scatter plot. For the feature extraction of the image modality signal, firstly, convolution operations with kernel sizes of 7×7, 5×5, and 3×3 are performed in each channel in turn. While reducing the image size dimension, the spatial features of different depths of the image are extracted. After the IQ density constellation and AP scatter plot have extracted the features in parallel, they are spliced ​​together vertically to obtain the image signal fusion feature I=[I1;I2]. In order to fuse the sequence sample features and image sample features at the feature layer. We input the convolution output I into the fully connected layer, and after nonlinear transformation, map it to the S' dimensional space to obtain the image modality signal feature I'. The features S' and I' of the two information sources are combined by the connection function to generate joint features. The output layer uses the cross entropy loss function sofmax to map the output of the FC layer to the probability of belonging to each modulation category, and regards the label corresponding to the maximum probability as the modulation type prediction result of the input signal.

[0119] Step S106, extracting sequence modal signal features according to the IQ sequence signal and the AP sequence signal:

[0120] In a possible implementation, convolution operations with convolution kernel sizes of 1×2, 3×1, and 3×1 are sequentially performed on the IQ sequence signal, and channel features of different scales and dimensions are extracted to obtain a first feature;

[0121] The AP sequence signal is sequentially subjected to convolution operations with kernel sizes of 1×2, 3×1, and 3×1 to extract channel features of different scales and dimensions to obtain the second feature.

[0122] The first feature and the second feature are vertically spliced ​​together to obtain the sequence signal fusion feature:

[0123] The sequence signal fusion features are input into the LSTM network to extract the temporal features of the sequence signal and obtain the sequence modal signal features.

[0124] Step S107, extracting image modality signal features according to the IQ color density constellation diagram and the AP scatter diagram;

[0125] In a possible implementation, convolution operations with kernel sizes of 7×7, 5×5, and 3×3 are sequentially performed on the IQ color density constellation map, and spatial features of different depths of the image are extracted to obtain the third feature;

[0126] The AP scatter plot is sequentially subjected to convolution operations with kernel sizes of 7×7, 5×5, and 3×3, and spatial features of different depths of the image are extracted to obtain the fourth feature.

[0127] The third feature and the fourth feature are vertically spliced ​​together to obtain an image signal fusion feature;

[0128] The image signal fusion features are input into the fully connected layer and mapped to the sequence modal signal feature dimensional space after nonlinear transformation to obtain the image modal signal features.

[0129] Step S108, performing feature fusion according to the sequence modality signal feature and the image modality signal feature to obtain a joint feature;

[0130] In a possible implementation, the sequence modality signal features and the image modality signal features are fused through a connection function at the network feature layer to obtain a joint feature.

[0131] Step S109: Map the joint feature to each modulation category, and output the modulation category corresponding to the maximum probability as the modulation signal classification recognition result.

[0132] In a possible implementation, the joint feature is mapped to each modulation category according to the cross entropy loss function, and the modulation category corresponding to the maximum probability is output as the modulation signal classification recognition result.

[0133] In order to test the gain effect of the present invention, the public data set RML2018.10A was tested and compared with the CLDNN network and the DS-CLDNN network. The input of the CLDNN network is an IQ sequence signal, the input of the DS-CLDNN network is an IQ sequence signal and an AP sequence signal, and the input of the network AMFN proposed in the present invention is an IQ sequence signal, an AP sequence number, an IQ color density constellation diagram, and an AP scatter plot. Under a total of 25 signal-to-noise ratios from -20dB to 30dB, the average recognition accuracy of the CLDNN network is 51.4%, the average recognition accuracy of the DS-CLDNN network is 57.8%, and the average recognition accuracy of the network AMFN proposed in the present invention is 61.4%. The network AMFN proposed in the present invention can achieve a maximum gain effect of 10% compared with the CLDNN network and the DS-CLDNN network.

[0134] The present invention also provides a multimodal data fusion system for modulated signal classification and recognition, which is used to execute any of the above multimodal data fusion methods, including:

[0135] A signal acquisition module, used to acquire the original IQ signal;

[0136] An IQ sequence signal generating module is used to perform maximum value normalization and minimum value normalization processing on the amplitude of the original IQ signal, and generate an IQ sequence signal according to the normalized IQ signal;

[0137] An AP sequence signal generating module, used for calculating the amplitude and phase of the AP sequence signal according to the original IQ, and generating the AP sequence signal according to the amplitude and phase;

[0138] An IQ color density constellation diagram generation module is used to convert the original IQ signal into a set of points on a complex plane to generate an IQ color density constellation diagram;

[0139] An AP scatter plot generation module is used to project the original IQ signal in the amplitude-phase dimension to generate an AP scatter plot;

[0140] A sequence modal signal feature extraction module, used for extracting sequence modal signal features according to the IQ sequence signal and the AP sequence signal;

[0141] An image modality signal feature extraction module is used to extract image modality signal features according to an IQ color density constellation diagram and an AP scatter diagram;

[0142] A feature fusion module is used to fuse features according to sequence modal signal features and image modal signal features to obtain joint features;

[0143] The output module is used to map the joint features to each modulation category, and output the modulation category corresponding to the maximum probability as the modulation signal classification recognition result.

[0144] The purpose of the present invention is to provide a multimodal data fusion method and system for modulation signal classification and recognition, which generates multimodal data through the original IQ signal, then fuses the multimodal data in the feature layer of the neural network, and finally trains the network model under the supervision of the loss function to complete the classification of the modulated signal. Compared with the method of using single modal data as input for modulation recognition, the recognition effect of this method is greatly improved.

[0145] The above are only specific embodiments of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A multimodal data fusion method for modulation signal classification and recognition, characterized in that: include: Step S101, obtaining an original IQ signal; Step S102, performing maximum value normalization and minimum value normalization processing on the amplitude of the original IQ signal, and generating an IQ sequence signal according to the normalized IQ signal; Step S103, calculating the amplitude and phase of the AP sequence signal according to the original IQ signal, and generating the AP sequence signal according to the amplitude and phase; Step S104, converting the original IQ signal into a set of points on a complex plane to generate an IQ color density constellation diagram; Step S105, projecting the original IQ signal in the amplitude-phase dimension to generate an AP scatter plot; Step S106, extracting sequence modal signal features according to the IQ sequence signal and the AP sequence signal; Step S107, extracting image modality signal features according to the IQ color density constellation diagram and the AP scatter diagram; Step S108, performing feature fusion according to the sequence modality signal feature and the image modality signal feature to obtain a joint feature; Step S109: Map the joint feature to each modulation category, and output the modulation category corresponding to the maximum probability as a modulation signal classification and recognition result.

2. The multimodal data fusion method according to claim 1, characterized in that: The step S102 includes: The received original IQ signal (r I , r Q ) The amplitude is normalized to the maximum value and the minimum value, the range is (-1, 1), the formula is as follows: The IQ sequence signal samples of length n are organized in the matrix S IQ In the above example, an IQ sequence signal S is generated. IQ , the formula is as follows: In the formula, r I is the in-phase component, r Q is the orthogonal component, S IQ is the IQ sequence signal, n is the length, r I (n) is the IQ signal after normalization of the minimum amplitude, r Q (n) is the IQ signal after normalization of the maximum amplitude.

3. The multimodal data fusion method according to claim 1, characterized in that: The step S103 includes: The amplitude is normalized by L2 norm, and the normalized amplitude range is (0, 1). The phase is normalized between (-1, 1). The formula is as follows: Organize the AP sequence signal samples of length n in the matrix S AP In the process, AP sequence signal S is generated. AP , the formula is as follows: In the formula, (r I , r Q ) is the original IQ signal, r I is the in-phase component, r Q is the orthogonal component, S AP is the AP sequence signal, n is the length, r A (n) is the amplitude, r Q (n) is the phase.

4. The multimodal data fusion method according to claim 1, characterized in that: The step S104 includes: The original IQ signal (r I , r Q ) is converted to a set of points on the complex plane, the formula is as follows: I IQ =[r I (n),r Q (n)]=[(r I (0),r Q (0)),(r I (1),r Q (1)),…,(r I (n-1),r Q (n-1))] In the formula, r I is the in-phase component, r Q is the orthogonal component, I IQ is the set of points in the IQ color density constellation diagram, and n is the length.

5. The multimodal data fusion method according to claim 1, characterized in that: The step S105 comprises: Project the original IQ signal in the amplitude-phase dimension, the formula is as follows: I AP =[r A (n),r P (n)]=[(r A (0),r P (0)),(r A (1),r P (1)),…,(r A (n-1),r P (n-1))] In the formula, r A (n) is the amplitude, r Q (n) is the phase, I AP is the set of points in the IQ color density constellation diagram, and n is the length.

6. The multimodal data fusion method according to claim 1, characterized in that: The step S106 includes: Performing convolution operations with convolution kernel sizes of 1×2, 3×1, and 3×1 on the IQ sequence signal in sequence, and extracting channel features of different scales and dimensions to obtain a first feature; The AP sequence signal is sequentially subjected to convolution operations with convolution kernel sizes of 1×2, 3×1, and 3×1, and channel features of different scales and dimensions are extracted to obtain a second feature; vertically splicing the first feature and the second feature together to obtain a sequence signal fusion feature; The sequence signal fusion features are input into the LSTM network to extract the time series features of the sequence signal to obtain the sequence modal signal features.

7. The multimodal data fusion method according to claim 1, characterized in that: The step S107 includes: Perform convolution operations with convolution kernel sizes of 7×7, 5×5, and 3×3 on the IQ color density constellation map in sequence, and extract spatial features of different depths of the image to obtain a third feature; The AP scatter plot is sequentially subjected to convolution operations with convolution kernel sizes of 7×7, 5×5, and 3×3, and spatial features of different depths of the image are extracted to obtain a fourth feature; vertically splicing the third feature and the fourth feature together to obtain an image signal fusion feature; The image signal fusion feature is input into the fully connected layer, and after nonlinear transformation, it is mapped to the sequence modal signal feature dimensional space to obtain the image modal signal feature.

8. The multimodal data fusion method according to claim 1, characterized in that: The step S108 comprises: In the network feature layer, the sequence modality signal features and the image modality signal features are fused through a connection function to obtain a joint feature.

9. The multimodal data fusion method according to claim 1, characterized in that: The step S109 includes: The joint feature is mapped to each modulation category according to the cross entropy loss function, and the modulation category corresponding to the maximum probability is output as the modulation signal classification recognition result.

10. A multimodal data fusion system for modulation signal classification and recognition, characterized in that: include: A signal acquisition module, used to acquire the original IQ signal; An IQ sequence signal generating module, used for performing maximum value normalization and minimum value normalization processing on the amplitude of the original IQ signal, and generating an IQ sequence signal according to the normalized IQ signal; An AP sequence signal generating module, configured to calculate the amplitude and phase of the AP sequence signal according to the original IQ, and generate the AP sequence signal according to the amplitude and phase; An IQ color density constellation diagram generating module, used for converting the original IQ signal into a set of points on a complex plane to generate an IQ color density constellation diagram; An AP scatter plot generation module, configured to project the original IQ signal in an amplitude-phase dimension to generate an AP scatter plot; A sequence mode signal feature extraction module, used to extract sequence mode signal features according to the IQ sequence signal and the AP sequence signal; An image modality signal feature extraction module, used to extract image modality signal features according to the IQ color density constellation diagram and the AP scatter diagram; A feature fusion module, used for performing feature fusion according to the sequence modality signal features and the image modality signal features to obtain a joint feature; The output module is used to map the joint feature to each modulation category, and output the modulation category corresponding to the maximum probability as the modulation signal classification and recognition result.

Images