Unmanned aerial vehicle image transmission signal modulation mode identification system and method based on multi-modal fusion

Abstract

The invention discloses an unmanned aerial vehicle image transmission signal modulation mode identification system and method based on multi-modal fusion. The method comprises the following steps: (S100) collecting and preprocessing unmanned aerial vehicle communication signal data; (S200) converting the one-dimensional time-domain signal into a two-dimensional Gramer angle field graph and estimating a two-dimensional time-frequency graph of the image transmission signal; (S300) carrying out feature merging; (S400) generating a training set and a test set; (S500) constructing a VGG-ResNet hybrid architecture deep learning network and carrying out training; and (S600) identifying the image transmission signal modulation mode of the unmanned aerial vehicle to be identified. The method disclosed by the invention is based on a multi-modal fusion technology and shows remarkable advantages, and data features of different modals can be comprehensively utilized by fusing the one-dimensional time domain signal and the two-dimensional time frequency graph information to realize more comprehensive description of data.

Description

Technical Field

[0001] The present invention relates to a method for identifying a modulation mode of an image transmission signal of an unmanned aerial vehicle, and in particular to a system and method for identifying a modulation mode of an image transmission signal of an unmanned aerial vehicle based on multimodal fusion. Background Art

[0002] The rapid development of UAV technology has led to its widespread application in many fields such as aerial photography, environmental monitoring, agricultural plant protection, and traffic monitoring. As the information transmission bridge between the UAV and the ground control station, the choice of signal modulation method of the image transmission system is directly related to the efficiency and security of data transmission. However, traditional UAV image transmission signal modulation method identification methods, such as spectrum analysis, feature extraction, and pattern matching, often appear to be inadequate when faced with complex and changing signal environments and an increasing number of modulation types. These traditional methods not only require manual design of feature extractors, which is a large workload, but also have difficulty dealing with the nonlinearity and time-varying nature of the signal, resulting in limited recognition accuracy and robustness.

[0003] To solve the above problems, deep learning technology, with its powerful automatic feature extraction and learning capabilities, provides a new idea for the recognition of modulation methods of drone image transmission signals. By constructing a deep neural network model, deep learning can automatically extract features from signal data and accurately identify different modulation methods through large-scale data training. This method not only reduces the reliance on manual feature design and reduces the workload, but also significantly improves the accuracy and robustness of recognition. In addition, deep learning can also cope with the nonlinearity and time-varying nature of signals, and supports automatic recognition of multiple modulation types, making it possible to realize intelligent recognition of modulation of drone image transmission signals.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present invention is to provide a system and method for identifying the modulation mode of UAV image transmission signals based on multimodal fusion. The method is based on multimodal fusion technology and shows significant advantages. By fusing one-dimensional time domain signals with two-dimensional time-frequency graph information, it can comprehensively utilize data features of different modes to achieve a more comprehensive description of the data. This fusion strategy helps to extract richer and more distinctive features, thereby significantly improving the accuracy of classification and recognition.

[0006] In order to achieve the above object, the present invention provides a method for identifying the modulation mode of UAV image transmission signals based on multimodal fusion, the method comprising:

[0007] (S100) UAV communication signal data collection and preprocessing

[0008] The analog / digital device inside the receiver performs analog-to-digital conversion on the received analog signal, and converts the received signal into the in-phase component through orthogonal demodulation and digital down-conversion. i and the orthogonal components s q Two digital baseband signals, then s i 、s q The two signals are combined to obtain the combined time domain signal s(t)=s i +j·s q , j represents the imaginary unit;

[0009] (S200) Converting the one-dimensional time domain signal into a two-dimensional Grammar angle field diagram and estimating the two-dimensional time-frequency diagram of the image transmission signal

[0010] The Gram angular field transform combines the data points of the time domain signal s(t) in pairs, calculates the cosine of the angle between them, and expresses the result as the pixel value of the image;

[0011] The estimated two-dimensional time-frequency diagram of the image transmission signal is: the combined time domain signal is analyzed with time as the horizontal axis and frequency as the vertical axis. By using synchronous compression transformation to process the collected data points, tfr can be obtained for each output sst Time spectrum, and then calculate the tfr of each set of data along the time axis sst The maximum value of the time-frequency spectrum is obtained by concatenating the maximum values ​​of each group of data to obtain the tfr of all sample data. sst Maximum time-frequency spectrum diagram; Extraction of UAV image transmission signal based on robust principal component analysis RPCA: According to the difference in the correlation between the remote control signal and the image transmission signal in the time-frequency domain, the remote control signal and the image transmission signal are respectively allocated to the sparse matrix and the low-rank matrix using the low-rank matrix recovery method; RPCA simultaneously recovers the low-rank matrix A and the sparse matrix E by minimizing the L2 norm of the reconstruction error; the minimized reconstruction error is the time-frequency matrix tfr sst The difference between (t, f) and the sum of the low-rank matrix A and the sparse matrix E;

[0012] (S300) Feature Merging

[0013] The Gram angular field map and the time-frequency map are combined as two RGB channels into a multi-channel feature map;

[0014] (S400) Generate training set and test set

[0015] For each modulation signal, a drone signal sample is obtained. Each drone signal sample corresponds to two features: a Ram angle field map and a time-frequency map. Part of the two-dimensional feature map is used for the training set, and the other part is used for the test set.

[0016] (S500) Build a VGG-ResNet hybrid architecture deep learning network and train it

[0017] The Channel Attention module is embedded in the VGG16 deep learning network, and the residual network is introduced. The multi-modal fusion is performed to obtain the VGG-ResNet hybrid architecture deep learning network for training.

[0018] (S600) Identifying the modulation mode of the drone image transmission signal to be identified: inputting a sample image of the drone signal to be identified into a trained VGG-ResNet hybrid architecture deep learning network, and outputting the identification result of the drone image transmission signal.

[0019] Preferably, in step (S200), the one-dimensional time domain signal is converted into a two-dimensional Grammar angle field diagram, comprising:

[0020] (S211) Time Domain Signal Preprocessing

[0021] Normalize the time domain signal s(t), the normalization formula is:

[0022]

[0023] In formula (1), μ is the mean of s(t), σ is the standard deviation of s(t);

[0024] (S212) Calculate the cosine value of the angle

[0025] The normalized time domain signal s norm Each data point in (t) is considered as a point in the vector space, and the cosine of the angle between these points is calculated; the Gram matrix G is constructed, where the elements of the matrix are Indicates time t i and time t k The cosine of the angle between the data points;

[0026] The cosine of the angle is:

[0027]

[0028] In formula (2), || represents the modulus; since s norm (t) is a one-dimensional signal, which can be regarded as a point in a two-dimensional vector space, with time t corresponding to the x-axis and the signal value s norm (t) corresponds to the y-axis; simplifying formula (2) to:

[0029]

[0030] (S213) Generate Gram angle field map

[0031] Map the angle cosine value in the Gram matrix to the pixels of the two-dimensional image; each pixel value I(x,y) of the image corresponds to the angle cosine value between different moments in the time domain signal data

[0032] The following linear transformation is used Mapping to the range [0,255]:

[0033]

[0034] In formula (4), Indicates rounding down.

[0035] Preferably, in step (S200), the synchronous compression transformation expression is:

[0036]

[0037] tf stft (t,f)=FFT(s(t)·h(t)) (6)

[0038] In equations (5) and (6), s(t) is the time domain signal, h(t) is the window function, and here we take It means partial derivative; FFT is Fourier transform, and j is the imaginary unit.

[0039] Preferably, in step (S200), the RPCA problem is transformed into a convex optimization problem, the goal of which is to minimize the sum of the nuclear norm of the low-rank matrix A and the L1 norm of the sparse matrix E, while ensuring that tfr sst (t,f)=A+E; the optimization problem is expressed as:

[0040]

[0041] In formula (7), Indicates finding the minimum value for the low-rank matrix A and the sparse matrix E, and st represents the constraint condition; ∥∥ * represents the nuclear norm of the matrix, ∥∥1 represents the L1 norm of the matrix, and λ is the balance parameter between the low-rank term and the sparse term.

[0042] Preferably, in step (S400), the signal modulation method includes: 2PSK, 8PSK, 4QAM, 16QAM, 64QAM, 256QAM.

[0043] Preferably, in step (S400), the training set and the test set are combined in a ratio of 7:3.

[0044] Preferably, in step (S500), the structure of the VGG-ResNet hybrid architecture deep learning network includes: an input layer, a convolutional layer block one, a convolutional layer block two, a convolutional layer block three, a convolutional layer block four, a convolutional layer block five, a fully connected layer and a Softmax layer connected in sequence; in each convolutional layer block, a residual block, a channel attention module and a pooling layer are sequentially arranged after the convolutional layer; the convolutional layer block one and the convolutional layer block two contain two convolutional layers; the convolutional layer block three to the convolutional layer block five contain three convolutional layers; the input layer can accept feature maps of 2 channels.

[0045] Preferably, the convolutional layer and the fully connected layer are set with an activation function Leaky ReLU; the Leaky ReLU activation function is:

[0046] f(x R )=max(αx R ,x R ) (8)

[0047] In formula (8), α is a hyperparameter with a value between (0, 1); max() means to find the maximum value, x R is the input variable of the activation function; when x R ≥0, f(x R )=x R , that is, the output is the same as the input; when x R <0, F(x R )=αx R , that is, the output is the input multiplied by the hyperparameter α.

[0048] Preferably, in step (S500), training a VGG-ResNet hybrid architecture deep learning network includes: using a random sampling algorithm to disrupt the order of sample images in the training set to obtain a training set after the order is disrupted; setting the number of training cycles Epoch, after all samples are sent to the neural network, after multiple iterations of training to ensure that all data are in convergence at the end; selecting a batch size Batch Size, setting a learning rate Learning Rate, and modifying the loss function to a dynamic weighted loss function MSE total ; Input the shuffled drone sample image training set and validation set into the VGG-ResNet hybrid architecture deep learning network, and continuously iterate and update the parameters until the predetermined number of training rounds is reached or other stopping conditions are met, and a trained VGG-ResNet hybrid architecture deep learning network is obtained.

[0049] Among them, the dynamic weighted loss function MSE total The calculation steps are:

[0050] 1) Initialize weights

[0051] Set initial weights w1 and w2 for the Gram angular field diagram and time-frequency diagram respectively;

[0052] 2) Calculate the basic MSE

[0053] For each feature: Gram angular field map and time-frequency map, calculate its mean square error MSE respectively; suppose that for a certain sample, the model's predicted values ​​for the Gram angular field map and time-frequency map are The true values ​​are y1 and y2 respectively, then the mean square error loss MSE of each feature is ch It is expressed as:

[0054]

[0055] In formula (9), n is the number of elements in the sample, and ch,el are the predicted value and true value of the el-th element of the ch-th feature respectively;

[0056] 3) Dynamically adjust weights

[0057] According to the mean square error loss MSE of each feature ch , dynamically adjust the weights; use MSE ch The reciprocal of is used as the weight factor, and the adjusted weight is:

[0058]

[0059] 4) Calculate weighted loss

[0060] Using dynamically adjusted weights and Calculate the weighted loss function:

[0061]

[0062] 5) Update the model

[0063] Use the calculated weighted loss function to update the model parameters;

[0064] 6) Iterative training

[0065] During the training process, steps 2) to 5) are repeated until a predetermined number of training rounds is reached or other stopping conditions are met.

[0066] Another object of the present invention is to provide a UAV image transmission signal modulation mode recognition system based on multimodal fusion, which adopts the UAV image transmission signal modulation mode recognition method based on multimodal fusion to identify the UAV image transmission signal modulation mode.

[0067] The system and method for identifying modulation mode of UAV image transmission signal based on multimodal fusion of the present invention have the following advantages:

[0068] (1) This paper proposes an innovative method for identifying the modulation mode of UAV image transmission signals. This method is based on multimodal fusion technology and shows significant advantages. By fusing one-dimensional time domain signals with two-dimensional time-frequency graph information, it can comprehensively utilize data features of different modalities to achieve a more comprehensive description of the data. This fusion strategy helps to extract richer and more distinctive features, thereby significantly improving the accuracy of classification and recognition.

[0069] (2) The present invention accurately analyzes complex signals through synchronous compression transformation, and uses the Gram Angular Field method to convert the one-dimensional time domain signal into a two-dimensional Gram Angular Field (GAF) graph, integrating the one-dimensional time domain signal features and the two-dimensional time-frequency graph features to achieve sophisticated multi-feature representation. At the same time, the present invention constructs a VGG-ResNet hybrid architecture, which, while retaining the basic structure of VGG16 (i.e., continuous convolutional layers and pooling layers), introduces residual connections inside the convolutional layer blocks to alleviate the gradient vanishing problem and improve feature learning capabilities. After each convolutional layer block (i.e., before each pooling layer), a Channel Attention module is inserted to optimize the activation of the feature map, so that the network focuses on important information and ignores irrelevant or redundant features. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 The present invention is a flow chart of a method for identifying modulation modes of UAV image transmission signals based on multimodal fusion.

[0071] Figure 2 It is a schematic diagram of a UAV image transmission signal modulation mode identification system based on multimodal fusion of the present invention.

[0072] Figure 3 It is the Gram angle field diagram of the UAV time domain signal of the present invention.

[0073] Figure 4 It is a time-frequency diagram of the synchronous compression transformation of the UAV image transmission signal of the present invention.

[0074] Figure 5 This is a block diagram of the VGG-ResNet hybrid architecture deep learning network structure of the present invention.

[0075] Figure 6 This is the confusion matrix diagram of the VGG-ResNet hybrid architecture deep learning network of the present invention. DETAILED DESCRIPTION

[0076] The technical solutions in the embodiments of the present invention are described clearly and completely below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0077] The present invention proposes an innovative method for identifying the modulation mode of UAV image transmission signals and introduces a multimodal fusion strategy to improve the performance of identifying the modulation mode of UAV image transmission signals. Multimodal fusion refers to the fusion of information from different sensors or different data sources to obtain more comprehensive and accurate information. In the identification of the modulation mode of UAV image transmission signals, signal data from different frequency bands and different modulation modes are fused to extract richer feature information. Through the multimodal fusion strategy, the recognition performance of the deep neural network model is further improved, and more accurate and efficient identification of the modulation mode of UAV image transmission signals is achieved.

[0078] The deep learning-based UAV image transmission signal modulation recognition technology, combined with the multimodal fusion strategy, shows broad application prospects and important research value. With the continuous advancement and improvement of deep learning technology and the continuous development of UAV technology, it is believed that this technology will play an important role in more fields and provide stronger support for the development and application of UAV technology.

[0079] In terms of specific implementation, the VGG-ResNet hybrid architecture deep learning network is adopted. By stacking multiple layers of convolutional layers, features in images or signals can be efficiently extracted, especially the subtle and complex modulation features in drone image transmission signals. Its small convolution kernel design not only reduces the number of parameters, but also enhances the nonlinearity of the network, further enriching the feature representation. In addition, the introduction of an efficient channel attention mechanism dynamically adjusts the parameters of the feature channel so that the network can focus on key features, thereby improving recognition accuracy and efficiency. At the same time, this mechanism also reduces redundant information and reduces computational costs. Among them, the use of the LeakyReLU activation function is also a significant highlight, which effectively avoids the gradient vanishing problem of the ReLU activation function when the input is negative, thereby improving the training effect and generalization ability of the model.

[0080] The following is a detailed description of the system and method for identifying the modulation mode of a drone image transmission signal based on multimodal fusion provided by the present invention through Example 1.

[0081] Example 1

[0082] A method for identifying modulation modes of UAV image transmission signals based on multimodal fusion, such as Figure 2As shown, it is a schematic diagram of the UAV image transmission signal modulation mode identification system based on multimodal fusion of the present invention, wherein the UAV transmits the image transmission signal to the ground remote control station, the remote control station transmits the remote control signal to the UAV, and the signal collection device collects the remote control signal and the image transmission signal.

[0083] like Figure 1 FIG. 1 is a flowchart of a method for identifying modulation modes of UAV image transmission signals based on multimodal fusion according to the present invention. The method comprises:

[0084] (S100) UAV communication signal data collection and preprocessing

[0085] The data acquisition system based on the AD9361 zero-IF receiver architecture is used to collect signals from a certain model of DJI drone. The analog / digital device inside the receiver performs analog-to-digital conversion on the received analog signal, and converts the received signal into the in-phase component through orthogonal demodulation and digital down-conversion. i and the orthogonal components s q (Subscript i: In-phase, represents the in-phase component, which refers to the signal with the same vector direction; subscript q: Quadrature, represents the orthogonal component, that is, the two signal vectors are orthogonal) two digital baseband signals, and then s i 、s q The two signals are combined to obtain the combined time domain signal s(t)=s i +j·s q , j represents the imaginary unit t represents time.

[0086] (S200) Converting the one-dimensional time domain signal into a two-dimensional Grammar angle field diagram and estimating the two-dimensional time-frequency diagram of the image transmission signal

[0087] The Gram angular field transform combines the data points of the time domain signal s(t) in pairs, calculates the cosine of the angle between them, and represents the result as the pixel value of the image, which helps to capture the dynamic and periodic characteristics in the time domain signal.

[0088] (S210) The one-dimensional time domain signal is converted into a two-dimensional Gram angle field diagram, and the specific steps are as follows:

[0089] (S211) Time Domain Signal Preprocessing

[0090] Normalize the time domain signal s(t) to eliminate the influence of different dimensions on the result. The normalization formula is:

[0091]

[0092] In formula (1), μ is the mean of s(t) and σ is the standard deviation of s(t).

[0093] (S212) Calculate the cosine value of the angle

[0094] The normalized time domain signal s norm Each data point in (t) is considered as a point in the vector space, and the cosine of the angle between these points is calculated. Construct the Gram matrix G, where the elements of the matrix are Represents different time t i and time t k The cosine of the angle between the data points.

[0095] The cosine of the angle is:

[0096]

[0097] In formula (2), || represents the modulus. norm (t) is a one-dimensional signal, which can be regarded as a point in a two-dimensional vector space (time t corresponds to the x-axis, i.e., the horizontal axis, and the signal value s norm (t) corresponds to the y-axis, i.e., the vertical axis). Formula (2) can be simplified as:

[0098]

[0099] This simplification is based on s norm (t) The hypothesis has been normalized and its modulus is 1.

[0100] (S213) Generate Gram angle field map

[0101] Map the angle cosine value in the Gram matrix to the pixels of the two-dimensional image. Each pixel value I(x,y) of the image corresponds to the angle cosine value between different moments in the time domain signal data. Since the cosine of the angle is a number between -1 and 1, in practical applications, further transformation or standardization may be required to better represent it in the image. For example, the following linear transformation can be used to convert Mapping to the range [0,255]:

[0102]

[0103] In formula (4), Indicates rounding down.

[0104] like Figure 3 As shown in the figure, in the Gram angle field diagram, the horizontal and vertical coordinates actually represent the pixel positions in the image. These pixel positions indirectly reflect the relationship between data points at different times in the time domain signal through the mapping of the cosine value of the angle. Therefore, although the horizontal and vertical coordinates themselves do not directly correspond to the time point or value of the time domain signal, the image they form can reveal the dynamic and periodic characteristics of the time domain signal.

[0105] (S220) estimating a two-dimensional time-frequency diagram of the image transmission signal, the specific steps are as follows:

[0106] (S221) Perform synchronous compression transformation on the combined time domain signal to obtain tfr sst Time-frequency distribution

[0107] Since the drone signal is a non-stationary signal, the synchronous compression transform is used to analyze the merged time domain signal with time as the horizontal axis and frequency as the vertical axis. By processing the collected 8000×2048 data points, each group of data is 2048 points, and a total of 8000 analyses are performed, each time representing a time point. The 2048 points correspond to the signal frequency within the 40MHz bandwidth, and the frequency of each point is 0.019532625MHz.

[0108] The above has been processed 8000 times by synchronous compression transformation, and each output can get tfr sst Time spectrum; then calculate the tfr of each set of data along the time axis sst The maximum value of the time-frequency spectrum is obtained by concatenating the maximum values ​​of each group of data to obtain the tfr of all sample data. sst Time-frequency spectrum maximum plot.

[0109] The synchronous compression transformation expression is:

[0110]

[0111] tf stft (t, f) = FFT(s(t)·h(t)) (6)

[0112] In equations (5) and (6), s(t) is the time domain signal, h(t) is the window function, and here we take It means partial derivative; FFT is Fourier transform, and j is the imaginary unit.

[0113] The synchronous compression transform has many significant advantages: 1) First, it optimizes the layout of time-frequency coefficients through the synchronous compression operator, so that the signal energy is focused on the instantaneous frequency, which significantly improves the time-frequency resolution, effectively solves the ambiguity problem in traditional methods, and ensures the clarity and accuracy of the analysis results; 2) Secondly, this technology can reconstruct the signal in compressed sampling and greatly reduce the number of sampling points, which is particularly critical for big data signal processing and can significantly reduce storage and processing costs; 3) In addition, the synchronous compression transform is flexible in implementation, and the window function and parameters can be adjusted according to actual needs to adapt to a variety of signal types and analysis scenarios; 4) Finally, this technology also has the potential for further research and application, including optimizing instantaneous frequency estimation, innovating rearrangement strategies and expanding the scope of application, showing broad development prospects.

[0114] (S222) Extraction of UAV Image Transmission Signal Based on Robust Principal Component Analysis (RPCA)

[0115] According to the difference in the correlation between the remote control signal and the image transmission signal in the time-frequency domain, the method of low-rank matrix recovery is used to allocate the parts with strong correlation to the low-rank matrix, while the parts with weak correlation are stored in the sparse matrix. The image transmission signal exists continuously within the bandwidth range, while the remote control signal is a stepped oblique (or broken) line with a frequency changing over time in the time-frequency domain. Therefore, when separating the two, most of the image transmission signal is divided into the low-rank matrix, and the remote control signal is divided into the sparse matrix, thereby achieving the purpose of separating the remote control signal from the image transmission signal.

[0116] RPCA minimizes the reconstruction error (the instantaneous spectrum matrix tfr sst The L2 norm of the difference between (t,f) and the sum of the low-rank matrix A and the sparse matrix E) is used to simultaneously recover the low-rank matrix A and the sparse matrix E. RPCA shows strong robustness when processing data containing noise and outliers.

[0117] The RPCA problem is transformed into a convex optimization problem, the goal is to minimize the sum of the nuclear norm of the low-rank matrix A and the L1 norm of the sparse matrix E, while ensuring that tfr sst (t,f)=A+E. The optimization problem can be expressed as:

[0118]

[0119] In formula (7), Indicates finding the minimum value for the low-rank matrix A and the sparse matrix E, and st represents the constraint condition; ∥∥ * represents the nuclear norm of the matrix, ∥∥1 represents the L1 norm of the matrix, and λ is the balance parameter between the low-rank term and the sparse term, where λ=0.1.

[0120] Solving equation (7) yields the low-rank matrix A, which is the separated UAV image transmission signal information. In practice, the CVX module in the Matlab tool for solving convex optimization problems is used to solve the low-rank matrix A.

[0121] like Figure 4 As shown, it is a time-frequency diagram of the synchronous compression transformation of the UAV image transmission signal, where the modulation methods of the UAV image transmission signal are the time-frequency diagrams corresponding to six types, namely 2PSK, 8PSK, 4QAM, 16QAM, 64QAM, and 256QAM.

[0122] (S300) Feature Merging

[0123] The Gram angular field map and the time-frequency map are combined as two RGB channels into a multi-channel feature map.

[0124] Feature merging is the process of combining multiple 2D feature maps into a multi-channel feature map. In this example, there are two 2D feature maps, namely the Gram angular field map and the time-frequency map. These two feature maps will be merged into a multi-channel feature map as two RGB channels (actually this is an analogy because feature maps are not real color channels).

[0125] The specific steps are as follows:

[0126] (S310) Preparing feature map

[0127] Make sure that each feature map has the same size, for example, 224×224.

[0128] (S320) Merging feature maps

[0129] Create a new empty feature map with the same dimensions as a single feature map but with 2 channels.

[0130] Copy the first feature map (Gram's angular field map) to the first channel of the new feature map.

[0131] Copy the second feature map (frequency map) to the second channel of the new feature map.

[0132] (S330) Processing the merged feature map

[0133] The merged feature map is now a 224×224×2 multi-channel feature map.

[0134] This multi-channel feature map is used as input to the deep learning model.

[0135] (S400) Generate data set (training set, test set)

[0136] Referring to the data format of a certain model of DJI drone, the modulation modes of the image transmission signals generated by simulation are 2PSK, 8PSK, 4QAM, 16QAM, 64QAM, 256QAM, etc. For each modulation signal, the signal-to-noise ratio ranges from -10dB to 15dB, with an interval of 5dB. At each signal-to-noise ratio, 1000 drone signal samples are obtained, and each drone signal sample corresponds to two features (Lambda angle field map and time-frequency map), with 1000×2 two-dimensional feature maps; 700×2 of them are used for training set, and the remaining 300×2 are used for test set.

[0137] Therefore, there are a total of 36000×2 drone 2D feature map samples, of which 25200×2 2D feature map samples are used to constitute the training set, and 10800×2 2D feature map samples are used to constitute the test set, that is, the training set and the test set are combined in a ratio of 7:3.

[0138] (S500) Build a VGG-ResNet hybrid architecture deep learning network and train it

[0139] The process of building a VGG-ResNet hybrid architecture deep learning network and training it is as follows:

[0140] (S510) embedding a ChannelAttention module into the VGG16 deep learning network, introducing a residual network, and fusing multiple modalities to obtain a VGG-ResNet hybrid architecture deep learning network;

[0141] like Figure 5 As shown, the VGG-ResNet hybrid architecture deep learning network and its parameter description:

[0142] The activation function in the VGG16 deep learning network is set to Leaky ReLU, and the residual network (ResNet) is introduced, and the rest remains unchanged. The following is the improved VGG-ResNet hybrid architecture deep learning network structure and its corresponding parameters:

[0143] 1) Input layer

[0144] Input size: 224×224×2 (Gram angle field map, time-frequency map);

[0145] Data type: floating point number;

[0146] Normalization: Normalize the amplitude value of the input feature to the range of [-1,1].

[0147] 2) Convolutional layer block 1

[0148] Convolutional layer 1-1: 64 3×3 convolution kernels, stride 1, padding 'SAME', activation function Leaky ReLU (alpha value can be set as needed, usually 0.01);

[0149] Convolutional layer 1-2: 64 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0150] Residual block 1: 64 channels of input, 64 channels of output, two 3×3 convolutional layers (activation function is Leaky ReLU);

[0151] Channel attention module 1: compression ratio is 16;

[0152] Pooling layer 1: 2×2 max pooling with a stride of 2.

[0153] 3) Convolutional layer block 2

[0154] Convolutional layer 2-1: 128 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0155] Convolutional layer 2-2: 128 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0156] Residual block 2: 128 channels of input, 128 channels of output, two 3×3 convolutional layers (activation function is Leaky ReLU);

[0157] Channel attention module 2: compression ratio is 16;

[0158] Pooling layer 2: 2×2 max pooling with a stride of 2.

[0159] 4) Convolutional layer block three

[0160] Convolutional layer 3-1: 256 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0161] Convolutional layer 3-2: 256 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0162] Convolutional layer 3-3: 256 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0163] Residual block 3: 256 input channels, 256 output channels, three 3×3 convolutional layers (activation function is Leaky ReLU);

[0164] Channel attention module 3: compression ratio is 16;

[0165] Pooling layer 3: 2×2 max pooling with a stride of 2.

[0166] 5) Convolutional layer block 4

[0167] Convolutional layer 4-1: 512 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0168] Convolutional layer 4-2: 512 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0169] Conv4-3: 512 3×3 kernels, stride 1, padding ‘SAME’, activation function Leaky ReLU;

[0170] Residual block 4: 512 input channels, 512 output channels, three 3×3 convolutional layers (activation function is Leaky ReLU);

[0171] Channel attention module 4: compression ratio is 16;

[0172] Pooling layer 4: 2×2 max pooling with a stride of 2.

[0173] 6) Convolutional layer block five

[0174] Convolutional layer 5-1: 512 3×3 kernels, stride 1, padding “SAME”, activation function Leaky ReLU;

[0175] Convolutional layer 5-2: 512 3×3 kernels, stride 1, padding “SAME”, activation function Leaky ReLU;

[0176] Convolutional layer 5-3: 512 3×3 kernels, stride 1, padding “SAME”, activation function Leaky ReLU;

[0177] Residual block 5: 512 input channels, 512 output channels, three 3×3 convolutional layers (activation function is Leaky ReLU);

[0178] Channel attention module 5: compression ratio is 16;

[0179] Pooling layer 5: 2×2 max pooling with a stride of 2.

[0180] 7) Fully connected layer

[0181] Fully connected layer 1: 4096 nodes, activation function Leaky ReLU;

[0182] Fully connected layer 2: 4096 nodes, activation function Leaky ReLU;

[0183] Fully connected layer 3 (output layer): 1000 nodes, the output layer contains a Softmax classifier.

[0184] The above three fully connected (FC) layers: the first two FC have 4096 channels each, the third FC performs 1000-way classification and therefore contains 1000 channels, one channel for each class, and the last layer is a soft-max layer. The configuration of the fully connected layers is similar in all networks, and all hidden layers are equipped with a rectified (Leaky ReLU) nonlinearity.

[0185] The above network structure combines the regularity of VGG16 and the residual connection characteristics of ResNet, aiming to improve the training efficiency and performance of the network.

[0186] Further, the Leaky ReLU activation function is explained. The Leaky ReLU activation function is:

[0187] f(x R )=max(αx R ,x R ) (8)

[0188] In formula (8), α is a hyperparameter with a value between (0, 1); max() means to find the maximum value, x R is the input variable of the activation function. Specifically:

[0189] When x R ≥0, f(x R )=x R , that is, the output is the same as the input.

[0190] When x R <0, f(x R )=αx R , that is, the output is the input multiplied by the hyperparameter α.

[0191] α enables Leaky ReLU to maintain a certain gradient when the input is negative, thus avoiding the gradient vanishing problem of the ReLU activation function when the input is negative. Therefore, Leaky ReLU can improve the training effect and generalization ability of the model to a certain extent.

[0192] Furthermore, the multimodal fusion of the input layer is explained as follows: In the VGG-ResNet hybrid architecture deep learning network structure, in order to adapt to the 224×224×2 input, the input layer size is adjusted so that it can accept feature maps of 2 channels. This design allows the model to process and fuse information from one-dimensional features (time domain signals) and two-dimensional features (time-frequency maps), which is expected to improve the performance of the model on various tasks.

[0193] Furthermore, the Channel Attention module embedded in the VGG16 deep learning network is explained: The Channel Attention module is embedded in the VGG16 deep learning network, and the channel attention module is inserted after each convolution layer block (that is, before each pooling layer). The channel attention module helps the model focus on more useful feature channels, thereby improving the performance of the model.

[0194] Integrating the Channel Attention mechanism into the VGG16 network can bring two advantages: 1) First, it can improve the response of key features: by dynamically adjusting the importance of different channel features, the model can pay more attention to the feature channels that have an important impact on the recognition of modulation modes, thereby improving the accuracy and efficiency of recognition; 2) Second, it helps to reduce redundant information: by suppressing unimportant feature channels, the channel attention mechanism can reduce redundant information in the model processing process, thereby speeding up the reasoning speed and reducing the computational cost.

[0195] Furthermore, the embedded ResNet residual blocks in the VGG16 deep learning network are explained as follows: In each convolutional layer block (convolutional layer block 1 to convolutional layer block 5), there is a ResNet residual block. These residual blocks are located after two or three consecutive convolutional layers and immediately before the channel attention module. Specifically:

[0196] In the convolutional layer block 1, residual block 1 is located after two 3×3 convolutional layers;

[0197] In the convolutional layer block 2, residual block 2 is located after two 3×3 convolutional layers;

[0198] In convolutional layer blocks 3 to 5, each convolutional layer block contains three 3×3 convolutional layers, followed by a residual block (residual block 3 to residual block 5).

[0199] ResNet residual blocks can alleviate the gradient vanishing problem, improve feature learning capabilities and model performance, and accelerate the model training process.

[0200] The rest of the VGG-ResNet hybrid architecture deep learning network remains consistent with the VGG16 deep learning network and will not be described in detail here.

[0201] The process of extracting and classifying features by the VGG-ResNet hybrid architecture deep learning network of the present invention is roughly as follows:

[0202] 1) Initial feature extraction: The input image or signal first passes through the convolutional layer and pooling layer of the VGG network architecture for initial feature extraction. These features will be passed to the subsequent ResNet network architecture for further processing.

[0203] 2) Deep feature extraction: In the ResNet network architecture, input features are extracted through multiple convolutional blocks. Each convolutional block uses residual connections to retain feature information and learn deeper feature representations.

[0204] 3) Feature fusion and classification: Finally, the VGG-ResNet hybrid architecture fuses the extracted deep features and classifies them through the fully connected layer and the Softmax activation function. In the feature extraction of drone image transmission signals, these deep features can be used to identify subtle and complex modulation features.

[0205] In summary, the VGG-ResNet hybrid architecture deep learning network achieves an efficient feature extraction process by combining the advantages of both VGG and ResNet network architectures.

[0206] (S520) Training VGG-ResNet hybrid architecture deep learning network

[0207] Use a random sampling algorithm to disrupt the order of sample images in the training set to obtain a randomized training set.

[0208] Set the number of training cycles to 50. After all samples are sent to the neural network, multiple iterations of training are performed to ensure that all data converge in the end.

[0209] The selection of batch size (the number of samples selected for a single training iteration) plays a key role in influencing the optimization level and speed of the model. A batch size that is a power of two GPUs can achieve better performance. Set the batch size to 16.

[0210] Set the learning rate Learning Rate, which is used to control the amplitude of parameter updates in gradient descent. Set LearningRate to 0.001.

[0211] The loss function is changed to the dynamic weighted loss function MSF total .

[0212] The training set and validation set of the shuffled drone sample images (Gram angular field images, time-frequency images) are input into the VGG-ResNet hybrid architecture deep learning network in batches of 70×2 and 30×2 images respectively, until the predetermined number of training rounds is reached or other stopping conditions are met, and a trained VGG-ResNet hybrid architecture deep learning network is obtained.

[0213] Further, the dynamic weighted loss function MSE is explained totalDesign: A dynamic weighted loss function is used for different inputs (Gram angular field map, time-frequency map) to adjust the importance of different features during training. The following is a method for calculating the dynamic weighted loss function based on feature importance, which allows the weights to be dynamically adjusted according to the performance of the features during training.

[0214] Furthermore, the above dynamic weighted loss function calculation steps are as follows:

[0215] 1) Initialize weights

[0216] Initial weights w1 and w2 are set for the Gram angular field map and the time-frequency map, respectively. For example, the initial setting is w1 = 0.5, w2 = 0.5, indicating that the two have the same importance at the beginning of training.

[0217] 2) Calculate the basic MSE

[0218] For each feature (Gram angular field map, time-frequency map), calculate its mean square error (MSE) respectively. Assume that for a certain sample, the model's predicted values ​​for the Gram angular field map and time-frequency map are The true values ​​are y1 and y2 respectively. Then the MSE of each feature can be expressed as:

[0219] The mean squared error loss MSE for each feature ch as follows:

[0220]

[0221] In formula (9), n is the number of elements in the sample, and ch,el are the predicted value and true value of the el-th element of the ch-th feature respectively.

[0222] 3) Dynamically adjust weights

[0223] According to the mean square error loss MSE of each feature ch , dynamically adjust the weights. Here we use MSE ch The reciprocal of MSE is used as the weight factor. Since the smaller the MSE, the better the model performs on the feature, a higher weight is given to continue optimizing the feature. However, directly using the reciprocal of MSE may result in a weight that is too large or too small, so it can be normalized.

[0224] Adjusted weights:

[0225]

[0226] 4) Calculate weighted loss

[0227] Using dynamically adjusted weights and Calculate the weighted loss function:

[0228]

[0229] 5) Update the model

[0230] Use the calculated weighted loss function to update the model parameters.

[0231] 6) Iterative training

[0232] During the training process, steps 2) to 5) are repeated until a predetermined number of training rounds is reached (for example, the predetermined number of training rounds is set to 50 times) or other stopping conditions are met.

[0233] Note: Forecasted value It is the result of the VGG-ResNet hybrid architecture deep learning network processing the original value of the input feature through a series of convolutional layers, pooling layers, and fully connected layers. The true value y ch,el It is the target value used to train the network, that is, the result that the network is expected to predict, which here represents the actual category of the sample.

[0234] (S600) Identify the modulation mode of the image transmission signal of the drone to be identified

[0235] The sample images (Gram angle field image and time-frequency image) of the drone signal to be identified are input into the trained VGG-ResNet hybrid architecture deep learning network in batches of 30×2 images, and the recognition results of the drone image transmission signal are output.

[0236] The recognition effect of the present invention is further described below in conjunction with the experimental platform:

[0237] The experimental platform uses Windows 11 system, and the environment configuration is Python 3.8.19 and PyTorch 2.3.0. The server configuration is 13th Gen Intel(R) Core(TM) i9-13900HX, 16G memory. The graphics card is NVIDIA GeForce RTX 4060 Laptop GPU. All models are run on GPU, and all parameters except the model are the same.

[0238] The accuracy of the method of the present invention in identifying the modulation mode of the UAV image transmission signal on the Gram angle field diagram and time-frequency diagram test set of the UAV signal is estimated, and the recognition accuracy of the model for various modulation modes of the UAV image transmission signal is obtained by using the confusion matrix. Figure 6As shown in the figure, the vertical axis of the confusion matrix is ​​the true label of the model for the drone remote control signal, the horizontal axis is the predicted label of the model test set, the diagonal line represents the correct prediction accuracy of different labels, and the numbers in the grid indicate the prediction accuracy.

[0239] See also Figure 6 Among them, the recognition effect of the image transmission signal with the modulation mode of 8PSK is relatively poor, and there will be some modulation signal recognition errors in the modulation modes of 4QAM and 16QAM, but the recognition accuracy of all signals is also above 91%.

[0240] Although the content of the present invention has been described in detail through the above preferred embodiments, it should be appreciated that the above description should not be considered as a limitation of the present invention. After reading the above content, it will be apparent to those skilled in the art that various modifications and substitutions of the present invention will occur. Therefore, the protection scope of the present invention should be limited by the appended claims.

Claims

1. A method for identifying modulation modes of UAV image transmission signals based on multimodal fusion, characterized in that: The method includes: (S100) UAV communication signal data collection and preprocessing The analog / digital device inside the receiver performs analog-to-digital conversion on the received analog signal, and converts the received signal into the in-phase component through orthogonal demodulation and digital down-conversion. i and the orthogonal components s q Two digital baseband signals, then s i 、s q The two signals are combined to obtain the combined time domain signal s(t)=s i +j·s q , j represents the imaginary unit; (S200) Converting the one-dimensional time domain signal into a two-dimensional Grammar angle field diagram and estimating the two-dimensional time-frequency diagram of the image transmission signal The Gram angular field transform combines the data points of the time domain signal s(t) in pairs, calculates the cosine of the angle between them, and expresses the result as the pixel value of the image; The estimated two-dimensional time-frequency diagram of the image transmission signal is: the combined time domain signal is analyzed with time as the horizontal axis and frequency as the vertical axis. By using synchronous compression transformation to process the collected data points, tfr can be obtained for each output sst Time spectrum, and then calculate the tfr of each set of data along the time axis sst The maximum value of the time-frequency spectrum is obtained by concatenating the maximum values ​​of each group of data to obtain the tfr of all sample data. sst Maximum time-frequency spectrum diagram; Extraction of UAV image transmission signal based on robust principal component analysis RPCA: According to the difference in the correlation between the remote control signal and the image transmission signal in the time-frequency domain, the remote control signal and the image transmission signal are respectively allocated to the sparse matrix and the low-rank matrix using the low-rank matrix recovery method; RPCA simultaneously recovers the low-rank matrix A and the sparse matrix E by minimizing the L2 norm of the reconstruction error; the minimized reconstruction error is the time-frequency matrix tfr sst The difference between (t, f) and the sum of the low-rank matrix A and the sparse matrix E; (S300) Feature Merging The Gram angular field map and the time-frequency map are combined as two RGB channels into a multi-channel feature map; (S400) Generate training set and test set For each modulation signal, a drone signal sample is obtained. Each drone signal sample corresponds to two features: a Ram angle field map and a time-frequency map. Part of the two-dimensional feature map is used for the training set, and the other part is used for the test set. (S500) Build a VGG-ResNet hybrid architecture deep learning network and train it The Channel Attention module is embedded in the VGG16 deep learning network, and the residual network is introduced. The multi-modal fusion is performed to obtain the VGG-ResNet hybrid architecture deep learning network for training. (S600) Identifying the modulation mode of the drone image transmission signal to be identified: inputting a sample image of the drone signal to be identified into a trained VGG-ResNet hybrid architecture deep learning network, and outputting the identification result of the drone image transmission signal.

2. According to the method for identifying modulation modes of UAV image transmission signals based on multimodal fusion according to claim 1, it is characterized in that: In step (S200), the one-dimensional time domain signal is converted into a two-dimensional Gram angular field diagram, including: (S211) Time Domain Signal Preprocessing Normalize the time domain signal s(t), the normalization formula is: In formula (1), μ is the mean of s(t), σ is the standard deviation of s(t); (S212) Calculate the cosine value of the angle The normalized time domain signal s norm Each data point in (t) is considered as a point in the vector space, and the cosine of the angle between these points is calculated; Construct a Gram matrix G, where the elements of the matrix Indicates time t i and time t k The cosine of the angle between the data points; The cosine of the angle is: In formula (2), || represents the modulus; since s norm (t) is a one-dimensional signal, which can be regarded as a point in a two-dimensional vector space, with time t corresponding to the x-axis and the signal value s norm (t) corresponds to the y-axis; simplifying formula (2) to: (S213) Generate Gram angle field map Map the angle cosine value in the Gram matrix to the pixels of the two-dimensional image; each pixel value I(x, y) of the image corresponds to the angle cosine value between different moments in the time domain signal data The following linear transformation is used Mapping to the range [0, 255]: In formula (4), Indicates rounding down.

3. According to the method for identifying modulation modes of UAV image transmission signals based on multimodal fusion according to claim 1, it is characterized in that: In step (S200), the synchronous compression transformation expression is: tfr stft (t,f)=FFT(s(t)·h(t)) (6) In equations (5) and (6), s(t) is the time domain signal, h(t) is the window function, and here we take It means to find partial derivatives; FFT is Fourier transform, and j is the imaginary unit.

4. According to the method for identifying modulation mode of UAV image transmission signal based on multimodal fusion according to claim 1, it is characterized in that: In step (S200), the RPCA problem is transformed into a convex optimization problem, the goal of which is to minimize the sum of the nuclear norm of the low-rank matrix A and the L1 norm of the sparse matrix E, while ensuring that tfr sst (t, f) = A + E; the optimization problem is expressed as: In formula (7), It means finding the minimum value for the low-rank matrix A and the sparse matrix E, and st represents the constraint condition; |||| * represents the nuclear norm of the matrix, || ||1 represents the L1 norm of the matrix, and λ is the balance parameter between low-rank terms and sparse terms.

5. According to the method for identifying modulation mode of UAV image transmission signal based on multimodal fusion according to claim 1, it is characterized in that: In step (S400), the signal modulation method includes: 2PSK, 8PSK, 4QAM, 16QAM, 64QAM, and 256QAM.

6. The method for identifying modulation modes of UAV image transmission signals based on multimodal fusion according to claim 1 is characterized in that: In step (S400), a training set and a test set are combined in a ratio of 7:

3.

7. The method for identifying modulation modes of UAV image transmission signals based on multimodal fusion according to claim 1 is characterized in that: In step (S500), the structure of the VGG-ResNet hybrid architecture deep learning network includes: an input layer, a convolutional layer block 1, a convolutional layer block 2, a convolutional layer block 3, a convolutional layer block 4, a convolutional layer block 5, a fully connected layer and a Softmax layer connected in sequence. x layer; In each convolutional layer block, the residual block, channel attention module and pooling layer are sequentially arranged after the convolutional layer; The convolutional layer block 1 and the convolutional layer block 2 contain two convolutional layers; The convolutional layer block three to the convolutional layer block five contain three convolutional layers; The input layer can accept feature maps of 2 channels.

8. The method for identifying modulation modes of UAV image transmission signals based on multimodal fusion according to claim 7 is characterized in that: The convolutional layer and the fully connected layer are set with activation function Leaky ReLU; The Leaky ReLU activation function is: f(x R )=max(αx R ,x R ) (8) In formula (8), α is a hyperparameter with a value between (0, 1); max() means finding the maximum value; When x R ≥0, f(x R )=x R , that is, the output is the same as the input; When x R <0, f(x R )=αx R , that is, the output is the input multiplied by the hyperparameter α; x R is the input variable of the activation function.

9. The method for identifying modulation modes of UAV image transmission signals based on multimodal fusion according to claim 1 is characterized in that: In step (S500), training a VGG-ResNet hybrid architecture deep learning network includes: Use a random sampling algorithm to disrupt the order of sample images in the training set to obtain a randomized training set; Set the number of training cycles Epoch, after sending all samples into the neural network, and then train them for multiple iterations to ensure that all data converges in the end; Select the batch size, set the learning rate, and change the loss function to the dynamic weighted loss function MSE. total ; The shuffled drone sample image training set and validation set are input into the VGG-ResNet hybrid architecture deep learning network, and the parameters are continuously updated iteratively until the predetermined number of training rounds is reached or other stopping conditions are met, thereby obtaining a trained VGG-ResNet hybrid architecture deep learning network. Among them, the dynamic weighted loss function MSE total The calculation steps are: 1) Initialize weights Set initial weights w1 and w2 for the Gram angular field diagram and time-frequency diagram respectively; 2) Calculate the basic MSE For each feature: Gram angular field map and time-frequency map, calculate its mean square error MSE respectively; suppose that for a certain sample, the model's predicted values ​​for the Gram angular field map and time-frequency map are The true values ​​are y1 and y2 respectively, then the mean square error loss MSE of each feature is ch It is expressed as: In formula (9), n is the number of elements in the sample, and ch,el are the predicted value and true value of the el-th element of the ch-th feature respectively; 3) Dynamically adjust weights According to the mean square error loss MSE of each feature ch , dynamically adjust the weights; use MSE ch The reciprocal of is used as the weight factor, and the adjusted weight is: 4) Calculate weighted loss Using dynamically adjusted weights and Calculate the weighted loss function: 5) Update the model Use the calculated weighted loss function to update the model parameters; 6) Iterative training During the training process, steps 2) to 5) are repeated until a predetermined number of training rounds is reached or other stopping conditions are met.

10. A UAV image transmission signal modulation mode recognition system based on multimodal fusion, characterized in that: The system uses the method for identifying the modulation mode of the UAV image transmission signal based on multimodal fusion as described in any one of claims 1 to 9 to identify the modulation mode of the UAV image transmission signal.

Images