Dynamic Open Space UAV Recognition Method and Device Based on Multi-View Feature Fusion

Abstract

The present invention proposes a dynamic open space UAV recognition method and device based on multi-view feature fusion, which relates to the technical field of UAV recognition, and includes: for the obtained known-class UAV remote control signals, characterizing the same sample from multiple domain perspectives to generate three different views in the time domain, frequency domain, and time-frequency domain; designing a three-branch multi-view deep feature extraction and fusion recognition network to respectively extract the deep feature information of the three different views, and fusing the output features of the three-branch multi-view deep feature extraction and fusion recognition network; using the Siamese network framework to optimize and train the three-branch multi-view deep feature extraction and fusion recognition network by combining the center loss, contrast loss, and classification cross-entropy loss; based on the trained three-branch multi-view deep feature extraction and fusion recognition network, proposing an open-set recognition algorithm based on the boundary model to conduct UAV discrimination. The present invention represents the UAV signal features from multiple domains and realizes multi-level feature characterization.

Description

Technical Field

[0001] The present invention relates to the technical field of UAV identification, and particularly to a method and device for identifying UAVs in a dynamic open space by fusing multi-view features. Background Art

[0002] With the development of UAV technology, its application fields have gradually expanded and are widely used in various fields such as aerial photography, agricultural monitoring, and entertainment activities. While UAVs provide convenience for people, the characteristics of UAVs such as miniaturization, easy operation, low-altitude flight, slow speed, and good concealment have become tools for illegal intrusion, illegal snooping, and even destruction.

[0003] Currently, there are mainly four ways to detect and identify UAVs, namely, detection based on audio sound waves, detection based on computer vision images, detection based on radar, and detection based on radio frequency signals. The detection method based on audio sound waves identifies UAVs according to the unique propeller rotation noise of different UAVs, that is, audio fingerprints. However, when in a complex environment with a high noise level, and small and medium-sized UAVs have a small sound, the detection effect is poor. The detection method based on visual images detects and identifies UAV targets in the images by imaging the monitored area, including visible light images and infrared images, etc. However, as small targets, UAVs are prone to missed detection and false detection. When the weather environment is poor and the visibility at night is low, for example, in foggy weather, high-definition images of UAVs cannot be captured, and the UAVs fly at a low altitude and a slow speed, and the battery-powered drive makes their infrared characteristics not obvious. The active detection method based on radar uses the echo signal generated when the emitted electromagnetic wave encounters a target during propagation to detect the existence of the target. However, the radar detection system is expensive and is only effective for large and fast-moving targets. Since UAVs fly at a low speed and a low altitude and have a small radar cross-section, the radar detection method has a problem of low detection accuracy for UAV targets. The detection method based on radio frequency signals collects the radio frequency signals emitted by UAVs through radio frequency sensors, analyzes the characteristics of the radio frequency signals to identify UAVs, which is a passive detection method, more concealed, not affected by occlusion such as vegetation and buildings, and has the advantages of high sensitivity, real-time monitoring, low cost, small computational complexity, and no echo interference, and is not affected by the material characteristics of the UAV body and weather illumination, and the detection conditions are relatively good.

[0004] In practical applications, lawbreakers often select UAV devices of the same brand and the same model as legal UAVs to conduct illegal activities. These illegal UAVs disguise their own ID numbers, MAC addresses and other identity information, bypass the protocol-based authentication method, and because their appearance is exactly the same as that of legal UAVs, methods based on radar, optoelectronics and infrared, and sound waves cannot successfully detect them.

[0005] In this case, the UAV recognition method based on radio frequency signals has more advantages. As a radiation source device, each UAV, as a unique radiation source individual, has unique hardware characteristics hidden in the transmitted signal. The inherent radio frequency fingerprint of each radiation source device is unique, stable, and non-forgeable, that is, it is very difficult to be tampered with and replicated (even if identity information such as MAC and IP addresses is forged). Therefore, the hardware fingerprint information can be extracted from the transmitted radio frequency signal to accurately perceive and characterize the identity of UAV devices.

[0006] Currently, most of the research on UAV recognition based on radio frequency signals is based on the closed-set assumption, that is, the category of the UAV device to be recognized belongs to the categories in the model training set. However, illegal UAV devices are unknown to the recognition system. In the application of the real electromagnetic environment, it is mostly a dynamic open scenario, and new unknown UAV devices may appear at any time. It is almost impossible to cover all UAV categories in the training library. Therefore, it is necessary to break the limitation of the incorrect recognition of unknown radiation sources caused by most closed-set recognition assumptions, and ensure that while accurately recognizing known UAV categories, the recognition of new unknown UAV categories can be achieved. In addition, most of the existing UAV recognition methods rely on feature extraction from a single feature map, and the feature extraction is not comprehensive enough and may lose key and detailed information. There is an urgent need to study a method suitable for accurate UAV identity perception and accurate recognition of unknown UAVs in dynamic open scenarios.

[0007] In the application with the patent application number 202411415469.1 and the patent name "Open-set Recognition Method and System for UAV Signals Based on Metric Learning", an open-set recognition method for UAVs based on deep neural networks and metric learning is proposed. However, this application only extracts features from a single feature view of the STFT time-frequency diagram, and the time-frequency diagram obtained by STFT does not have good time-domain and frequency-domain resolutions, lacking in extracting details, local features, and feature comprehensiveness. In addition, the improved KNN unknown discrimination method based on distance thresholds is restricted by the feature space distribution. If the extracted features are not accurate enough or the distributions of different categories in the feature space are relatively scattered and not clustered, there is uncertainty in unknown class discrimination and the recognition performance cannot be guaranteed.

[0008] "BISSIAM: Bispectrum Siamese Network Based Contrastive Learning for UAV Anomaly Detection" proposed in IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING converts UAV signals into bispectrum features as inputs, uses a contrastive learning model based on the Siamese network to learn vector encodings, and proposes a similarity-based fingerprint matching mechanism to detect unknown UAVs. However, it only uses a single feature view, only the bispectrum feature map, and cannot refine the extraction of detailed features. The symmetric loss and mutual information loss of the image-enhanced bispectrum pair based on the same sample only assist the neural network in extracting category-related features and cannot constrain the spatial distribution of features of different category samples. Moreover, this fingerprint matching method is essentially still based on distance and threshold, assuming that the distances between points within a category will not be greater than the maximum distance. If the feature space distribution of known category samples is not sufficiently concentrated or is irregularly distributed, misjudgments are likely to occur based on the distance threshold, and unknown categories may be misjudged as known categories. Summary of the Invention

[0009] To solve the above technical problems, the present invention proposes a method and device for identifying UAVs in a dynamically open space with multi-view feature fusion. The specific technical solutions are as follows:

[0010] A method for identifying UAVs in a dynamically open space with multi-view feature fusion includes:

[0011] Step 1: Characterize the same sample from multiple domain perspectives for the obtained known-class UAV remote control signals to generate three different views: time domain, frequency domain, and time-frequency domain;

[0012] Step 2: Design a three-branch multi-view deep feature extraction and fusion recognition network to respectively extract deep feature information of the three different views, and fuse the features output by the three-branch multi-view deep feature extraction and fusion recognition network;

[0013] Step 3: Use the Siamese network framework to optimize and train the three-branch multi-view deep feature extraction and fusion recognition network by combining center loss, contrastive loss, and classification cross-entropy loss;

[0014] Step 4: Based on the trained three-branch multi-view deep feature extraction and fusion recognition network, propose an open-set recognition algorithm based on the boundary model to achieve accurate discrimination of unknown UAVs.

[0015] A device for identifying UAVs in a dynamically open space with multi-view feature fusion includes:

[0016] The multi-view feature representation generation module characterizes the same sample from a multi-domain perspective for the obtained known-class UAV remote control signals, generating three different views: time domain, frequency domain, and time-frequency domain.

[0017] The multi-branch multi-view deep feature extraction and fusion recognition module; includes a three-branch multi-view deep feature extraction and fusion recognition network, which respectively extracts deep feature information of three different views, and fuses the output features of the three-branch multi-view deep feature extraction and fusion recognition network.

[0018] The multi-branch network optimization training module based on the siamese network and joint loss function; uses the siamese network framework to optimize and train the three-branch multi-view deep feature extraction and fusion recognition network by combining center loss, contrast loss, and classification cross-entropy loss.

[0019] The open-set recognition module based on the boundary model; based on the trained three-branch multi-view deep feature extraction and fusion recognition network, proposes an open-set recognition algorithm based on the boundary model to achieve accurate discrimination of unknown UAVs.

[0020] An electronic device includes: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the above-mentioned multi-view feature fusion dynamic open-space UAV recognition method.

[0021] A computer-readable storage medium stores executable instructions thereon, and when the instructions are executed by a processor, the processor implements the above-mentioned multi-view feature fusion dynamic open-space UAV recognition method.

[0022] The present invention has the following beneficial effects:

[0023] (1) A multi-view feature extraction method is proposed, which transforms the original signal to extract features from three different perspectives: time-domain IQ diagram, power spectral density, and Hilbert time-frequency spectrogram, representing UAV signal features from multiple domains and multiple angles in the time domain, frequency domain, and time-frequency domain, realizing comprehensive and multi-level feature characterization. Moreover, the Hilbert time-frequency spectrogram has good time-domain resolution and frequency resolution, and can finely characterize details and local characteristics.

[0024] (2) For the three different view features, deep networks are respectively designed for deep feature extraction, and a three-branch deep feature extraction and fusion recognition network of "time-domain IQ diagram - 2DCNN", "power spectral density - 1DCNN", and "Hilbert time-frequency spectrogram - improved lightweight ResNet" is proposed. There are associations and differences between different view features of the same sample, and the extraction and fusion of the three view features can realize the full utilization of effective information of different views and the complementary advantages of information.

[0025] (3) It is proposed to use the Siamese network structure for network training, and jointly optimize the feature extraction network with center loss, Siamese network contrast loss, and classification cross-entropy loss, so that the feature distribution output by the three-branch feature extraction and fusion network meets the requirements of "minimizing the distance between classes and maximizing the distance within classes", making the feature distributions of each UAV category more concentrated around the category center and the boundaries clearer. At the same time, it is ensured that the features are class-discriminable and the classification is accurate.

[0026] (4) Based on the trained three-branch feature extraction and fusion recognition network, use the deep fusion features to fit the Weibull boundary model, and use the OpenMax algorithm to correct the output scores of each class of the network to obtain the output results of each known class and unknown class, realizing the accurate discrimination of unknown UAVs. At the same time, the recognition ability of known UAVs is ensured, and the recognition accuracy of UAVs in open space is effectively improved. Brief Description of the Drawings

[0027] Figure 1 It is a flowchart of the present invention;

[0028] Figure 2 It is a schematic diagram of a three-branch multi-view deep feature extraction and fusion network;

[0029] Figure 3 It is a schematic diagram of a Siamese network and joint loss;

[0030] Figure 4 It is a multi-view feature representation result diagram;

[0031] Figure 5 It is a multi-view fusion feature distribution diagram of 5 known-class UAVs;

[0032] Figure 6 It is a multi-view fusion feature distribution diagram of 5 known-class UAVs and 3 unknown classes;

[0033] Figure 7 It is a confusion matrix result diagram of the recognition results of known-class and unknown-class UAVs;

[0034] Figure 8 It is a recognition result diagram of known-class and unknown-class UAVs under different open-set degrees. Detailed Embodiment

[0035] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other. To achieve the above objectives, the present invention adopts the following technical solutions.

[0036] The present invention proposes a dynamic open space UAV recognition method based on multi-view feature fusion. The specific process is as Figure 1 shown, including:

[0037] Step 1: For the obtained known-class UAV remote control signals, characterize the same sample from multiple domains to generate three different views in the time domain, frequency domain, and time-frequency domain; characterizing the same sample from multiple domains includes: time-domain IQ diagram, power spectral density, and Hilbert time-frequency spectrum Figure 3 features of different perspectives, and realizing comprehensive and multi-level feature characterization based on multi-view features.

[0038] Time-domain IQ diagram: Directly display the time-domain IQ signal on the amplitude-time plane, and directly use the plane image as a view in the time domain;

[0039] Power spectral density: Perform Fourier transform on the time-domain IQ signal to obtain the frequency-domain representation , and calculate the logarithmic power spectral density through the following formula :

[0040] (1)

[0041] Hilbert time-frequency spectrum diagram: First, use the variational mode decomposition (VMD) algorithm to decompose the time-domain IQ signal into time-modal components. Perform Hilbert transform (Hilbert Transform, HT) on each decomposed time-modal component to obtain the Hilbert time-frequency spectrum diagram, as shown in the following formula:

[0042] (2)

[0043] where and are the instantaneous amplitude and instantaneous frequency of the th time-modal component respectively, is the phase function, is 's Hilbert transform. According to the instantaneous amplitude and instantaneous frequency of each time-modal component, the Hilbert spectrum can be obtained, where is the Hilbert spectrum of each time-modal. The operation represents taking the real number operation, represents the sampling points on the time axis, is the frequency point on the frequency axis.

[0044] Based on its marginal spectrum, obtain the energy concentration region, and intercept the region containing useful information in the Hilbert time-frequency spectrogram to obtain a sub-spectrogram. The Hilbert time-frequency spectrogram has good time-domain resolution and frequency resolution, and can finely characterize details and local characteristics.

[0045] Step 2: Design a three-branch multi-view deep feature extraction and fusion recognition network to extract three different view feature information respectively. Propose a 2D convolutional neural network for the time-domain IQ diagram, a 1D convolutional network for the power spectral density, and an improved lightweight ResNet for the Hilbert time-frequency spectrogram. Fusion the output features of the three-branch multi-view deep feature extraction and fusion recognition network. The three-branch multi-view deep feature extraction and fusion recognition network is as Figure 2 shown. Input the time-domain IQ diagram into the first-branch network, which successively includes a first-level convolutional layer (convolution kernel size k1×k1, feature channel number 16), a pooling layer, a batch normalization layer, a RELU activation layer, a second-level convolutional layer (convolution kernel size k2×k2, feature channel number 32), a pooling layer, a batch normalization layer, a RELU activation layer, and a fully connected layer 1-1. Input the power spectral density feature into the second-branch network, which successively includes a first-level convolutional layer (convolution kernel size 1×k3, feature channel number 16), a pooling layer, a batch normalization layer, a RELU activation layer, a second-level convolutional layer (convolution kernel size 1×k4, feature channel number 32), a pooling layer, a batch normalization layer, a RELU activation layer, and a fully connected layer 2-1. Input the Hilbert time-frequency spectrogram into the third-branch network, which successively includes a first-level convolutional layer (convolution kernel size k5×k5, feature channel number 32), a pooling layer, a batch normalization layer, a RELU activation layer, a residual structure, a pooling layer, a RELU activation layer, and a fully connected layer 3-1. The residual structure includes a second-level convolutional layer (convolution kernel size k6×k6, feature channel number 32), a batch normalization layer, a RELU activation layer, and a third-level convolutional layer (convolution kernel size k7×k7, feature channel number 32). The input of the second-level convolutional layer and the third-level convolutional layer are added as the output of the residual structure. Fusion connect the outputs of the three-branch networks to form multi-view fusion features. Make full use of the effective information of different views, and then perform UAV category recognition through two fully connected classification layers. Among them, the structure of the three-branch multi-view deep feature extraction and fusion network is denoted as and the fully connected classification layer is denoted as . See Figure 3 for the multi-branch network part. It includes a two-dimensional convolutional network, a one-dimensional convolutional network, and an improved ResNet network. The Figure 1 , Figure 2 , and Figure 3 respectively pass through and fuse the outputs of the three networks in to obtain multi-view fusion features. , and then through a fully-connected classification layer including a fully-connected layer 1 and a fully-connected layer 2 , and the predicted label of the UAV category is obtained through the SoftMax function.

[0046] Step 3: Use the siamese network framework to optimize and train the above three-branch multi-view deep feature extraction and fusion recognition network by combining center loss, contrast loss, and classification cross-entropy loss, making full use of the association and difference information between different views of UAV signal samples, so that the three-branch multi-view deep feature extraction and fusion network can accurately represent the signal features of UAV devices, and make the feature distributions of each UAV category more concentrated around the category center and the feature distribution boundaries clearer. The siamese network framework and the combined loss function are shown in Figure 3 , the siamese network framework contains two identical core networks. The multi-view features of sample 1 and sample 2 of the known class sample pair are respectively input into the two core networks, and the predicted label 1, predicted label 2, and the fusion features of the two networks obtained through the two core networks , calculate the center loss, siamese network contrast loss, and classification cross-entropy loss.

[0047] Take the above three-branch deep feature extraction and fusion recognition network and the fully-connected classification layer as two identical core networks of the siamese network. Randomly select two known-class UAV samples as a sample pair (the two samples may belong to the same category, that is, the positive sample pair label is set to 1, or they may not belong to the same category, that is, the negative sample pair label is set to 0), and input the two samples in the sample pair into the two core networks of the siamese network respectively;

[0048] Calculate the center loss using the multi-view fusion features output from the core network. The expression is as follows:

[0049] (3)

[0050] Among them, is the number of samples in the training data set, is the multi-view fusion feature output from the core network corresponding to a certain sample, represents the category label of the nth sample, then is the category feature center.

[0051] Calculate the contrast loss using the fusion features output from the sample pair. The expression is as follows:

[0052] (4)

[0053] Among them, represents the distance between the features of the two samples in the ith sample pair, that is:

[0054] (5)

[0055] is the number of sample pairs in the training dataset, represents the two samples in the i-th sample pair, represents the label indicating whether the two samples belong to the same class. If they belong to the same class, then the value is 1. If they do not belong to the same class, then the value is 0. is the set threshold. If the distance between the features of two samples from different classes exceeds the threshold, then this loss value is very small and can be regarded as 0.

[0056] Use the fully connected classification layer of the core network and the output of the SoftMax function to calculate the cross-entropy loss. Let the true class label of the n-th sample be , and the predicted label of the core network be . The expression of the classification cross-entropy loss function is as follows:

[0057] (6)

[0058] Among them, is the number of known classes, represents the probability that the n-th sample belongs to the c-th class, satisfying and .

[0059] Jointly optimize and train the core feature extraction network in the siamese network using three loss functions. The joint loss function is expressed as shown in Equation (7). By the center loss and the contrastive loss to constrain the feature space distribution and optimize the distance between different samples in the feature space, so that the output feature distribution is as concentrated as possible around the respective class feature centers, satisfying "minimizing the inter-class distance and maximizing the intra-class distance". Through the cross-entropy loss , make the features have class discriminability and ensure accurate classification.

[0060] (7)

[0061] Among them, are all weight factors greater than 0 and less than 1, used to control the proportion of each loss function in the joint loss function.

[0062] Step 4. Based on the trained three-branch feature extraction and fusion recognition network, an open-set recognition algorithm based on the boundary model is proposed to accurately discriminate unknown drones, while ensuring the recognition ability of known drones, effectively improving the recognition accuracy of drones in open spaces. The open-set recognition algorithm includes:

[0063] Step 4.1. Based on the characteristics of known-class drone samples, fit the Weibull boundary distribution and establish a Weibull boundary model;

[0064] Step 4.2. Calculate the classification probability of the test sample, correct the known-class probability based on the Weibull boundary model, and calculate the probability that it belongs to the unknown class;

[0065] Step 4.3. Discriminate unknown drones and recognize known-class drones according to the probability.

[0066] Specifically, Step 4.1 is as follows: Use the multi-view fusion features output by the core network to fit the Weibull distribution and construct a Weibull boundary model.

[0067] Activation vector and mean activation vector: The output of the second fully connected layer in the fully connected classification layer in the core network is called the activation vector (AV). For each known class, calculate the mean of the activation vectors of all correctly classified samples in the training set to obtain the mean activation vector (MAV) of each class, which represents the center of the sample feature space distribution of this class and is expressed as follows:

[0068] (8)

[0069] where, is the mean activation vector of class c, is the number of correctly classified samples in class c, is the activation vector of the j-th sample in class c;

[0070] Distance set and Weibull distribution: Calculate the Euclidean distance between the activation vector of each correctly classified sample in each known class and the mean activation vector MAV of this class, that is, the distance between the activation vector of the j-th sample and the mean activation vector MAV of class c, to form the distance set of this class. Use the Weibull distribution in extreme value theory to fit the distance set of each known class and construct a Weibull boundary model. The Weibull distribution is a probability distribution used to describe extreme value events and can well characterize the extreme values in the distance set. Its probability density function and cumulative distribution function (CDF) are expressed as:

[0071] (9)

[0072] (10)

[0073] Among them, is the scale parameter, is the shape parameter, represents a certain input distance value.

[0074] Step 4.2 is specifically as follows: Calculate the classification probability of the test sample based on the OpenMax algorithm, correct the probability of the known class based on the Weibull boundary model, and calculate its probability of belonging to the unknown class.

[0075] The OpenMax algorithm is specifically as follows: First, through the trained three-branch feature extraction and fusion recognition network, output the corresponding fully connected layer output score vector , calculate to the center of each known class to obtain . Based on the Weibull distribution of each known class, calculate the corrected score, and the formula is as follows:

[0076] (11)

[0077] Among them, is the distance of the test sample to the mean activation vector MAV of class c, and are respectively the scale parameter and the shape parameter in the Weibull distribution parameters of class c.

[0078] The score of the test sample belonging to the unknown class is the sum of the difference between the original probability score and the corrected score:

[0079] (12)

[0080] Normalize the scores of the test sample predicted as each class and the unknown class, and map them to classification probabilities through SoftMax, that is:

[0081] (13)

[0082] Step 4.3 is specifically as follows: If the maximum value in the above classification probability is greater than the threshold, then the test sample is judged as this class; if the maximum probability value is less than the threshold, then it is judged as an unknown class. Specific Example 1:

[0084] Step 1: Select the remote control (RC) signals of 17 different drones from the public dataset as the recognition objects. Select 5 categories as known categories, with 900 samples for each category. Obtain their multi-view feature representations, namely time-domain IQ diagrams, power spectral densities, and Hilbert time-frequency spectra respectively. Figure 3 Schematic diagrams of different perspective features are as Figure 4 , Figure 4 In (a) of Figure 4 is the time-domain IQ diagram, Figure 4 in (b) of

[0085] is the power spectral density, and Figure 2 in (c) of

[0086] is the Hilbert time-frequency spectrum diagram. Figure 3 Step 2: Design a three-branch multi-view deep feature extraction and fusion recognition network to extract three different view feature information respectively. Propose a 2D convolutional neural network for the time-domain IQ diagram, a 1D convolutional network for the power spectral density feature, and an improved lightweight ResNet for the Hilbert time-frequency spectrum diagram. Fuse the output features of the three-branch network. The structure of the three-branch multi-view deep feature extraction and fusion network is as Figure 5 shown, and then input the multi-view fusion features into two fully connected layers for classification.

[0087] Step 3: Use the siamese network framework to optimize and train the above three-branch multi-view deep feature extraction and fusion recognition network by combining center loss, contrastive loss, and classification cross-entropy loss, as Figure 6 shown. Make full use of the correlation and difference information between different views of the drone signal samples, so that the three-branch multi-view deep feature extraction and fusion network can accurately represent the signal features of the drone device, and make the feature distributions of each drone category more concentrated around the category center, and the feature distribution boundaries are clearer. The feature distributions of 5 known drone categories are as Figure 7 shown, and it can be seen that the features of different categories are respectively concentrated around their own centers. Figure 8As shown, it can be seen that the method of the present invention can maintain a high recognition accuracy under different open-set degrees. As the open-set degree increases, that is, the number of unknown classes increases, the recognition rate of the method of the present invention decreases slightly. When the open-set degree is 0, it means that all test samples belong to known classes and do not contain unknown classes. The open-set degree is calculated as follows:

[0088] (14)

[0089] wherein, represents the number of known classes used for training, represents the number of UAV classes (known + unknown) for testing, is the number of UAV classes to be recognized.

[0090] In the present invention, in addition to the time-domain IQ diagram, power spectral density, and Hilbert time-frequency spectrogram in the multi-view, other feature views can also be used, such as differential constellation diagram (DCTF), texture feature vector based on histogram of oriented gradients (HOG), multi-dimensional entropy feature sequence, etc., which all have a positive effect on accurately characterizing UAV features.

[0091] For the deep feature extraction network for different view features, certain adjustments and changes to the specific structure of each branch network still have the ability to represent features.

[0092] The contrast loss of the dual network structure of the Siamese network is proposed to "minimize the intra-class distance and maximize the inter-class distance", and the triplet loss obtained by the triple-core network structure (the above multi-view multi-branch deep feature extraction and fusion recognition network as three identical core networks of the Siamese network) can also realize the constraint effect on the feature space distribution.

[0093] In addition to constructing the boundary model with the Weibull distribution, the Gumbel distribution, Fréchet distribution, etc. in extreme value theory can also be used.

[0094] The present invention also provides a dynamic open-space UAV recognition device for multi-view feature fusion, including:

[0095] A multi-view feature representation generation module, which represents the same sample from multiple domain perspectives for the obtained remote control signals of known-class UAVs, and generates three different views: time domain, frequency domain, and time-frequency domain;

[0096] A multi-branch multi-view deep feature extraction and fusion recognition module; including a three-branch multi-view deep feature extraction and fusion recognition network, which respectively extracts deep feature information of three different views and fuses the output features of the three-branch multi-view deep feature extraction and fusion recognition network;

[0097] Multi-branch network optimization training module based on Siamese network and joint loss function; using the Siamese network framework, jointly optimizing and training the three-branch multi-view deep feature extraction and fusion recognition network with center loss, contrastive loss, and categorical cross-entropy loss;

[0098] Open-set recognition module based on boundary model; based on the trained three-branch multi-view deep feature extraction and fusion recognition network, proposing an open-set recognition algorithm based on the boundary model to achieve accurate discrimination of unknown drones.

[0099] The present invention also provides an electronic device, including: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a method for identifying drones in a dynamic open space with multi-view feature fusion according to the present invention.

[0100] The present invention also provides a computer-readable storage medium, on which executable instructions are stored, and when the instructions are executed by a processor, the processor implements a method for identifying drones in a dynamic open space with multi-view feature fusion according to the present invention.

Claims

1. A method for identifying drones in a dynamic open space with multi-view feature fusion, characterized in that, Including: Step 1: Characterize the same sample from multiple-domain perspectives for the obtained known-class UAV remote control signals, generating three different views: time domain, frequency domain, and time-frequency domain; Step 2: Design a three-branch multi-view deep feature extraction and fusion recognition network to extract deep feature information of the three different views respectively, and fuse the output features of the three-branch multi-view deep feature extraction and fusion recognition network; Step 3: Use the Siamese network framework to optimize and train the three-branch multi-view deep feature extraction and fusion recognition network by combining center loss, contrast loss, and classification cross-entropy loss; Step 4: Based on the trained three-branch multi-view deep feature extraction and fusion recognition network, propose an open-set recognition algorithm based on the boundary model to achieve accurate discrimination of unknown UAVs; In step 3, the Siamese network framework uses the three-branch multi-view deep feature extraction and fusion recognition network as the core network, including the three-branch multi-view deep feature extraction and fusion network structure and the fully connected classification layer. The three-branch multi-view deep feature extraction and fusion network structure is denoted as , and the fully connected classification layer is denoted as . The fully connected classification layer includes two fully connected layers; Take the three-branch multi-view deep feature extraction and fusion recognition network as two identical core networks of the Siamese network, randomly select two known-class UAV samples as a sample pair, and input the two samples in the sample pair into the two core networks of the Siamese network respectively; Calculate the center loss using the multi-view fusion features output from the core network, and the expression is as follows: （3） Among them, is the number of samples in the training dataset, is the multi-view fusion feature output in the core network corresponding to a certain sample, represents the class label of the nth sample, then it is the feature center of the class; Calculate the contrast loss using the fusion features output from the sample pair, and the expression is as follows: （4） Among them, represents the distance between the two sample features in the i-th sample pair, that is: （5） is the number of sample pairs in the training data set, represents the two samples in the i-th sample pair, represents the label indicating whether the two samples belong to the same class. If they belong to the same class, then the value is 1. If they do not belong to the same class, then the value is 0, is the set threshold; Using a fully-connected classification layer and the output of the SoftMax function to calculate the cross-entropy loss. Let the true class label of the -th sample be , and the predicted label by the core network be . The expression of the categorical cross-entropy loss function is as follows: （6） Among them, is the number of known categories, represents the probability that the nth sample belongs to the cth category, and satisfies and ; Optimize and train the core network in the Siamese network by combining three loss functions. The combined loss function is shown as in Equation (7); （7） wherein, are all weighting factors; The open-set recognition algorithm includes: Step 4.1: Based on the known-class UAV sample features, fit the Weibull boundary distribution to establish a Weibull boundary model; Step 4.2: Calculate the classification probability of the test sample, correct the known-class probability based on the Weibull boundary model, and calculate its probability of belonging to the unknown class; Step 4.3: Discriminate unknown UAVs and recognize known-class UAVs according to the probability.

2. The dynamic open space UAV recognition method with multi-view feature fusion according to claim 1, characterized in that, The three different views of time domain, frequency domain, and time-frequency domain include: Time-domain IQ diagram: The time-domain IQ signal is directly displayed on the amplitude-time plane, and the planar image is directly used as a view in the time domain; Power Spectral Density: The time-domain IQ signal is Fourier-transformed to obtain a frequency-domain representation , and the logarithmic power spectral density is calculated by the following formula : （1） Hilbert time-frequency spectrogram: First, use the variational mode decomposition algorithm to decompose the time-domain IQ signal into time modal components. Perform Hilbert transform on each decomposed time modal component to obtain the Hilbert time-frequency spectrogram, as shown in the following formula: （2） Among them, and are respectively the instantaneous amplitude and instantaneous frequency of the th time-modal component, is the phase function, is the Hilbert transform of . According to the instantaneous amplitude and instantaneous frequency of each time-modal component, the Hilbert spectrum is obtained, where is the real part operation, represents the sampling points on the time axis, is the frequency points on the frequency axis.

3. The dynamic open space UAV recognition method with multi-view feature fusion according to claim 2, characterized in that, A 2D convolutional neural network is proposed for the time-domain IQ diagram, a 1D convolutional network is proposed for the power spectral density, and an improved lightweight ResNet is proposed for the Hilbert time-frequency spectrogram. The three-branch multi-view deep feature extraction and fusion network specifically includes inputting the time-domain IQ diagram into the 2D convolutional neural network, which is the first-branch network. This branch network sequentially includes a first-level convolutional layer with a convolutional kernel size of k1×k1 and 16 feature channels, a pooling layer, a batch normalization layer, a RELU activation layer, a second-level convolutional layer with a convolutional kernel size of k2×k2 and 32 feature channels, a pooling layer, a batch normalization layer, a RELU activation layer, and a fully connected layer; inputting the power spectral density feature into the 1D convolutional network, which is the second-branch network. This branch network sequentially includes a first-level convolutional layer with a convolutional kernel size of 1×k3 and 16 feature channels, a pooling layer, a batch normalization layer, a RELU activation layer, a second-level convolutional layer with a convolutional kernel size of 1×k4 and 32 feature channels, a pooling layer, a batch normalization layer, a RELU activation layer, and a fully connected layer; inputting the Hilbert time-frequency spectrogram into the lightweight ResNet, which is the third-branch network. This branch network sequentially includes a first-level convolutional layer with a convolutional kernel size of k5×k5 and 32 feature channels, a pooling layer, a batch normalization layer, a RELU activation layer, a residual structure, a pooling layer, a RELU activation layer, and a fully connected layer. The residual structure includes a second-level convolutional layer, a batch normalization layer, a RELU activation layer, and a third-level convolutional layer. The convolutional kernel size of the second-level convolutional layer is k6×k6 and the number of feature channels is 32. The convolutional kernel size of the third-level convolutional layer is k7×k7 and the number of feature channels is 32. The outputs of the second-level convolutional layer and the third-level convolutional layer are added as the output of the residual structure; the outputs of the three branch networks are fused and connected to form multi-view fusion features.

4. A dynamic open space UAV recognition method with multi-view feature fusion according to claim 1, characterized in that Step 4.1 includes: The output of the second fully connected layer in the fully connected classification layer is called the activation vector AV. For each known category, the mean of the activation vectors of all correctly classified samples in the training set is calculated to obtain the mean activation vector MAV for each category, which represents the center of the feature space distribution of the samples in that category and is expressed as follows: （8） Among them, is the mean activation vector of class c, is the number of correctly classified samples in class c, is the activation vector of the j-th sample in class c; Calculate the Euclidean distance between the activation vectors of all correctly classified samples in each known category and the mean activation vector MAV of that category , that is, the distance between the activation vector of the j-th sample and the mean activation vector MAV of category c, to form the distance set of this category. Use the Weibull distribution to fit the distance set of each known category and construct a Weibull boundary model. Its probability density function and cumulative distribution function are expressed as: （9） （10） Among them, is the scale parameter, is the shape parameter, represents a certain input distance value.

5. The method for identifying a dynamic open space drone with multi-view feature fusion according to claim 4, wherein In step 4.2, the classification probability of the test sample is calculated using the OpenMax algorithm. The OpenMax algorithm is specifically: First, through the trained three-branch multi-view deep feature extraction and fusion recognition network, the test samples are output The corresponding fully connected layer outputs a score vector , calculate to the center of each known class , and obtain . Based on the Weibull distribution of each known class, calculate the corrected score, and the formula is as follows: （11） Among them, is the mean activation vector (MAV) distance from the test sample to class c, and are the scale parameter and the shape parameter in the Weibull distribution parameters of class c, respectively; The scores of the test samples belonging to unknown categories are the original probability scores Sum the differences from the corrected scores: （12） Normalize the scores of predicting the test sample as each category and the unknown category, and map them to classification probabilities through SoftMax, that is: （13）。 6. The dynamic open space UAV recognition method with multi-view feature fusion according to claim 5, characterized in that, Step 4.3 includes: Classification probability If the maximum value among them is greater than the threshold, the test sample is classified as the category c. If the maximum probability value is less than the threshold, it is classified as an unknown category.

7. A method for identifying a dynamic open space drone with multi-view feature fusion according to claim 1, characterized in that, In step 1, the view can also include a differential constellation diagram, a texture feature vector based on the histogram of oriented gradients, and a multi-dimensional entropy feature sequence.

8. A dynamic open space UAV recognition method based on multi-view feature fusion according to claim 1, characterized in that The siamese network can also be a triple-core network structure.

9. A method for identifying a dynamic open space unmanned aerial vehicle with multi-view feature fusion according to claim 6, characterized in that, The Weibull distribution can also be the Gumbel distribution and the Fréchet distribution.

10. A multi-view feature fusion dynamic open space UAV recognition device corresponding to the method according to any one of claims 1-9, characterized in that, Includes: A multi-view feature representation generation module, which, for the obtained known-class UAV remote control signals, characterizes the same sample from multiple domain perspectives to generate three different views: time domain, frequency domain, and time-frequency domain; A multi-branch multi-view deep feature extraction and fusion recognition module; includes a three-branch multi-view deep feature extraction and fusion recognition network, which extracts deep feature information of three different views respectively, and fuses the output features of the three-branch multi-view deep feature extraction and fusion recognition network; Multi-branch network optimization training module based on Siamese network and joint loss function; using the Siamese network framework, jointly optimizing and training the three-branch multi-view deep feature extraction and fusion recognition network with center loss, contrastive loss, and classification cross-entropy loss; Open-set recognition module based on boundary model; based on the trained three-branch multi-view deep feature extraction and fusion recognition network, proposing an open-set recognition algorithm based on the boundary model to achieve accurate discrimination of unknown drones.

11. An electronic device, characterized in that, Comprising: One or more processors; A memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement a method for identifying drones in a dynamic open space with multi-view feature fusion according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, An executable instruction is stored thereon, and when the instruction is executed by a processor, the processor implements a method for identifying drones in a dynamic open space with multi-view feature fusion according to any one of claims 1 to 9.

Images