Cross-modal semantic alignment and feature fusion bionic unmanned aerial vehicle and bird intelligent identification method

Abstract

The invention relates to the technical field of bionic unmanned aerial vehicle recognition, in particular to a cross-modal semantic alignment and feature fusion bionic unmanned aerial vehicle and bird intelligent recognition method, which comprises the following steps: S1, constructing a neural network according to the following strategies: the input of the neural network is a sample pair consisting of a target image and a motion behavior sequence with a certain length; the target is a bionic unmanned aerial vehicle or a bird; an image feature vector of an input image is extracted by a first feature extractor, and a sequence feature vector of a motion behavior sequence is extracted by a second feature extractor; a feature fusion module fuses the image feature vector and the sequence feature vector to obtain an advanced feature; performing classification by a classifier based on the advanced features; s2, a neural network is trained, and a used loss function comprises a semantic alignment target loss function used for approaching the difference between an image feature vector and a sequence feature vector; and S3, using the trained neural network to identify the bionic unmanned aerial vehicle and the flying bird to realize robustness discrimination of the bionic unmanned aerial vehicle and the flying bird.

Description

Technical Field

[0001] The present invention relates to the technical field of bionic drone recognition, and in particular to a bionic drone and flying bird intelligent recognition method with cross-modal semantic alignment and feature fusion. Background Art

[0002] Drone warfare is an emerging combat mode that plays a significant role in intelligence collection, reconnaissance and surveillance, precision strikes, etc., and is reflected in multi-dimensional combat fields such as sea, land and air. With the continuous advancement of computer technology and material technology, as well as the needs of national defense strategic security, bionic drones have emerged in large numbers. Bionic drones can better integrate into the natural environment by simulating the flight characteristics and shape characteristics of birds, significantly reducing the probability of being discovered. The high similarity between bionic drones and flying birds also makes the accurate identification of bionic drones face great challenges.

[0003] The performance of traditional methods mainly depends on the accuracy of the target feature library, and its cognitive process requires a lot of manual participation, which is relatively difficult to build. In addition, the target features extracted by traditional methods are mostly shallow features, which are difficult to fully characterize the target characteristics and have poor generalization.

[0004] In recent years, with the rapid development of artificial intelligence, image classification methods based on deep learning have been widely used in the identification of drones and birds due to their powerful feature representation capabilities and accurate and fast recognition capabilities. The main idea is to automatically learn the features in the image through a convolutional neural network, and then use these learned features to discriminate the target. Unlike the drone identification problem, bionic drones are highly similar to birds in vision, which makes it very difficult to find their separable features only in the image domain, so it is difficult to achieve high-precision and stable identification. Therefore, it is very necessary to use other separable features of bionic drones and birds for auxiliary identification. Summary of the invention

[0005] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a bionic drone and flying bird intelligent identification method with cross-modal semantic alignment and feature fusion, so as to solve the technical problem that bionic drones and flying birds cannot be readily identified.

[0006] To achieve the above object, the present invention provides a bionic drone and bird intelligent recognition method with cross-modal semantic alignment and feature fusion, comprising:

[0007] S1 builds a neural network according to the following strategy:

[0008] The input of the neural network is a sample pair consisting of a target image and a certain length of motion behavior sequence;

[0009] The target is a bionic drone or bird;

[0010] The input image is extracted with an image feature vector by a first feature extractor, and the motion behavior sequence is extracted with a sequence feature vector by a second feature extractor;

[0011] The feature fusion module fuses the image feature vector and the sequence feature vector to obtain a high-level feature;

[0012] Classification is performed by a classifier based on high-level features;

[0013] S2 trains a neural network, wherein the loss function used includes a semantic alignment objective loss function for approximating the gap between the image feature vector and the sequence feature vector;

[0014] S3 uses trained neural networks to identify bionic drones and flying birds.

[0015] The beneficial effects of the present invention are as follows: the present invention utilizes the differences in movement behaviors between bionic drones and flying birds, introduces the movement behavior information of the two as an aid to improve distinguishability, performs preliminary feature extraction on both target image and sequence data, and then mines the consistent semantic information between cross-modal data by introducing a semantic alignment target loss function in training, and enhances the model's representation capability for cross-modal data through a feature fusion module, effectively solving the problems caused by the high similarity of targets in optical natural images, and achieving robust discrimination of bionic drones and flying birds. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 is a schematic diagram of a neural network in an embodiment of the present invention;

[0017] Figure 2 1 is an exemplary target image input into a neural network in an embodiment of the present invention, (a) and (b) are bionic drones, (c) and (d) are flying birds;

[0018] Figure 3 for Figure 2 Schematic diagram of the flight altitude data sequence corresponding to the target image in FIG.

[0019] Figure 4 Schematic diagram of the structure of the feature fusion module in an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The following describes the embodiments of the present invention through specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention.

[0021] A cross-modal semantic alignment and feature fusion method for intelligent recognition of bionic drones and flying birds, including:

[0022] S1 builds a neural network according to the following strategy:

[0023] The input of the neural network is a sample pair consisting of a target image and a certain length of motion behavior sequence;

[0024] The target is a bionic drone or bird;

[0025] The input image is extracted with an image feature vector by a first feature extractor, and the motion behavior sequence is extracted with a sequence feature vector by a second feature extractor;

[0026] The feature fusion module fuses the image feature vector and the sequence feature vector to obtain a high-level feature;

[0027] Classification is performed by a classifier based on high-level features;

[0028] S2 trains a neural network, wherein the loss function used includes a semantic alignment objective loss function for approximating the gap between the image feature vector and the sequence feature vector;

[0029] S3 uses trained neural networks to identify bionic drones and flying birds.

[0030] In this example, the constructed neural network is used to distinguish between bionic drones and flying birds (hereinafter referred to as targets) to achieve the effect of intelligent recognition. Its structure is basically as follows: Figure 1 shown.

[0031] The neural network inputs two modal information of the target to be identified, namely Figure 2 The image information of the bionic drone or bird shown and the Figure 3 The movement behavior sequence information shown, Figure 2 (a) and (b) are the image information of the bionic drone. Figure 2 (c) and (d) are image information of flying birds. In this example, the input image size is 224×224×3, but the present invention is not limited to this.

[0032] Because the flight trajectory of bionic drones is relatively stable, while the flight trajectory of birds is more changeable and highly maneuverable, the difference in their movement methods is used to assist optical natural images and enhance the ability to distinguish targets, so as to solve the problem that targets are difficult to distinguish due to their high similarity. Figure 3 (a) and (b) are the motion sequence information of the bionic drone. Figure 3(c) and (d) in the figure are the motion sequence information of the flying bird image. The unit of the horizontal axis of (a), (b), (c) and (d) is seconds. In this example, the motion behavior sequence is the dimensionless height data sequence of the target after normalization, such as Figure 3 As shown, the altitude data takes the take-off altitude as 0, not the ground altitude. The target fluctuates around the starting point during the flight. It can be intuitively concluded from the figure that the flight altitude change of the bionic drone is significantly different from that of the bird, which is sufficient to distinguish the targets. In addition to the flight altitude data of the target, there can also be other data that can reflect the significant differences between the two, such as the flight speed or acceleration of the target. The altitude data sequence in this example is the altitude data of 100 consecutive time points, and the interval between each two time points is 1s. The sequence can also be of other lengths as long as it can distinguish the different targets.

[0033] The input image information is a three-channel RGB image I(x,y). The first feature extractor extracts the image feature vector. The first feature extractor uses a two-dimensional feature extractor. In this example, the first feature extractor is a Resnet50 network. Figure 1 In the image, the Resnet2d feature extractor is used for annotation, where 2d represents two-dimensional. The expression is:

[0034]

[0035] In the formula, R 2d [.] indicates the operation after the first feature extractor;

[0036] Input movement behavior sequence The second feature extractor extracts the sequence feature vector. The second feature extractor uses a one-dimensional feature extractor. In this example, the second feature extractor is a Resnet18 network. Figure 1 In the example, the Resnet1d feature extractor is used for annotation, where 1d represents one dimension. The expression is:

[0037]

[0038] In the formula, R 1d [.] indicates the operation through a one-dimensional feature extractor.

[0039] The image feature vector and the sequence feature vector obtained by the first feature extractor and the second feature extractor enter the feature fusion module, which fuses the image feature vector and the sequence feature vector. Specifically, Figure 1As shown, the feature fusion module includes a channel attention module (CAModule), which is used to obtain the channel attention weights of the image feature vector and the sequence feature vector, and generate image weighted features and sequence weighted features; and a concatenation module (Concat), which is used to concatenate the image weighted features and the sequence weighted features in the channel dimension, and use the fully connected layer (Linear) for linear transformation to obtain a high-level feature. There are two CAModules, namely the first channel attention module (CAModule1) and the second channel attention module (CAModule2). The image feature vector is input into CAModule1, and the sequence feature vector is input into CAModule2. Since the structures of CAModule1 and CAModule2 are the same, for the convenience of description, the input image feature vector is and the sequence feature vector All are denoted as tensor F. The size of the image feature vector and the sequence feature vector are the same. In this example, the size of the image feature vector and the sequence feature vector are both (1, 512), but the present invention is not limited to this. The feature fusion module not only retains the global information of the input image feature vector and the sequence feature vector, but also effectively captures and enhances the relationship between the features extracted by the information of the two modalities through the channel attention mechanism. The feature fusion module fully integrates cross-modal information, enhances the representation ability of the neural network for cross-modal data, and solves the problem caused by the high similarity of the two types of targets in optical natural images, so as to achieve robust discrimination between bionic drones and flying birds.

[0040] In this example, the feature fusion module uses CAModule to complete the generation of channel attention weights, but it is not limited to this. Other attention mechanisms can also be used to generate attention weights.

[0041] In this example, if Figure 4 As shown, CAModule first performs global average pooling on the input tensor F (image feature vector / sequence feature vector), and the process is expressed as:

[0042] F AAP =AAP(F)

[0043] In the formula, AAP(·) represents the global average pooling operation, F AAP represents the output of F after the global average pooling operation, F AAP Only the channel information is retained.

[0044] After pooling, the number of channels is compressed and batch normalization is performed at the same time. The process is expressed as:

[0045] F BR =BN[RConv1×1 (F AAP )]

[0046] In the formula, F BR Represents the output after the above operations. RConv 1×1 (.) indicates the channel dimension reduction operation using 1×1 convolution, R indicates the channel compression multiple. In this example, R=16 is set, and BN[.] indicates batch normalization operation.

[0047] The channel number compression performed in this process can ensure that the feature fusion module is lightweight enough; the batch normalization operation accelerates the training process and improves the generalization ability of this neural network.

[0048] Then, the nonlinear transformation is introduced through the Swish activation function, and the compressed feature map is restored to the original number of channels using a 1×1 convolution layer. Then, the convolution output is mapped to the range of 0 to 1 through the sigmoid function to obtain the image feature vector / sequence feature vector channel attention weight. The process is expressed as:

[0049] F SI =σ{InConv 1×1 [Swish(F BR )]}

[0050] In the formula, F SI Represents the channel attention weight, including the channel attention weight of the image feature vector and the channel attention weight of the sequence feature vector; Swish(·) represents the Swish activation function, InConv 1×1 [.] indicates the use of 1×1 convolution operation to achieve channel dimension increase, and σ{.} indicates the Sigmoid function.

[0051] Finally, the Re-weight module multiplies the generated channel attention weights by the input feature vectors (image feature vectors and sequence feature vectors) element by element to generate the weighted feature Y Weighted , whose expression is:

[0052] Y Weighted =F⊙F SI

[0053] In the formula, ⊙ represents the multiplication of corresponding elements of the matrix.

[0054] Specifically, the Re-weight module multiplies the channel attention weight of the image feature vector by the image feature vector element by element to generate the image weighted feature Multiply the channel attention weight of the sequence feature vector by the sequence feature vector element by element to generate the sequence weighted feature and Uniformly denoted as Y Weighted .

[0055] Concat weights the image features and sequence weighted features Concatenate in the channel dimension and use Linear to perform linear transformation to obtain a high-level feature Z output , and its process output is expressed as:

[0056]

[0057] Where Linear(·) represents the fully connected layer, and Concat(·) represents the concatenation of high-level features of images and sequences in the channel dimension.

[0058] Finally, the classifier is based on the high-level features Z output to classify.

[0059] After the network neural network is constructed, the neural network is trained, wherein the loss function used includes a semantic alignment target loss function for approaching the gap between the image feature vector and the sequence feature vector;

[0060] like Figure 1 As shown in the figure, the value of the semantic alignment objective loss function is calculated by the semantic alignment module by comparing the aforementioned image feature vectors and the sequence feature vector In this way, the similarity measure between image and sequence modalities in the feature space can be optimized, the consistent semantic information between cross-modal data can be mined, and more semantic clues can be provided for the robust recognition of bionic drones and flying birds.

[0061] In this example, the function K that calculates the similarity Similarity (.)Using the cosine similarity function Cos Similarity (.), the calculation expression is:

[0062]

[0063] The similarity calculation between the image feature vector and the sequence feature vector is not limited to this example.

[0064] By comparing the image feature vector and the sequence feature vector The similarity is used to quantify the distance between the image feature vector and the sequence feature vector. In other words, the similarity is further transformed to calculate the distance between the image feature vector and the sequence feature vector. The smaller the distance, the more similar the two are in the feature space. The specific expression is:

[0065]

[0066] In the formula, K Distance (.) is a function that represents the distance between two eigenvectors, K Distance (.) ranges from 0 to 2;

[0067] In neural network training, semantic alignment needs to be performed by shortening the distance between the image feature vector and the sequence feature vector. Therefore, the objective loss function of this semantic alignment is The mean square error MSE[.] is used to constrain the cosine distance between the image feature vector and the sequence feature vector to 0, and then the semantic alignment target loss is calculated. The expression is:

[0068]

[0069] In order to ensure the consistency of feature optimization between the fused features and the modalities, the loss function used in training the neural network also includes the classification loss function of the sequence features. It is used to optimize the classification performance of sequence features alone. In this example, the classification loss of sequence features uses binary cross entropy loss to optimize the classification performance of sequence features alone. The classification loss of sequence features is calculated as follows:

[0070]

[0071] In the formula, l i,c Indicates the target category label 0 or 1, N indicates the total number of samples, Cl indicates the number of categories, It represents the prediction probability of the sequence feature, that is, the prediction probability of the classification corresponding to the sequence feature vector that is not fused with the image feature. In this example, the prediction probability of the sequence feature is the prediction probability of the output classification of the sequence feature vector that is not fused but passes through CAModule2.

[0072] The overall loss function for training a neural network is:

[0073]

[0074] In the formula, represents the total loss, It represents the classification loss after feature fusion. α, β and γ represent the weight hyperparameters of the loss function of each part, which are used to control the contribution of the loss of each part to the total loss. The sum of α, β and γ is 1.

[0075] Among them, the classification loss after feature fusion uses binary cross entropy loss to optimize the classification after feature fusion and calculate the classification loss after feature fusion The expression is:

[0076]

[0077] In the formula, l i,c Indicates the target category label 0 or 1, Represents the predicted probability of the fused neural network, N represents the total number of samples, and Cl represents the number of categories.

[0078] The above training strategy is used to continuously iterate and optimize the neural network until convergence.

[0079] Finally, the trained neural network is used to identify bionic drones and flying birds.

[0080] This embodiment uses 10,000 samples for training and testing, wherein the ratio of training data to testing data is 7:3.

[0081] During training and testing, α, β, and γ are set to 1, 1, and 0.1.

[0082] Finally, this method is compared with nine existing technologies, as shown in Table 1 below. The first seven rows show the classification and recognition results of seven single-modal RCS feature image classification methods on the RCS feature image dataset. The results show that ConvNeXt-V2_s and Efficientnet-V2_m perform well in all evaluation indicators, and their accuracy, precision, recall and F1_score all exceed the other five methods, showing excellent classification performance. Although ConvNeXt-V2_s and Efficientnet-V2_m are better than the other five methods in classification performance, there is still room for improvement in their overall classification results. This limitation shows that the performance of the model will be limited when relying solely on image single-modal data for classification. However, the present invention can further improve the classification performance by fusing the feature information of image modality and sequence modality. First, when selecting a two-dimensional feature extractor, Efficientnet-V2_m and Resnet101 have more parameters and slower inference speed, so they are not selected. The inference speed of ConvNeXt_s and ConvNeXt-V2_s is not ideal either. Similarly, although Densenet-121 has fewer parameters, it has more convolution processes than Resnet50, resulting in a slower inference speed. As for Resnet34, although it has the fastest inference speed, its performance is the worst. Therefore, we finally chose Resnet50 with moderate parameters and the fastest inference speed as the feature extractor for the image modality. Then, for the motion sequence modality, a one-dimensional feature extractor was selected that uses the Resnet18 structure with smaller parameters for feature extraction. It can effectively capture the key features in the motion behavior sequence and significantly reduce the time cost of training and inference.

[0083] The last three rows of Table 1 show that in the classification tasks of bionic drones and birds, the lightweight DC-DSCNN method has high real-time requirements but low classification accuracy. The GRU+Alexnetatt method is not outstanding in terms of accuracy and real-time performance because the Alexnet network layer is shallow and performs poorly when processing more complex images. In contrast, the recognition results of this method and the other two best performing cross-modal methods currently, this method CSAFF performs best in all evaluation criteria, with a classification accuracy of 95.25%, a precision of 95.30%, a recall of 95.26%, and an F1_score of 95.27%, which greatly exceeds other methods. On the other hand, the confusion matrix shows that the false alarm rate of this method for bionic drones is only 4.49%, and the false alarm rate for birds is only 4.92%, which is significantly lower than other methods. In addition, although the parameter amount proposed by this method is 29.03M and the inference speed is slightly slower, it performs better in balancing classification performance and computing resource requirements, has high classification accuracy, good recognition performance, can more accurately identify bionic drones and birds, significantly reduce false positive and false negative rates, and ensure the reliability and accuracy of the classification results.

[0084] Table 1 Comparison between this method and the prior art

[0085]

[0086] The above embodiments are merely illustrative of the methods and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the art may modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by a person of ordinary skill in the art without departing from the spirit and technical concept disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A bionic drone and bird intelligent recognition method based on cross-modal semantic alignment and feature fusion, characterized in that: include, S1 builds a neural network according to the following strategy: The input of the neural network is a sample pair consisting of a target image and a certain length of motion behavior sequence; The target is a bionic drone or bird; The input image is extracted with an image feature vector by a first feature extractor, and the motion behavior sequence is extracted with a sequence feature vector by a second feature extractor; The feature fusion module fuses the image feature vector and the sequence feature vector to obtain a high-level feature; Classification is performed by a classifier based on high-level features; S2 trains a neural network, wherein the loss function used includes a semantic alignment objective loss function for approximating the gap between the image feature vector and the sequence feature vector; S3 uses trained neural networks to identify bionic drones and flying birds.

2. The method according to claim 1, characterized in that Semantic alignment objective loss function in neural network training The mean square error MSE[.] is used to constrain the distance between the image feature vector and the sequence feature vector to 0. The expression is: In the formula, K Distance (.) is a function that represents the distance between two eigenvectors.

3. The method according to claim 2, characterized in that By comparing the image feature vector and the sequence feature vector The similarity is used to quantify the distance between the image feature vector and the sequence feature vector. The specific expression is: In the formula, K Similarity (.) is the function for calculating similarity.

4. The method according to claim 3, characterized in that The function for calculating similarity uses the cosine similarity function, which is: Where Cos Similarity (.) is the cosine similarity function.

5. The method according to claim 4, characterized in that In training neural networks, the loss functions used also include the classification loss function of sequence features. It is used to optimize the classification performance of sequence features alone. The expression is: In the formula, l i,c Indicates the target category label 0 or 1, N indicates the total number of samples, Cl indicates the number of categories, Represents the predicted probability of the sequence feature, that is, the predicted probability of the classification corresponding to the sequence feature vector that is not fused with the image feature.

6. The method according to claim 5, characterized in that The overall loss function for training a neural network is: In the formula, represents the total loss, represents the classification loss after feature fusion, α, β and γ represent the weight hyperparameters of the loss function of each part respectively; in, The expression is: In the formula, l i,c Indicates the target category label 0 or 1, Represents the predicted probability of the fused neural network, N represents the total number of samples, and Cl represents the number of categories.

7. The method according to any one of claims 1 to 6, characterized in that: The motion behavior sequence is a flight altitude data sequence of the target.

8. The method for intelligent recognition of bionic drones and flying birds by cross-modal semantic alignment and feature fusion according to claim 7 is characterized in that: The first feature extractor is the Resnet50 network, and the second feature extractor is the Resnet18 network.

9. The method according to claim 1, characterized in that: The feature fusion module includes: The channel attention module is used to obtain the channel attention weights of the image feature vector and the sequence feature vector, and generate image weighted features and sequence weighted features respectively; And a splicing module is used to splice the image weighted features and the sequence weighted features in the channel dimension, and use a fully connected layer to perform linear transformation to obtain a high-level feature.

10. The method according to claim 9, characterized in that The channel attention module first performs global average pooling on the image feature vector / sequence feature vector, then compresses the number of channels and performs batch normalization at the same time; then, a nonlinear transformation is introduced through the Swish activation function, and a 1×1 convolution layer is used to restore the compressed feature map to the original number of channels, and then, the convolution output is mapped to the range of 0 to 1 through the sigmoid function to obtain the channel attention weight of the image feature vector / sequence feature vector; finally, the image feature vector and the sequence feature vector are element-wise multiplied with their respective channel attention weights to generate image weighted features and sequence weighted features.

Images