A target detection method based on language guidance in an unmanned airport scene

Abstract

The application discloses a target detection method based on language guidance in an unmanned airport scene, which comprises the following steps: S1, fine-tuning language prompt feature vectors; S2, training a target detection network: S21, extracting basic feature maps and inputting FPN to obtain multi-stage context feature maps; S22, obtaining multi-stage one-dimensional feature vectors through a feature compression network; S23, performing alignment-removal operation, and removing language description features about shooting conditions in the one-dimensional feature vectors by designing a loss function constraint; S24, inputting the multi-stage context feature maps into RPN and RCNN to complete target classification and positioning tasks, and obtaining a loss function of the target detection network; S25, summing the loss functions of S23 and S24 as a total loss, and performing a back propagation algorithm; and S26, repeating S21-S25 until a specified number of iterations is reached. The target detection network after language guidance can weaken or eliminate the interference caused by different shooting conditions in the feature extraction process, and further improve the performance of the target detector.

Description

Technical Field

[0001] The present invention belongs to the field of computer technology, and in particular relates to a target detection method based on language guidance in an unmanned aerial vehicle (UAV) scenario. Background Art

[0002] In recent years, multimodal machine learning (MMML), also known as MultiModal Machine Learning (MMML), has become a hot topic. It aims to achieve the ability to process and understand multimodal information through machine learning. Considering the fundamental challenges of future human-computer interaction, combining traditional computer vision with natural language learning has gradually attracted the attention of researchers. Meanwhile, the CLIP model has achieved great success in image classification, and a growing number of language-vision-based algorithmic models have emerged. As one of the fundamental tasks in computer vision, object detection is an inevitable trend to use language as a guide. Furthermore, the generalization and abstract features of language can better guide object detection models to extract invariant features. Leveraging language context to better adjust learning metrics allows object detection models to extract higher-level semantic relationships and significantly improve generalization performance.

[0003] In recent years, the widespread use of drones has gradually led to their entry into the civilian market, where they are now used in a variety of complex scenarios. Cameras are the most commonly used sensors on drones, and drone-based object detection algorithms are also gaining attention. However, the complex and ever-changing scenes created by drones present numerous challenges for drone object detection. Most existing solutions simply filter targets using multiple frames or modify deep learning network structures to improve detection performance. However, neither of these approaches addresses the fundamental problem of drone detection in complex environments.

[0004] Patent application number 201811424123.2 discloses a method for detecting drone targets in low-altitude, complex backgrounds, including the following steps: S1. Image filtering; S2. Suspected target extraction: grayscale thresholds, edge tracking thresholds, and size thresholds are set on the filtered image data, and suspected targets that meet the grayscale and size thresholds are extracted using edge tracking. S3. Background interference removal: The number of bright pixels in the neighborhood of the suspected target in step S2 is counted, and suspected targets with bright pixel counts exceeding the threshold are identified as ground objects or cloud background interference and removed. The remaining targets are suspected small targets in the sky. S4. Target track establishment based on temporal information correlation: The trajectory of the suspected small target is determined to further confirm the drone target. This solution targets infrared targets at low altitudes and does not integrate the richer visible light information at low altitudes. Furthermore, it does not effectively model target features when detecting targets, but only determines targets within two consecutive frames of video, which may lead to target misjudgment. Furthermore, the use of template filtering and other methods in the solution can overfit some scenes, resulting in poor generalization. In summary, the applicable scenarios of this solution are very limited, and it is impossible to utilize all the information of the airborne equipment to improve the detection performance.

[0005] Patent application number 202010659641.3 discloses a method and device for detecting variable-size targets in drones based on a multi-head self-attention mechanism. The method includes the following steps: S1, establishing a dataset; S2, establishing a network structure: Based on the dataset obtained in S1, a multi-head self-attention mechanism is used to establish a multi-head self-attention target detection head network. The backend of the multi-head self-attention target detection head network adopts the Faster RCNN framework, and the self-attention mechanism is reintroduced into the regression layer of the multi-head self-attention target detection head network; S3, performing network training in steps to obtain a trained multi-head self-attention target detection head network; S4, detecting the target object: The drone's onboard camera captures an image of the target object, pre-processes the image, and then inputs it into the multi-head self-attention target detection head network obtained in S3. The multi-head self-attention target detection head network outputs the detection result. This solution only adds a self-attention layer to the regression layer of the Faster-RCNN network, fails to fully integrate the image information collected in the drone scene, and does not fully utilize the drone's onboard sensors. This may lead to issues with model generalization. Summary of the Invention

[0006] The purpose of the present invention is to overcome the technical problem in existing drone target detection that the detection efficiency and detection accuracy of drone target detection technology fail to meet actual requirements due to environmental interference, etc., and to provide a target detection method based on language guidance in drone scenarios, which can reduce or eliminate the interference caused by different shooting conditions during the feature extraction process, thereby improving the performance of the target detector.

[0007] The object of the application is achieved by the technical solution of a target detection method based on language guidance in an unmanned aerial scene, comprising the following steps:

[0008] S1, fine-tuning the language prompt feature vector to strengthen its connection with the unmanned aerial scene; the specific method is:

[0009] S11, fill all shooting conditions in a fixed language template, convert different shooting conditions into language description prompts, and process the language description prompts through word segmentation and word vector encoding;

[0010] S12, fine-tune the template using a pre-trained language-visual model;

[0011] S13, embed the multiple sets of fine-tuned language description vectors about shooting conditions into the language encoder of CLIP to obtain a set of language vectors describing the shooting conditions;

[0012] S2, train the target detection network using language description guidance; the specific method is:

[0013] S21, the image data set obtained by the onboard camera of the unmanned aerial vehicle is extracted through the ResNet series backbone network to obtain the basic feature map, and then the basic feature map is input into the FPN feature pyramid network to obtain the multi-stage context feature map;

[0014] S22, the obtained multi-stage context feature map is input into the feature compression network with the same structure and different parameters to obtain the multi-stage one-dimensional feature vector;

[0015] S23, use the pre-trained model CLIP for alignment-removal operation, align the multi-stage one-dimensional feature vector obtained in S22 with the vector encoded by the CLIP image encoder, thereby ensuring that the multi-stage one-dimensional feature vector is mapped to the feature space of the language description; then, design a loss function constraint to remove the language description features related to the shooting conditions in the one-dimensional feature vector;

[0016] S24, input the multi-stage context feature map into the region proposal network RPN and the region convolutional neural network RCNN in turn to complete the target classification and positioning task, and obtain the loss function of the target detection network;

[0017] S25, sum the loss functions of S23 and S24 as the total loss function, and perform the back propagation algorithm;

[0018] S26, loop the process of S21-S25, and iteratively update the weight parameters in the network until the specified number of iterations is reached.

[0019] The beneficial effect of the present invention is that the present invention uses language guidance to enable the target detection network to remove the interference of shooting conditions during the encoding process, and the target detection network theme is based on Faster-RCNN as the detection algorithm framework. First, in the first part, a language description template is constructed, and the shooting conditions are filled into the language description template. Then, the language description is segmented and embedded, and the language description is converted into a learnable vector embedding, and fine-tuning training is performed on the language vector of the relevant shooting conditions. After the training is completed, the language description of the relevant shooting conditions is passed through the language model encoder to obtain a language feature vector group. Then, in the second part, the target detection network is trained to compress the multi-stage feature map obtained by the backbone network into a one-dimensional vector. The vector is constrained by the loss function, and the language features of the relevant shooting conditions are removed while retaining the basic language description information. Therefore, the target detection network after language guidance can weaken or eliminate the interference caused by different shooting conditions during the feature extraction process, thereby improving the performance of the target detector. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is the overall flow chart of this target detection method.

[0021] Figure 2 Method for generating sets of language vectors for shooting conditions.

[0022] Figure 3 It is the FSN network structure. DETAILED DESCRIPTION

[0023] When a drone reaches high altitude, its onboard camera captures images, which are influenced by various shooting conditions, including weather, altitude, and shooting angle. Images captured under various shooting conditions can differ significantly, for example, between an image captured by a high-altitude camera looking straight ahead during the day and an image captured by a low-altitude camera looking down at night. Although the objects or content contained in the images remain largely unchanged, the different shooting conditions can significantly alter the objects within them. This means that the characteristics of the same type (class) of objects are affected by the different shooting conditions, with the characteristics of the shooting conditions superimposed on the object's own characteristics. An obvious approach is to eliminate the influence of the shooting conditions, i.e., to remove the shooting condition features superimposed on the object. To achieve this, the shooting condition features must be stable and highly abstract, similar to the characteristics of language descriptions. UAV onboard sensors can promptly acquire the state of the shooting conditions, and then various complex shooting conditions can be converted into language descriptions. To this end, the present invention designs a solution that uses language to describe the characteristics of the shooting conditions, thereby enabling the object detector to eliminate the influence of the shooting conditions.

[0024] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0025] like Figure 1 As shown, the present invention provides a target detection method based on language guidance in a drone scenario, comprising the following steps:

[0026] S1, fine-tune the language prompt feature vector to strengthen its connection with the drone scene; Figure 2 As shown, the specific method is:

[0027] S11. All shooting conditions are combined and populated into a fixed language template. Different shooting conditions are converted into language description prompts, and the language description prompts are processed through word segmentation and word vector encoding. To convert different shooting conditions into language description prompts, the shooting conditions of the drone during the shooting process are first grouped and classified. Weather conditions include [Daytime, Nighttime, Foggy], altitude conditions include [High Altitude (>70m), Medium Altitude (30m-70m), Low Altitude (<30m)], and shooting angle conditions include [Front View, Side View, Bird's Eye View, Front-Side View]. All shooting condition combinations are then populated into a fixed language template, such as a picture taken by a drone at [Weather Condition], [Altitude Condition], and [Shooting Angle]. An example after filling in the template is: a picture taken by a drone at a high altitude during the daytime from a bird's-eye view.

[0028] After the language description prompts are segmented and encoded with word vectors, any of the shooting conditions can be expressed as:

[0029] e s =[V1][V2]…[V ∝ ]…[V β ]…[V M ]

[0030] Among them [V m ]∈R 512 , m∈{1, 2, ..., M}, represents an encoded word embedding, whose dimension is consistent with the CLIP model. M represents the number of words in a sentence. There are 36 possible combinations of shooting conditions, which are represented by E s ∈R 36×M×512 to represent all language description hint embeddings.

[0031] S12. Fine-tune the template using a pre-trained language-vision model such as CLIP. To enhance the relevance of the language description to the relevant scene and eliminate potentially ambiguous parts of the language description, it is necessary to treat the embeddings in the language description as learnable vector embeddings and fine-tune some vectors to make the description more accurate. The CLIP model includes an image encoder and a language (text) encoder. During training, the encoded features of the two different modalities are mapped into the same feature space. Therefore, the present invention uses the trained CLIP model to complete feature mapping of different modalities.

[0032] Assume that the word indices of height and angle in a sentence are α and β respectively, and only these two word embeddings are considered as learnable variables for fine-tuning.

[0033] The specific process is as follows: Input a drone image x, and the vector after encoding by CLIP image encoder is v x , then the similarity of this picture to the 36 different shooting condition language prompts is:

[0034] z x =[sim(v x ,T(e1)),sim(v x ,T(e2)),...,sim(v x , T(e 36 ))

[0035] Where T() is the text encoder of CLIP, Represents cosine similarity. The goal of fine-tuning is to increase the similarity between x and the relevant shooting conditions, so the similarity L is defined text for:

[0036]

[0037] is the label corresponding to the image, LCE is the cross entropy loss function, and τ is the temperature coefficient, set to 0.01. Minimizing Ltext during training can improve the relevance of the fine-tuned language prompts to the drone scene.

[0038] S13, embed multiple groups of fine-tuned language description vectors about shooting conditions into the language encoder of CLIP to obtain a group of language vectors describing the shooting conditions; record all the fine-tuned language description prompt embeddings as E′ s , use the following formula to generate the shooting condition feature vector L described in language s ∈R 1×512 :

[0039]

[0040] By embedding the hints before and after fine-tuning into the input text encoder and averaging them, we can improve the relevance of the text while minimizing the deviation of the original meaning caused by overfitting the image data. The feature vector of each word is recorded as l s .

[0041] S2. Use language description to guide the training of the target detection network. The main purpose is to remove the interference caused by all shooting conditions during the encoding process of the target detection network. The L generated in the first part sThe shooting conditions can be accurately described. Therefore, the features of the image after being encoded by the neural network should not contain any s To ensure this process, we first need to map the feature space of the image to the L s In the language space where L is located. Secondly, through the constraint conditions, the encoding features do not contain L s .

[0042] The specific method is:

[0043] S21. The image data set obtained by the drone's onboard camera is used to extract the basic feature map through the ResNet series backbone network. The basic feature map is then input into the FPN feature pyramid network to obtain a multi-stage context feature map.

[0044] S22, passing the obtained multi-stage context feature map through a feature compression network with the same structure but different parameters to obtain a multi-stage one-dimensional feature vector;

[0045] The FSN network structure is as follows Figure 3 As shown, it consists of an Avg average pooling layer, three Resblock residual blocks, a GAvg global average pooling layer, and a fully connected network. It can be expressed as:

[0046]

[0047] Where Avg represents the Avg average pooling layer, and the original f x The size of C×H×W is compressed to C×K×K, where K is a fixed value, such as 16. Res represents the residual block, and each residual block reduces the size of the feature map to 1 / 2 of the original size. GAvg averages the width and height of the feature map, and reduces the width and height to 1. FC converts the final feature map into a feature vector, whose dimension is the same as l s and v x consistent.

[0048] S23. Use the pre-trained model CLIP to perform alignment-removal operations. To remove the influence of the shooting conditions expressed by the language description, it is first necessary to align the visual features with the language description features. Because the vector after the image is encoded by CLIP's image encoder and the vector after the language description is encoded by CLIP's language encoder are in the same feature space, the multi-stage one-dimensional feature vector obtained in S22 is aligned with the vector encoded by the CLIP image encoder, thereby ensuring that the multi-stage one-dimensional feature vector is mapped to the feature space of the language description; then, a loss function constraint is designed to remove the group of language description feature vectors related to the shooting conditions in the one-dimensional feature vector, thereby guiding the neural network to learn image features that do not contain shooting conditions;

[0049] Through the CLIP model, With v x Similar, you can The features of are mapped into the language feature space. sim This can be achieved by:

[0050] L sim = -(1-S)log(S)

[0051] in represents cosine similarity. First, the loss function converts the similarity value range to [0, 1]. Using a logarithmic function to expand the loss function range can accelerate convergence. At the same time, to balance different loss conditions, (1-S) is used to dynamically adjust the loss weight.

[0052] Then, in order to remove the linguistic features related to the shooting conditions from the image features that have been mapped into complex sentences, it is necessary to Stay away from L s By L unsim To remove features:

[0053]

[0054] in N s =36, which is the number of combinations of all shooting conditions.

[0055] S24, input the multi-stage context feature map into the region proposal network RPN and the region convolutional neural network RCNN in sequence to complete the target classification and positioning tasks, and obtain the loss function of the target detection network;

[0056] S25, sum the loss functions of S23 and S24 as the total loss function, and perform the back propagation algorithm; the loss function of the target detection network is recorded as L det , using L unsim +L sim Constraining the acquisition of domain-invariant features, the total loss function during training is:

[0057] L total =L det +L unsim +L sim

[0058] Among them L det Represents the loss function of the target detection network, including regression box loss and classification loss;

[0059] S26. Loop through the S21-S25 process, iteratively update the various weight parameters in the network so that the network prediction output of the training data approaches the ideal output as much as possible. Stop training after reaching the specified number of iterations to obtain the final target detection network.

[0060] During the test, no tensors passed through the FSN module because FSN is only an auxiliary network. After training is completed, the FSN network can be removed and the number of parameters of the entire model is consistent with Faster-RCNN.

[0061] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and it should be understood that the scope of protection of the present invention is not limited to such specific descriptions and embodiments. Those skilled in the art can make various other specific variations and combinations based on the technical teachings disclosed in the present invention without departing from the essence of the present invention, and such variations and combinations are still within the scope of protection of the present invention.

Claims

1. A target detection method based on language guidance in a UAV scenario, characterized in that: The following steps are involved: S1. Fine-tune the language prompt feature vector to strengthen its connection with the drone scene; the specific method is: S11. All shooting conditions are combined and filled into a fixed language template, different shooting conditions are converted into language description prompts, and the language description prompts are processed by word segmentation and word vector encoding; S12. Fine-tune the template using the pre-trained language-vision model; S13, embedding the multiple groups of fine-tuned language description vectors about the shooting conditions into the language encoder of the input CLIP to obtain a group of language vectors describing the shooting conditions; S2. Use language description to guide the training of the target detection network; the specific method is: S21. The image data set obtained by the drone's onboard camera is used to extract the basic feature map through the ResNet series backbone network. The basic feature map is then input into the FPN feature pyramid network to obtain a multi-stage context feature map. S22, passing the obtained multi-stage context feature map through a feature compression network with the same structure but different parameters to obtain a multi-stage one-dimensional feature vector; S23. Using the pre-trained model CLIP, perform an alignment-removal operation to align the multi-stage one-dimensional feature vector obtained in S22 with the vector encoded by the CLIP image encoder, thereby ensuring that the multi-stage one-dimensional feature vector is mapped to the feature space of the language description; then, design a loss function constraint to remove the language description features related to the shooting conditions in the one-dimensional feature vector; S24, input the multi-stage context feature map into the region proposal network RPN and the region convolutional neural network RCNN in sequence to complete the target classification and positioning tasks, and obtain the loss function of the target detection network; S25, summing the loss functions of S23 and S24 as the total loss function, and performing the back propagation algorithm; S26. Repeat the process of S21-S25 to iteratively update the weight parameters in the network until the specified number of iterations is reached.

Images