Transform-based unmanned aerial vehicle visual processing method and device and storage medium

Abstract

The invention relates to an unmanned aerial vehicle visual processing method and device based on Transform and a storage medium. The method comprises the steps that image acquisition equipment installed on an unmanned aerial vehicle acquires an image; based on a backbone network Swin Transform, through a YOLO target detection algorithm, high-level feature representation of the image is extracted; processing features of the image at different scales based on the feature pyramid network, and generating a multi-scale feature map; predicting the category and bounding box of the target based on the detection head of the image on each scale; and generating a prediction bounding box of the image based on the anchor box of the YOLO. According to the unmanned aerial vehicle visual processing method and device based on the Transform, the defect that in a real scene, target detection is extremely difficult due to factors such as scene change, shooting distortion and target shielding when target detection is applied to an unmanned aerial vehicle can be overcome, and a relatively excellent image processing result is extracted.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence, and in particular relates to a Transformer-based unmanned aerial vehicle vision processing method, device and storage medium. Background Art

[0002] Transportation plays an important role in the fields of national economic construction and national defense construction, and is closely related to the daily life of the people. With the rapid development of road traffic, the construction of the road network has become more perfect, and it has become particularly important to maintain the order of the road. During urban construction, the form of human supervision is relatively traditional, with large investment and poor results, and unreasonable supervision may occur. Using drone remote sensing technology, with the help of new technical equipment, it is not necessary to use more manpower to control the equipment, and timely capture dynamic and high-definition images, so as to provide accurate data information for managers. The drone itself has the characteristics of small size, low cost, flexibility and efficiency, and strong environmental adaptability. It can complete the task of collecting high-resolution images in harsh environments. At the same time, compared with traditional collection methods, drone shooting can cover a wider path, and the collected road images are more comprehensive, saving a lot of manpower, material resources and financial resources.

[0003] With the rapid development of artificial intelligence, image target detection and tracking technology has grown into a long-standing and challenging problem and has been an active field for decades. Target detection is a basic task of computer vision. It is also the basis for image technology and computer vision technology to solve more complex or higher-level visual tasks, such as image segmentation, semantic interpretation, target tracking, image localization, activity recognition, etc. The purpose of target detection is to determine whether there is a target object in the instance category in an image, and if so, return the category and spatial position of the target object. However, in real scenes, due to scene transformation, shooting distortion, and target occlusion problems, target detection becomes extremely difficult, especially when target detection is applied to drones. The accuracy and speed of target detection are easily affected by the flight environment and lighting conditions, which increases the difficulty of research. Therefore, a large number of scholars are still studying this issue. Summary of the invention

[0004] The present invention provides a Transformer-based unmanned aerial vehicle vision processing method, device and storage medium, aiming to solve at least one of the technical problems existing in the prior art.

[0005] The technical solution of the present invention relates to a Transformer-based UAV vision processing method, which comprises the following steps:

[0006] S100, an image acquisition device installed on the UAV acquires images;

[0007] S200, based on the backbone network Swin Transformer, uses the YOLO target detection algorithm to extract high-level feature representation of the image;

[0008] S300, based on the feature pyramid network, processes the features of the image at different scales and generates multi-scale feature maps;

[0009] S400, based on the detection head at each scale of the image, predict the category and bounding box of the target;

[0010] S500: Generate a predicted bounding box of the image based on the YOLO anchor box.

[0011] Furthermore, the Transformer-based drone vision processing method also includes the following steps:

[0012] S600, calculating the difference between the prediction result and the true label based on the loss function;

[0013] S700, based on non-maximum suppression, remove redundant bounding boxes of the image and retain the most likely target box;

[0014] S800: Output the detected target category, confidence level, and bounding box information.

[0015] Furthermore, in step S200, the backbone network Swin Transformer is obtained through a self-supervised learning training process, and the self-supervised learning training process includes the following steps:

[0016] S210, collecting and preparing unlabeled training images;

[0017] S220, extracting image patches from the training image as training samples;

[0018] S230, constructing an Auto-Encoder task, inputting the training sample into the encoder of the backbone network SwinTransformer, the encoder maps the image to a latent space, and the decoder restores the original image patches from the latent space;

[0019] S240, introducing a Visual Tokens task, comparing the Token of the image block with the negative sample Token to maximize the similarity between positive samples and minimize the similarity between negative samples, wherein the Token of the image block is the output of the encoder;

[0020] S250, calculate the loss of Auto-Encoder task and Visual Tokens task;

[0021] S260, combining the losses of the Auto-Encoder task and the Visual Tokens task, and updating the parameters of the backbone network Swin Transformer through gradient back propagation;

[0022] S270 , repeating steps S210 to S260 , and iteratively using unlabeled training images for training.

[0023] Furthermore, in step S400, the detection head includes a classification branch detection head for predicting the target category and a regression branch detection head for predicting the coordinates of the image bounding box.

[0024] Furthermore, in step S600, the loss function includes a classification loss function and a bounding box regression loss function.

[0025] Furthermore, in step S210, training images are extracted based on the VisDrone dataset.

[0026] Furthermore, in step S220, the image blocks are obtained by random cropping.

[0027] Furthermore, in step S250, the loss of the Auto-Encoder task includes the reconstruction error, that is, the difference between the input image block and the decoder output; the loss of the Visual Tokens task is a contrast loss, which makes similar image blocks closer in the latent space.

[0028] Furthermore, the present invention also proposes a Transformer-based UAV visual processing device for implementing the above-mentioned Transformer-based UAV visual processing method, wherein the device comprises:

[0029] A data acquisition device, used for acquiring image data, wherein the data acquisition device is arranged on the drone;

[0030] The data prediction device is used to detect the image and draw a boundary frame for the image. The data prediction device is electrically connected to the data acquisition device.

[0031] Furthermore, the present invention also relates to a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement the above-mentioned method when executed by a processor.

[0032] Compared with the prior art, the present invention has the following characteristics.

[0033] The present invention uses a Transformer-based drone vision processing method, device and storage medium to overcome the difficulty of target detection due to scene change, shooting distortion, target occlusion and other factors in real scenarios when applying target detection on drones, and extract relatively excellent image processing results. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 This is a flowchart of the Transformer-based UAV vision processing method;

[0035] Figure 2 Flowchart of the self-supervised learning training process of the backbone network Swin Transformer in the Transformer-based drone vision processing method;

[0036] Figure 3 This is a schematic diagram of the Transformer-based UAV vision processing method;

[0037] Figure 4 This is a schematic diagram of a Transformer-based drone vision processing device;

[0038] Reference numerals

[0039] 100. Data collection device; 200. Data prediction device. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The concept, specific structure and technical effects of the present invention will be clearly and completely described below in combination with the embodiments and drawings to fully understand the purpose, scheme and effect of the present invention.

[0042] It should be noted that, unless otherwise specified, when a feature is referred to as being "fixed" or "connected" to another feature, it may be directly fixed or connected to another feature, or it may be indirectly fixed or connected to another feature. The singular forms "a", "said" and "the" used herein are also intended to include the plural forms, unless the context clearly indicates otherwise. In addition, unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art. The terms used in this specification are intended only to describe specific embodiments and are not intended to limit the invention. The term "and / or" used herein includes any combination of one or more of the related listed items.

[0043] It should be understood that although the terms first, second, third, etc. may be used to describe various elements in the present disclosure, these elements should not be limited to these terms. These terms are only used to distinguish elements of the same type from each other. For example, without departing from the scope of the present disclosure, the first element may also be referred to as the second element, and similarly, the second element may also be referred to as the first element. The use of any and all examples or exemplary language ("for example", "such as", etc.) provided herein is intended only to better illustrate embodiments of the present invention, and unless otherwise required, will not impose limitations on the scope of the present invention. In addition, the industry term "posture" used herein refers to the position and attitude of a certain element relative to a spatial coordinate system.

[0044] Reference Figures 1 to 4 , the embodiment of the present invention provides a Transformer-based UAV visual processing method, referring to Figure 1 and Figure 3 , the method specifically comprises the following steps:

[0045] S100, an image acquisition device installed on the UAV acquires images;

[0046] S200, based on the backbone network Swin Transformer, uses the YOLO target detection algorithm to extract high-level feature representation of the image;

[0047] S300, based on the feature pyramid network, processes the features of the image at different scales and generates multi-scale feature maps;

[0048] S400, based on the detection head at each scale of the image, predict the category and bounding box of the target;

[0049] S500: Generate a predicted bounding box of the image based on the YOLO anchor box.

[0050] Compared with the prior art, the present invention has the following characteristics.

[0051] The present invention uses a Transformer-based drone vision processing method and device to overcome the difficulty of target detection due to scene change, shooting distortion, target occlusion and other factors in real scenarios, and extract relatively excellent image processing results.

[0052] Specifically, (1) using the VisDrone dataset, the Transformer-based target detection algorithm has better detection accuracy for abstract targets than the traditional target detection algorithm. At the same time, it can be applied to situations such as dense targets, overlapping targets, target sizes, and perspective changes in images taken by drones and other equipment.

[0053] (2) Use TensorRT to accelerate AI reasoning and deploy small airborne terminal computing devices on drones. TensorRT is a high-performance deep learning reasoning optimizer that provides low latency and high throughput for reasoning applications, and executes 36 times faster than platforms that only use CPUs.

[0054] (3) Use Swin Transformer as the backbone network to extract features while retaining the One-Stage characteristics of the YOLO algorithm. Use the autonomous driving technology of drones to conduct real-time dynamic monitoring of different elements in the city, such as vehicles, people, facilities, and environment, and collect information at any time. And through the self-supervision method in the BEiT model, use a large amount of unlabeled low-cost data to train the model to improve the model detection accuracy.

[0055] Specifically, in terms of data collection, the present invention uses drone aerial photography monitoring images to obtain comprehensive data; in terms of target detection algorithms, the YOLO target detection algorithm is used as the basis, and Swin Transformer is used as the backbone network; in terms of drone visual monitoring, drone automatic driving technology sensors are used to dynamically monitor urban vehicles, people, facilities, environment and other elements in real time; real-time monitoring information is collected; self-supervised learning training backbone network uses a large amount of unlabeled low-cost data for self-supervised learning, and trains the backbone network. The self-supervised method in the BEiT model is used, combined with Auto-Encoder and Visual Tokens. In terms of image feature restoration, the Visual Tokens of the specified area of ​​the image are restored to replace the pixel-level image restoration method.

[0056] In some specific embodiments, in step S200, Swin Transformer Backbone: Swin Transformer is used as a backbone network to extract high-level feature representation of the image. Swin Transformer is a Transformer-based visual attention model with good receptive field and context understanding capabilities.

[0057] In some specific embodiments, in step S300, Feature Pyramid Network (FPN): in order to process features of different scales, FPN can be connected after Swin Transformer to generate multi-scale feature maps.

[0058] In some specific embodiments, in step S400, a detection head: There is a detection head at each scale, which is responsible for predicting the category and bounding box of the target. Generally, the detection head includes a classification branch (predicting the target category) and a regression branch (predicting the bounding box coordinates).

[0059] In some specific embodiments, in step S500, anchor boxes: YOLO uses anchor boxes to generate predicted bounding boxes. Anchor boxes are some predefined bounding boxes that perform object detection by matching with real bounding boxes.

[0060] Further, refer to Figure 1 and Figure 3 , the Transformer-based drone vision processing method also includes the following steps:

[0061] S600, calculating the difference between the prediction result and the true label based on the loss function;

[0062] S700, based on non-maximum suppression, remove redundant bounding boxes of the image and retain the most likely target box;

[0063] S800: Output the detected target category, confidence level, and bounding box information.

[0064] In some specific embodiments, in step S600, loss calculation: using a loss function to calculate the difference between the model prediction value and the true label. The loss function generally includes classification loss and bounding box regression loss.

[0065] In some specific embodiments, in step S700, non-maximum suppression (NMS) is used to remove redundant bounding boxes and retain the most likely target box.

[0066] In some specific embodiments, in step S800, the result is output: the detected target category, confidence level and bounding box information are finally output.

[0067] Further, refer to Figure 2 In step S200, the backbone network Swin Transformer is obtained through a self-supervised learning training process, and the self-supervised learning training process includes the following steps:

[0068] S210, collecting and preparing unlabeled training images;

[0069] S220, extracting image patches from the training image as training samples;

[0070] S230, constructing an Auto-Encoder task, inputting the training sample into the encoder of the backbone network SwinTransformer, the encoder maps the image to a latent space, and the decoder restores the original image patches from the latent space;

[0071] S240, introducing a Visual Tokens task, comparing the Token of the image block with the negative sample Token to maximize the similarity between positive samples and minimize the similarity between negative samples, wherein the Token of the image block is the output of the encoder;

[0072] S250, calculate the loss of Auto-Encoder task and Visual Tokens task;

[0073] S260, combining the losses of the Auto-Encoder task and the Visual Tokens task, and updating the parameters of the backbone network Swin Transformer through gradient back propagation;

[0074] S270 , repeating steps S210 to S260 , and iteratively using unlabeled training images for training.

[0075] Specifically, in some specific embodiments,

[0076] In step S210, input data preparation: collecting a large amount of unlabeled and low-cost image data.

[0077] In step S220, Patch Extraction: extract image patches from the image as training samples. These image patches can be randomly cropped or obtained through other strategies.

[0078] In step S230, the Auto-Encoder task: uses the image block as input, and a part of the backbone network constructs the Auto-Encoder task. The encoder maps the image block to the latent space, and the decoder attempts to restore the original image block from the latent space. The encoder is responsible for extracting key features from the input data, and the decoder restores the abstract representation generated by the encoder to the original input data, generating data that is as similar as possible to the original input from the high-dimensional, abstract representation.

[0079] In step S240, Contrastive Learning with Visual Tokens: Visual Tokens task is introduced, and the encoder output of the image block is used as a token. The token of the image block is compared with the tokens of other negative samples to maximize the similarity between positive samples and minimize the similarity between negative samples. The optimal bounding box is extracted by NMS technology. NMS is a commonly used technology in the field of object detection, which is used to select the optimal bounding box from the bounding box generated by the model to remove redundant detection results.

[0080] In step S250, Loss calculation: Calculate the loss of the Auto-Encoder task and the Visual Tokens task. The loss of the Auto-Encoder task usually includes the reconstruction error, that is, the difference between the input image block and the decoder output. The loss of the Visual Tokens task can use contrast loss to make similar image blocks closer in the latent space.

[0081] In step S260, gradient back propagation: the losses of the two tasks are combined and the parameters of the backbone network are updated through gradient back propagation.

[0082] In step S270, iterative training: repeat the above process, iteratively use unlabeled data for training, and obtain a pre-trained backbone network: after self-supervised learning, a backbone network pre-trained on unlabeled data is obtained.

[0083] Further, refer to Figure 1 and Figure 3 In step S400, the detection head includes a classification branch detection head for predicting the target category and a regression branch detection head for predicting the image bounding box coordinates.

[0084] Further, refer to Figure 1 and Figure 3 In step S600, the loss function includes a classification loss function and a bounding box regression loss function.

[0085] Further, refer to Figure 2 In step S210, training images are extracted based on the VisDrone dataset.

[0086] Further, refer to Figure 2 In the step S220, the image block is obtained by random cropping.

[0087] Further, refer to Figure 2 In step S250, the loss of the Auto-Encoder task includes the reconstruction error, that is, the difference between the input image block and the decoder output; the loss of the Visual Tokens task is the contrast loss, which makes similar image blocks closer in the latent space.

[0088] Further, refer to Figure 4 The present invention also proposes a Transformer-based UAV visual processing device for implementing the Transformer-based UAV visual processing method. The device comprises:

[0089] A data acquisition device 100 is used to collect image data. The data acquisition device 100 is arranged on a drone;

[0090] The data prediction device 200 is used to detect an image and perform boundary frame drawing on the image. The data prediction device 200 is electrically connected to the data acquisition device.

[0091] Furthermore, the present invention also relates to a computer-readable storage medium having program instructions stored thereon, wherein the program instructions implement the above-mentioned method when executed by a processor.

[0092] The Transformer-based drone vision processing method and device described above studies the application scenarios of drones in smart cities, focusing on computer vision technology based on deep learning methods. Explore new application scenarios that cutting-edge computer vision technology can bring, and improve computer vision technology to suit current scenarios to carry more reliable smart city services. When applying computer vision technology, a neural network algorithm with Transformer added will be used to perform real-time target detection in images collected by drones. The detection accuracy is improved with the help of a model structure with stronger learning ability, while maintaining the detection speed required for real-time target detection. At the same time, we will try to perform self-supervised learning in target detection tasks, so that the model can learn useful knowledge for target detection through a large number of unlabeled image data sets to solve the problem of insufficient data sets. Through the above methods, the automation of drones is finally realized, and a solution for image detection application scenarios in smart cities is given.

[0093] It should be appreciated that the method steps in the embodiments of the present invention can be implemented or implemented by computer hardware, a combination of hardware and software, or by computer instructions stored in a non-transitory computer readable memory. The method can use standard programming techniques. Each program can be implemented in a high-level process or object-oriented programming language to communicate with a computer system. However, if necessary, the program can be implemented in an assembly or machine language. In any case, the language can be a compiled or interpreted language. In addition, the program can be run on a programmed ASIC for this purpose.

[0094] Furthermore, the operations of the processes described herein may be performed in any suitable order unless otherwise indicated herein or otherwise clearly contradicted by context. The processes described herein (or variations and / or combinations thereof) may be performed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is executed collectively on one or more processors, by hardware, or a combination thereof. The computer program includes a plurality of instructions that may be executed by one or more processors.

[0095] Further, the method can be implemented in any type of computing platform that is operably connected to a suitable computer, including but not limited to a personal computer, a minicomputer, a mainframe, a workstation, a network or distributed computing environment, a separate or integrated computer platform, or in communication with a charged particle tool or other imaging device, etc. Various aspects of the present invention can be implemented in machine-readable code stored on a non-transitory storage medium or device, whether removable or integrated into a computing platform, such as a hard disk, an optical read and / or write storage medium, a RAM, a ROM, etc., so that it can be read by a programmable computer, and when the storage medium or device is read by the computer, it can be used to configure and operate the computer to perform the process described herein. In addition, the machine-readable code, or portions thereof, can be transmitted via a wired or wireless network. When such media includes instructions or programs that implement the steps described above in conjunction with a microprocessor or other data processor, the invention described herein includes these and other different types of non-transitory computer-readable storage media. When programmed according to the methods and techniques of the present invention, the present invention can also include the computer itself.

[0096] The computer program can be applied to input data to perform the functions described herein, thereby converting the input data to generate output data stored in a non-volatile memory. The output information can also be applied to one or more output devices such as a display. In a preferred embodiment of the present invention, the converted data represents physical and tangible objects, including specific visual depictions of physical and tangible objects produced on the display.

[0097] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the scope of protection of the present invention. Within the scope of protection of the present invention, its technical scheme and / or implementation method may have various modifications and changes.

Claims

1. The Transformer-based UAV visual processing method is characterized by: The method comprises the following steps: S100, an image acquisition device installed on the UAV acquires images; S200, based on the backbone network Swin Transformer, uses the YOLO target detection algorithm to extract high-level feature representation of the image; S300, based on the feature pyramid network, processes the features of the image at different scales and generates multi-scale feature maps; S400, based on the detection head at each scale of the image, predict the category and bounding box of the target; S500: Generate a predicted bounding box of the image based on the YOLO anchor box.

2. The Transformer-based UAV visual processing method according to claim 1, characterized in that: The following steps are also included: S600, calculating the difference between the prediction result and the true label based on the loss function; S700, based on non-maximum suppression, remove redundant bounding boxes of the image and retain the most likely target box; S800: Output the detected target category, confidence level, and bounding box information.

3. The Transformer-based UAV visual processing method according to claim 1, characterized in that: In step S200, the backbone network Swin Transformer is obtained through a self-supervised learning training process, and the self-supervised learning training process includes the following steps: S210, collecting and preparing unlabeled training images; S220, extracting image patches from the training image as training samples; S230, constructing an Auto-Encoder task, inputting the training sample into the encoder of the backbone network Swin Transformer, the encoder maps the image to a latent space, and the decoder restores the original image patches from the latent space; S240, introducing a Visual Tokens task, comparing the Token of the image block with the negative sample Token to maximize the similarity between positive samples and minimize the similarity between negative samples, wherein the Token of the image block is the output of the encoder; S250, calculate the loss of Auto-Encoder task and Visual Tokens task; S260, combining the losses of the Auto-Encoder task and the Visual Tokens task, and updating the parameters of the backbone network Swin Transformer through gradient back propagation; S270 , repeating steps S210 to S260 , and iteratively using unlabeled training images for training.

4. The Transformer-based UAV visual processing method according to claim 1, characterized in that: In step S400, the detection head includes a classification branch detection head for predicting the target category and a regression branch detection head for predicting the coordinates of the image bounding box.

5. The Transformer-based UAV visual processing method according to claim 2, characterized in that: In step S600, the loss function includes a classification loss function and a bounding box regression loss function.

6. The Transformer-based UAV visual processing method according to claim 3 is characterized in that: In the step S210, training images are extracted based on the VisDrone dataset.

7. The Transformer-based UAV visual processing method according to claim 3, characterized in that: In the step S220, the image blocks are obtained by random cropping.

8. The Transformer-based UAV visual processing method according to claim 3, characterized in that: In step S250, the loss of the Auto-Encoder task includes the reconstruction error, that is, the difference between the input image block and the decoder output; the loss of the Visual Tokens task is the contrast loss, which makes similar image blocks closer in the latent space.

9. A Transformer-based UAV visual processing device, used to implement the Transformer-based UAV visual processing method according to any one of claims 1 to 8, characterized in that: The device comprises: A data acquisition device (100) is used to acquire image data, wherein the data acquisition device (100) is arranged on a drone; The data prediction device (200) is used to detect an image and perform boundary frame drawing on the image, and the data prediction device (200) is electrically connected to the data acquisition device. 10 . A computer-readable storage medium having program instructions stored thereon, wherein the program instructions are executed by a processor to implement the method according to claim 1 .

Images