Multi-modal unmanned aerial vehicle image target detection method based on fine-tuning pre-training large model

Abstract

The invention provides a multi-modal unmanned aerial vehicle image target detection method based on a fine-tuning pre-training large model, and relates to the technical field of computer vision, and the method comprises the steps: obtaining an unmanned aerial vehicle image, and carrying out the preprocessing and marking of the unmanned aerial vehicle image, and obtaining an unmanned aerial vehicle data set; the method comprises the following steps: training and verifying Fast Segment Anything to obtain a target segmentation model after fine tuning training; the method comprises the following steps of: constructing a target network model, and obtaining an initial target detection model by adopting Grouping DINO and Fast Segment Anything as large models for pre-training; s5, constructing a speech recognition model; and S6, adding the speech recognition model into the initial target detection model to obtain a multi-mode full-automatic unmanned aerial vehicle image target detection model. According to the method, transfer learning is realized, and the problems of low detection precision and poor generalization caused by small unmanned aerial vehicle image data volume and high complexity in the prior art are effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular provides a multimodal drone image target detection method based on fine-tuning a pre-trained large model. Background Art

[0002] As drone technology continues to mature, drone image target detection technology, as one of the key technologies for drone applications, has gradually become a hot research topic. Drone image target detection has very broad application prospects in environmental monitoring, urban planning, disaster response and other fields. However, due to the complexity and variability of drone aerial images, drone image target detection faces a series of challenges such as background interference, target scale changes, target shape feature changes, etc., and it is necessary to seek more accurate, robust and responsive solutions.

[0003] In recent years, the rise of deep learning technology has brought revolutionary breakthroughs to drone image target detection, especially the target detection algorithms based on convolutional neural networks, such as the YOLO series, which have achieved remarkable results. However, due to the complexity and variability of drone aerial images, drone image target detection faces a series of challenges such as background interference, target scale changes, and target shape feature changes. At the same time, drone aerial images also face the problems of small image data volume and poor image quality. Under these conditions, it is extremely difficult to train an accurate and robust target detection model.

[0004] To address this problem, large pre-trained models have become a research breakthrough. These models have learned rich feature representations and semantic information through pre-training on massive data, and have been widely used in various computer vision tasks. However, there are a series of challenges in directly applying general pre-trained models to target detection in drone images. First, drone images have their own characteristics, which may make existing pre-trained models fail in feature extraction and target positioning. Secondly, the scale difference between the target and the background in drone images is large, and the target may be very small, which requires the model to have better scale adaptability. In addition, due to the real-time requirements of drone tasks, the target detection algorithm needs to have a faster reasoning speed while ensuring accuracy.

[0005] Therefore, the method of drone image target detection based on fine-tuning pre-trained large models has gradually become a research hotspot. Summary of the invention

[0006] The purpose of this application is to solve the problems of low detection accuracy and slow detection speed in existing drone target detection.

[0007] In order to achieve the above object, the present invention adopts the following technical solution: a multimodal drone image target detection method based on fine-tuning pre-trained large model, comprising:

[0008] S1, obtain drone images by shooting with drones;

[0009] S2, preprocessing and labeling the drone images to obtain the drone dataset;

[0010] S3. Use the drone dataset to train and verify Fast Segment Anything to obtain a fine-tuned target segmentation model.

[0011] S4, build the target network model, use GroundingDINO and Fast SegmentAnything trained in S3 as the pre-trained large model. The former obtains the detection box through text prompts, and then inputs the detection box into the latter to generate the segmentation mask of all objects in the image, and obtains the initial object detection model;

[0012] S5. Build a speech recognition model;

[0013] S6. Add the speech recognition model trained in S5 to the initial target detection model obtained in S4 to obtain a multimodal fully automatic drone image target detection model.

[0014] The multimodal drone image target detection method based on fine-tuning the pre-trained large model provided by the present invention also has such a technical feature that S2 includes:

[0015] S2.1: White edge processing of drone images;

[0016] S2.2: Perform data enhancement by flipping, rotating, splicing, cropping, and introducing Gaussian noise on the drone images to obtain an expanded drone image dataset;

[0017] S2.3: The expanded UAV image dataset is manually labeled using the labeling software LabelImg, and finally an image-matched and size-standardized UAV dataset is obtained.

[0018] The multimodal drone image target detection method based on fine-tuning the pre-trained large model provided by the present invention also has such a technical feature that S3 includes:

[0019] S3.1: Divide the drone dataset into training set and validation set;

[0020] S3.2: Load Fast Segment Anything model parameters;

[0021] S3.3: Input the training set into the network model for training and fine-tuning to obtain a fine-tuning model, substitute the verification set into the fine-tuning model for verification and optimization, and obtain the target segmentation model parameters and the target segmentation model after fine-tuning training.

[0022] The multimodal drone image target detection method based on fine-tuning the pre-trained large model provided by the present invention also has such a technical feature that the training fine-tuning in S3.3 includes:

[0023] Get the Fast Segment Anything model and pre-trained weights;

[0024] Freeze the prompt-guided selection in the Fast Segment Anything model and the backbone and neck networks in YOLOv8-seg so that they do not change during task-specific training;

[0025] The drone dataset is input into the Fast Segment Anything model for training, and some parameters of the YOLOv8-seg detection head in the Fast Segment Anything model are fine-tuned.

[0026] The multimodal drone image target detection method based on fine-tuning the pre-trained large model provided by the present invention also has such a technical feature that S5 includes:

[0027] S5.1: Deploy Whisper, a Transformer-based language recognition model;

[0028] S5.2: Prepare data and divide the data set into training set and validation set according to the preset allocation ratio;

[0029] S5.3: Use the training set to train the language recognition model Whisper, and use the validation set to verify and optimize it to obtain the language recognition model.

[0030] The multimodal drone image target detection method based on fine-tuning the pre-trained large model provided by the present invention also has the following technical features: the multimodal fully automatic drone image target detection model in S6 uses inference data of audio information and image information.

[0031] The multimodal drone image target detection method based on fine-tuning the pre-trained large model provided by the present invention also has the following technical feature: the audio information is a natural language obtained by an electronic transcription device or a voice acquisition device.

[0032] Beneficial Effects

[0033] The multimodal drone image target detection method based on fine-tuning the pre-trained large model proposed in the present invention is pre-trained on a large-scale data set. After learning the general feature representation, the pre-trained model is fine-tuned for the specific task of drone image target detection, realizing transfer learning, and effectively solving the problems of low detection accuracy and poor generalization caused by the small amount and high complexity of drone image data in the prior art. The target detection model finally obtained can be directly deployed on resource-constrained embedded devices, realizing online real-time target detection of drone images, and the introduction of the speech recognition module also enhances the human-computer interaction capability of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0035] Figure 1 A flowchart of a multimodal UAV image target detection method based on fine-tuning a pre-trained large model provided by the present invention;

[0036] Figure 2a is a visible light image of a drone in visDrone2019 in an embodiment of the present invention;

[0037] Figure 2b for Figure 2a Target detection results. DETAILED DESCRIPTION

[0038] The present application is further described in detail below in conjunction with the accompanying drawings and examples, but it should be noted that these implementation methods are not limitations of the present application, and equivalent changes or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these implementation methods are all within the scope of protection of the present application.

[0039] In the description of the embodiments of the present application, it should be understood that the terms "center", "longitudinal", "lateral", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the creation of the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the creation of the present application.

[0040] In addition, the terms "first", "second", "third", etc. are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first", "second", etc. may explicitly or implicitly include one or more of the features. In the description of the invention of this application, unless otherwise specified, the meaning of "plurality" is two or more.

[0041] The terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the invention of this application can be understood by specific circumstances.

[0042] like Figure 1 As shown, the embodiment of the present application provides a multimodal drone image target detection method based on fine-tuning a pre-trained large model, including:

[0043] S1. Obtain drone images by shooting with a drone. The drone carries a camera for aerial photography and sends the captured images through a wireless communication module to obtain drone images.

[0044] S2, preprocessing and labeling the drone images to obtain the drone dataset;

[0045] S3. Use the drone dataset to train and verify Fast Segment Anything to obtain a fine-tuned target segmentation model.

[0046] S4, build the target network model, use GroundingDINO and Fast SegmentAnything trained in S3 as the pre-trained large model. The former obtains the detection box through text prompts, and then inputs the detection box into the latter to generate the segmentation mask of all objects in the image, and obtains the initial object detection model;

[0047] S5. Build a speech recognition model, use multiple Chinese and English speech recognition data sets to train and optimize the speech recognition model, and obtain a speech recognition model that can realize the conversion of English audio to English text and Chinese audio to English text;

[0048] S6. Add the speech recognition model trained in S5 to the initial target detection model obtained in S4 to obtain a multimodal fully automatic drone image target detection model.

[0049] In some embodiments, the S2 includes:

[0050] S2.1: Perform white edge processing on the drone image; white edge processing can prevent image distortion in the subsequent adjustment process;

[0051] S2.2: Perform data enhancement by flipping, rotating, splicing, cropping, and introducing Gaussian noise on the drone images to obtain an expanded drone image dataset;

[0052] S2.3: The expanded UAV image dataset is manually labeled using the labeling software LabelImg, and finally an image-matched and size-standardized UAV dataset is obtained.

[0053] The size of the image is 1024*1024, and the mask of the image is saved as a json file.

[0054] In some embodiments, the S3 includes:

[0055] S3.1: Divide the drone dataset into training set and validation set, with the ratio of training set: validation set = 8:2;

[0056] S3.2: Load Fast Segment Anything model parameters;

[0057] S3.3: Input the training set into the network model for training and fine-tuning to obtain a fine-tuning model, substitute the verification set into the fine-tuning model for verification and optimization, obtain the target segmentation model parameters and the target segmentation model after fine-tuning training, and obtain the target segmentation model weights and name them my_fastsam.pt.

[0058] In some embodiments, the training fine-tuning in S3.3 includes:

[0059] Get the Fast Segment Anything model and pre-trained weights. The weight file is named FastSAM-x.pt.

[0060] Freeze the prompt-guided selection in the Fast Segment Anything model and the backbone and neck networks in YOLOv8-seg so that they do not change with the training of a specific task and are only used to extract general features;

[0061] Input the drone dataset into the Fast Segment Anything model for training, and fine-tune some parameters of the YOLOv8-seg detection head in the Fast Segment Anything model;

[0062] Use the default hyperparameter settings to train the model for 200 epoches, check the training results every 20 epoches, and terminate the training early if the loss no longer decreases or overfitting occurs.

[0063] In some embodiments, the S4 includes:

[0064] S4.1 Install the Grounding Dino model and download the pre-trained weights as groundingdino_swint_ogc.pth;

[0065] S4.2 Install the Fast Segment Anything model and download the training weights my_fastsam.pt;

[0066] S4.3 uses the detection box output by the Grounding Dino model as the input of the Fast Segment Anything model to obtain the target detection model.

[0067] In some embodiments, the S5 includes:

[0068] S5.1: Deploy Whisper, a Transformer-based language recognition model;

[0069] S5.2: Prepare data and divide the data set into training set and validation set according to the preset allocation ratio;

[0070] S5.3: Use the training set to train the language recognition model Whisper, and use the validation set to verify and optimize it to obtain the language recognition model.

[0071] In some embodiments, the prepared data of S5.2 only needs to include the input audio sample and the corresponding transcription text, and no additional information is considered. The specific steps are as follows:

[0072] S5.2.1 When audio data is first loaded, it should be resampled to 16KHz;

[0073] S5.2.2 calls the Whisper feature extractor, which mainly includes two operations.

[0074] S5.2.2.1 The input length of all samples is unified to 30 seconds by padding or truncating a batch of samples. For samples shorter than 30 seconds, 0 is added to the end of the audio signal. For samples longer than 30 seconds, they are directly truncated to 30 seconds.

[0075] S5.2.2.2 transform the audio input into a log mel spectrogram;

[0076] S5.2.3 Use Whisper tokenizer to encode the recorded text into IDs;

[0077] S5.2.4 The data set allocation ratio is 8:2.

[0078] In some embodiments, the S5.3 includes:

[0079] S5.3.1 converting the logarithmic Mel spectrogram into a PyTorch tensor;

[0080] S5.3.2 Define the word error rate (WER) metric to evaluate the model;

[0081] S5.3.2.1 Loading WER metrics from Evaluate

[0082] S5.3.2.2 Define the compute_metrics function to accept the model output and return the WER metric, and calculate the WER between the output text and the real text

[0083] S5.3.3 Load the pre-trained checkpoint and configure it correctly for training;

[0084] S5.3.4 Define training parameters. Specifically, set the learning rate to 1e-5, the batch size to 16, train 200 epoches, check the training results every 20 epoches, and terminate the training early if the loss no longer decreases or overfitting occurs; save the optimal model training weights and name them my_whisper.pt;

[0085] S5.3.5 Start training.

[0086] In some embodiments, the multimodal fully automatic drone image target detection model in S6 uses inference data of audio information and image information.

[0087] In some embodiments, the audio information is natural language obtained by an electronic transcription device or a voice acquisition device.

[0088] For example Figure 2a The visible light image of the drone in visDrone2019 shown in the figure has cars, trees and other objects in the background. Using the method provided in any of the above embodiments, the image is subjected to target detection and segmentation using the pre-recorded voice with the content "Please help me find trees, cars, and people." The result is as follows: Figure 2b shown.

[0089] The above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application. The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and variations can be made without departing from the technical principles of the present application, and these improvements and variations should also be regarded as the protection scope of the present application.

Claims

1. A multimodal drone image target detection method based on fine-tuning pre-trained large model, characterized in that: include: S1, obtain drone images by shooting with drones; S2, preprocessing and labeling the drone images to obtain the drone dataset; S3. Use the drone dataset to train and verify Fast Segment Anything to obtain a fine-tuned target segmentation model. S4, build the target network model, use GroundingDINO and Fast Segment Anything trained in S3 as the pre-trained large model. The former obtains the detection box through text prompts, and then inputs the detection box into the latter to generate the segmentation mask of all objects in the image, and obtains the initial object detection model; S5. Build a speech recognition model; S6. Add the speech recognition model trained in S5 to the initial target detection model obtained in S4 to obtain a multimodal fully automatic drone image target detection model.

2. The multimodal drone image target detection method based on fine-tuning pre-trained large model according to claim 1 is characterized in that: The S2 includes: S2.1: White edge processing of drone images; S2.2: Perform data enhancement by flipping, rotating, splicing, cropping, and introducing Gaussian noise on the drone images to obtain an expanded drone image dataset; S2.3: The expanded UAV image dataset is manually labeled using the labeling software LabelImg, and finally an image-matched and size-standardized UAV dataset is obtained.

3. The multimodal drone image target detection method based on fine-tuning pre-trained large model according to claim 1 is characterized in that: The S3 includes: S3.1: Divide the drone dataset into training set and validation set; S3.2: Load Fast Segment Anything model parameters; S3.3: Input the training set into the network model for training and fine-tuning to obtain a fine-tuning model, substitute the verification set into the fine-tuning model for verification and optimization, and obtain the target segmentation model parameters and the target segmentation model after fine-tuning training.

4. The multimodal drone image target detection method based on fine-tuning pre-trained large model according to claim 3 is characterized in that: The training fine-tuning in S3.3 includes: Get the Fast Segment Anything model and pre-trained weights; Freeze the prompt-guided selection in the Fast Segment Anything model and the backbone and neck networks in YOLOv8-seg so that they do not change during task-specific training; The drone dataset is input into the Fast Segment Anything model for training, and some parameters of the YOLOv8-seg detection head in the Fast Segment Anything model are fine-tuned.

5. The multimodal drone image target detection method based on fine-tuning pre-trained large model according to claim 1 is characterized in that: The S5 includes: S5.1: Deploy Whisper, a Transformer-based language recognition model; S5.2: Prepare data and divide the data set into training set and validation set according to the preset allocation ratio; S5.3: Use the training set to train the language recognition model Whisper, and use the validation set to verify and optimize it to obtain the language recognition model.

6. The multimodal drone image target detection method based on fine-tuning pre-trained large model according to claim 1 is characterized in that: The multimodal fully automatic drone image target detection model in S6 uses audio information and image information as inference data.

7. The multimodal drone image target detection method based on fine-tuning pre-trained large model according to claim 6 is characterized in that: The audio information is natural language obtained by an electronic transcription device or a voice collection device.

Images