Ship target detection method based on visual language model

Abstract

The present invention discloses a method for detecting a ship target based on a visual language model, and the steps include: establishing a ship target detection model based on a visual language model, the model including a YOLO feature extraction branch, a visual language module branch and a detection head; the YOLO feature extraction branch includes a YOLO Backbone, which is used to extract a plurality of first feature maps of different scales from an input image; the visual language module branch includes a block module and a visual language module, which extracts a second feature map of the same scale as each first feature map; each group of mutually corresponding first feature maps and second feature maps are spliced ​​respectively, and then each group of spliced ​​feature maps is simultaneously input into the detection head to obtain a detection result. The present invention adds a visual language module, utilizes the excellent generalization performance and understanding ability of the visual language model to extract the semantic information of the image, assists the target detection task, improves the generalization performance of the model and enables it to have a certain semantic understanding ability.

Description

Technical Field

[0001] The invention belongs to the technical field of specific computing models, and in particular relates to a ship target detection method. Background Art

[0002] There are two main types of target detection methods. The first type is the classic two-stage method, which divides the entire detection process into two steps: first, predicting candidate regions that may contain targets through a neural network; second, using a classification network to predict the categories of these regions. Algorithms such as R-CNN and SPPNet are typical representatives of this type of method. The second type of method uses a single deep neural network to directly predict the bounding box and category of the target. Since it omits the step of predicting candidate regions, it is called a one-stage method, and its representative algorithm is YOLO. Both types of methods have good performance in accuracy, but the one-stage method is usually better than the two-stage method in efficiency because it predicts both the bounding box and the category at the same time, making the reasoning speed faster.

[0003] However, in order to achieve good detection results, traditional target detection algorithms usually rely on a large amount of labeled data. The acquisition cost of such data is high and the collection process is complicated. Especially in target detection tasks in specific fields, such as ship target detection or defect detection, it is particularly difficult to obtain high-quality and large quantities of labeled data. In addition, due to the particularity of these tasks, large-scale public datasets are often not applicable, resulting in poor pre-training results. Therefore, the detection accuracy and generalization performance of traditional target detection models are usually poor. Summary of the invention

[0004] The present invention proposes a ship target detection method based on a visual language model, the purpose of which is to solve the problems of poor detection accuracy and generalization performance under the condition of limited data volume.

[0005] The technical solution of the present invention is as follows:

[0006] A method for detecting a ship target based on a visual language model, comprising the following steps:

[0007] Step 1: Establish a ship target detection model based on the visual language model;

[0008] The ship target detection model includes a YOLO feature extraction branch and a visual language module branch, and also includes a detection head;

[0009] The YOLO feature extraction branch includes a YOLO Backbone, which is used to extract a plurality of first feature maps of different scales from an input image;

[0010] The visual language module branch includes a block module and a visual language module; the block module is used to divide the input image into multiple groups of block input images according to different segmentation numbers, and each segmentation number corresponds to the size of each first feature map extracted by the YOLO feature extraction branch;

[0011] Each group of block input images is input into the visual language module to obtain a second feature map with the same scale as the corresponding first feature map;

[0012] Each group of mutually corresponding first feature maps and second feature maps are spliced ​​respectively, and then each group of spliced ​​feature maps is simultaneously input into the detection head to obtain the detection result;

[0013] Step 2: training the above-mentioned ship target detection model based on the visual language model;

[0014] Step 3: Input the ship image as an input image into the trained ship target detection model based on the visual language model to obtain the detection result.

[0015] As a further improvement of the ship target detection method based on the visual language model: the visual language module includes a Prompt predictor, a BLIP image-to-text model and a CLIP text encoder;

[0016] The prompt predictor adaptively predicts the debugging template prompt according to the content and position of each block input image. The BLIP image-to-text model generates the corresponding descriptive text according to each block input image. The debugging template prompt and the descriptive text corresponding to each block input image are spliced ​​and input into the CLIP text encoder to obtain the corresponding block output features. The process is expressed as:

[0017] ;

[0018] in, is the first block obtained after the input image is divided Row, No. The column-wise block input image, represents the processing of the Prompt predictor, It represents the processing process of the BLIP graph-to-text model. Represents the processing of the CLIP text encoder, For the block input image The corresponding block output features;

[0019] Then, all the block output features of each component block input image are recombined to obtain a second feature map corresponding to the component block output feature.

[0020] As a further improvement of the ship target detection method based on the visual language model, a multi-stage method is used for training in step 2:

[0021] Step 2-1: Train YOLO Backbone based on the ship detection training set;

[0022] Step 2-2, train the Prompt predictor;

[0023] Step 2-3: Use the trained YOLO Backbone and Prompt predictor to train the detection head.

[0024] As a further improvement of the ship target detection method based on the visual language model, in step 2-1, before training the YOLO Backbone, the pre-trained weights are used as the initial weights of the YOLO Backbone.

[0025] As a further improvement of the ship target detection method based on the visual language model, the specific process of step 2-2 is:

[0026] Step 2-2-1, construct a ship predictor training set, which includes a number of block images and category labels corresponding to the classification images, and the category labels are represented as , is the number of vessel categories, 1 means that the block image is The positive sample of the category is 0, which means that the block image is Negative samples of categories;

[0027] When constructing, first divide several original images into blocks to obtain block images, and then annotate the category labels for each block image: set all element values ​​of the category labels of all block images to 0, traverse the target boxes in the original image, and if the center point of a target box falls into a block image, set the target box corresponding to the first If there are multiple categories, the element value corresponding to the category label of the block image is set to 1;

[0028] Step 2-2-2: Construct a CLIP classifier based on the Prompt predictor. Suppose the processing of the CLIP image encoder of the CLIP classifier is , the text encoder is , the prompt predictor is , vessel category The category name is , then the CLIP classifier accepts the input block image The confidence of each category label output is:

[0029] ;

[0030] in, The first block obtained after the original image is divided into Row, No. The tiled image of the columns, for No. The confidence of each category, a block image has confidence level, represents the 2-norm of a vector, Defined as: ;

[0031] Step 2-2-3: Calculate the loss function based on the confidence and category label and train the CLIP classifier.

[0032] As a further improvement of the ship target detection method based on the visual language model: in step 2-2, only the Prompt predictor in the entire CLIP classifier is a trainable module, and the weights of the remaining parts are fixed.

[0033] As a further improvement of the ship target detection method based on the visual language model: in step 2-2-3, the CLIP classifier is trained using the binomial cross entropy loss as the loss function. The binomial cross entropy loss function corresponding to a certain block image is defined as follows:

[0034] .

[0035] As a further improvement of the ship target detection method based on the visual language model: in step 2-3, the Prompt predictor that has been trained in step 2-2 is also fine-tuned with a small learning rate.

[0036] As a further improvement of the ship target detection method based on the visual language model: the detection head adopts the detection head of the YOLO model.

[0037] As a further improvement of the ship target detection method based on the visual language model: the input image is a square; in the block module, for a certain group of block input images, the number of divisions is set to , the input image will be divided into Part, get The input image is divided into blocks.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] 1. The present invention adds a visual language module to the traditional one-stage YOLO target detection algorithm, and uses the excellent generalization performance and understanding ability of the large visual language model to extract the semantic information of the image, thereby assisting the main target detection task, improving the generalization performance of the model and enabling it to have a certain semantic understanding ability.

[0040] 2. The present invention adopts a blocking approach in the visual language module branch. On the one hand, it enables the model to obtain information at different positions of the input image and improves the ability to capture local information. On the other hand, it also improves the multi-scale expression ability of the model and extracts the same multi-scale features as YOLO, laying the foundation for the splicing of the two branch feature maps.

[0041] 3. The present invention adds a Prompt predictor in parallel with the BLIP graph-generated text model in the visual language module, which outputs a debug template Prompt according to the block input image, and then splices the debug template Prompt and the descriptive text output by the BLIP graph-generated text model and inputs them into the CLIP text encoder. Since the BLIP graph-generated text model is trained based on a large amount of general data, it has a poor effect on the ship detection data set in a specific domain. By introducing the Prompt predictor trained based on the ship predictor training set of interest, the output result of the BLIP graph-generated text model can be well corrected.

[0042] 4. The present invention adopts a multi-stage training strategy, which can improve the stability of the training process, better allocate computing resources, and ensure that each branch can reach the optimal state, thereby improving the overall performance of the entire model. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a structural schematic diagram of a ship target detection model based on a visual language model in the present invention;

[0044] Figure 2 It is a structural diagram of the visual language module in the present invention. DETAILED DESCRIPTION

[0045] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments.

[0046] A method for detecting a ship target based on a visual language model, comprising the following steps:

[0047] Step 1: Establish a ship target detection model based on the visual language model.

[0048] like Figure 1 The ship target detection model includes a YOLO feature extraction branch and a visual language module branch, and also includes a detection head.

[0049] The YOLO feature extraction branch includes a YOLO Backbone, which is used to extract a plurality of first feature maps of different scales from an input image.

[0050] In this embodiment, the input image is ,in is the width and height of the image, 3 is the number of color channels, then the three groups of first feature maps output by YOLO Backbone are large-scale feature maps, medium-scale feature maps, and small-scale feature maps, and the sizes of the three groups are , and .

[0051] The visual language module branch includes a block module and a visual language module. The block module is used to divide the input image into multiple groups of block input images according to different segmentation numbers, and each segmentation number corresponds to the size of each first feature map extracted by the YOLO feature extraction branch.

[0052] For a group of block input images, let the number of divisions be ,and (Right now Can be The input image will be divided into two parts along the width and height directions. Part, get The block input images are represented as follows: , .

[0053] In this embodiment, the width / height dimensions of the three first feature maps are , and ,but , the three groups of block input images obtained are: The size is The block input image, The size is The block input image, The size is The block input image.

[0054] Each group of block input images is input into the visual language module to obtain a second feature map with the same scale as the corresponding first feature map.

[0055] Specifically, Figure 2, the visual language module includes a Prompt predictor, a BLIP graph-to-text model and a CLIP text encoder. The Prompt predictor adaptively predicts the debugging template Prompt according to the content and position of each block input image, and the BLIP graph-to-text model generates the corresponding descriptive text according to each block input image. The debugging template Prompt and the descriptive text corresponding to each block input image are spliced ​​and input into the CLIP text encoder to obtain the corresponding block output features. The process can be expressed as:

[0056] ;

[0057] in, is the first block obtained after the input image is divided Row, No. The column-wise block input image, represents the processing of the Prompt predictor, It represents the processing process of the BLIP graph-to-text model. Represents the processing of the CLIP text encoder, For the block input image The corresponding block output features.

[0058] Then, all the block output features of each component block input image are recombined to obtain the second feature map corresponding to the output feature of the component block. The size of the second feature map is In this embodiment, the second feature map also has three dimensions, respectively , and , which correspond to the sizes of the corresponding first feature maps respectively.

[0059] The detection head adopts the detection head of the YOLO model. Each group of mutually corresponding first feature maps and second feature maps are spliced ​​respectively, and then the spliced ​​feature maps of each group are simultaneously input into the detection head to obtain the detection result.

[0060] Step 2: Train the above-mentioned ship target detection model based on the visual language model.

[0061] For the object detection problem in the specific domain of ship detection, it is necessary to perform specific training and fine-tuning on a small dataset of interest. At the same time, in order not to destroy the effect and generalization performance of the large visual language model, this model is trained in a multi-stage manner.

[0062] Step 2-1: Train YOLO Backbone based on the ship detection training set.

[0063] In the first stage, the model is trained according to the traditional YOLO target detection algorithm, so that the YOLO Backbone can extract the key features of the image in the specific domain of the ship detection task. Before training, the weights pre-trained on other larger data sets can be selected as the initial weights of the YOLO Backbone. The ship detection training set is a training set that includes ship target image samples.

[0064] Step 2-2: Train the Prompt predictor.

[0065] The second stage adopts the framework of multi-label classification tasks, ignores the target box in the small dataset and only considers the target category to train the Prompt predictor in the visual language module.

[0066] The specific steps are:

[0067] Step 2-2-1, construct a ship predictor training set, which includes a number of block images and category labels corresponding to the classification images, and the category labels are represented as , is the number of vessel categories, 1 means that the block image is The positive sample of the category is 0, which means that the block image is Negative samples of each category.

[0068] When constructing, first divide several original images into blocks to obtain block images, and then annotate the category labels for each block image: set all element values ​​of the category labels of all block images to 0, traverse the target boxes in the original image, and if the center point of a target box falls into a block image, set the target box corresponding to the first If there are multiple categories, the element value corresponding to the category label of the block image is set to 1.

[0069] Step 2-2-2: Construct a CLIP classifier based on the Prompt predictor. Suppose the processing of the CLIP image encoder of the CLIP classifier is , the text encoder is , the prompt predictor is , vessel category The category name is , then the CLIP classifier accepts the input block image The confidence of each category label output is:

[0070] ;

[0071] in, The first block obtained after the original image is divided into Row, No. The tiled image of the columns, for No. The confidence of each category, a block image has confidence level, represents the 2-norm of a vector, Defined as: .

[0072] Step 2-2-3: Use Binary Cross Entropy Loss as the loss function to train the CLIP classifier. It should be noted that this stage is aimed at training the Prompt predictor, so only the Prompt predictor is a trainable module in the entire CLIP classifier, and the weights of the rest are fixed.

[0073] The binomial cross entropy loss function corresponding to a block image is defined as follows:

[0074] .

[0075] Step 2-3: Use the trained YOLO Backbone and Prompt predictor to train the detection head.

[0076] Since YOLO Backbone already has good domain-specific image feature extraction capabilities, the BLIP graph-to-text model and CLIP text encoder have been fully trained in a large number of data sets, the above networks do not need to be trained or fine-tuned at this stage, and only the network parameters of the final detection head can be updated with gradients.

[0077] Furthermore, in order to adapt to new tasks, the Prompt predictor trained in the previous stage can also be fine-tuned with a small learning rate at this stage.

[0078] Step 3: Input the ship image as an input image into the trained ship target detection model based on the visual language model to obtain the detection result.

[0079] It should be noted that it is obvious to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments, and that the present invention can be implemented in other specific forms without departing from the spirit or essential features of the present invention. The scope of the present invention is defined by the claims rather than the above description.

Claims

1. A method for detecting ship targets based on a visual language model, characterized in that the steps include: Step 1: Establish a ship target detection model based on the visual language model; The ship target detection model includes a YOLO feature extraction branch and a visual language module branch, and also includes a detection head; The YOLO feature extraction branch includes a YOLO Backbone, which is used to extract a plurality of first feature maps of different scales from an input image; The visual language module branch includes a block module and a visual language module; the block module is used to divide the input image into multiple groups of block input images according to different segmentation numbers, and each segmentation number corresponds to the size of each first feature map extracted by the YOLO feature extraction branch; Each group of block input images is input into the visual language module to obtain a second feature map with the same scale as the corresponding first feature map; Each group of mutually corresponding first feature maps and second feature maps are spliced ​​respectively, and then each group of spliced ​​feature maps is simultaneously input into the detection head to obtain the detection result; Step 2: training the above-mentioned ship target detection model based on the visual language model; Step 3: Input the ship image as an input image into the trained ship target detection model based on the visual language model to obtain the detection result.

2. The method for detecting ship targets based on a visual language model according to claim 1, characterized in that: The visual language module includes a Prompt predictor, a BLIP graph-to-text model, and a CLIP text encoder; The prompt predictor adaptively predicts the debugging template prompt according to the content and position of each block input image, and the BLIP graph-to-text model generates the corresponding descriptive text according to each block input image. The debugging template Prompt and the descriptive text corresponding to each block input image are concatenated and input into the CLIP text encoder to obtain the corresponding block output features. The process is expressed as: ； in, is the first block obtained after the input image is divided Row, No. The column-wise block input image, represents the processing of the Prompt predictor, It represents the processing process of the BLIP graph-to-text model. Represents the processing of the CLIP text encoder, For the block input image The corresponding block output features; Then, all the block output features of each component block input image are recombined to obtain a second feature map corresponding to the component block output feature.

3. The method for detecting ship targets based on a visual language model according to claim 1, characterized in that: In step 2, a multi-stage approach is used for training: Step 2-1: Train YOLO Backbone based on the ship detection training set; Step 2-2, train the Prompt predictor; Step 2-3: Use the trained YOLO Backbone and Prompt predictor to train the detection head.

4. The method for detecting a ship target based on a visual language model according to claim 3, characterized in that: In step 2-1, before training YOLO Backbone, the pre-trained weights are used as the initial weights of YOLO Backbone.

5. The method for detecting ship targets based on a visual language model as claimed in claim 3, characterized in that: The specific process of step 2-2 is as follows: Step 2-2-1, construct a ship predictor training set, which includes a number of block images and category labels corresponding to the classification images, and the category labels are represented as , is the number of vessel categories, 1 means that the block image is The positive sample of the category is 0, which means that the block image is Negative samples of categories; When constructing, first divide several original images into blocks to obtain block images, and then annotate the category labels for each block image: set all element values ​​of the category labels of all block images to 0, traverse the target boxes in the original image, and if the center point of a target box falls into a block image, set the target box corresponding to the first If there are multiple categories, the element value corresponding to the category label of the block image is set to 1; Step 2-2-2: Construct a CLIP classifier based on the Prompt predictor. Suppose the processing of the CLIP image encoder of the CLIP classifier is , the text encoder is , the prompt predictor is , vessel category The category name is , then the CLIP classifier accepts the input block image The confidence of each category label output is: ； in, The first block obtained after the original image is divided into Row, No. The tiled image of the columns, for No. The confidence of each category, a block image has confidence level, represents the 2-norm of a vector, Defined as: ; Step 2-2-3: Calculate the loss function based on the confidence and category label and train the CLIP classifier.

6. The method for detecting ship targets based on a visual language model as claimed in claim 5, characterized in that: In step 2-2, only the Prompt predictor is a trainable module in the entire CLIP classifier, and the weights of the rest are fixed.

7. The method for detecting ship targets based on a visual language model as claimed in claim 5, characterized in that: In step 2-2-3, the CLIP classifier is trained using binomial cross entropy loss as the loss function. The binomial cross entropy loss function corresponding to a block image is defined as follows: 。 8. The method for detecting a ship target based on a visual language model as claimed in claim 3, characterized in that: In step 2-3, the Prompt predictor that has been trained in step 2-2 is also fine-tuned with a small learning rate.

9. The method for detecting ship targets based on a visual language model according to any one of claims 1 to 8, characterized in that: The detection head adopts the detection head of the YOLO model.

10. The method for detecting ship targets based on a visual language model according to any one of claims 1 to 8, characterized in that: The input image is a square. In the block module, for a group of block input images, the number of divisions is , the input image will be divided into Part, get The input image is divided into blocks.

Images