Open target detection model optimization method based on multi-modal large language model

Abstract

The invention discloses an open target detection model optimization method based on a multi-modal large language model. The method comprises the following steps of (1) describing an environment in a picture by utilizing a Vision Language Model (VLM), and (2) constructing an environment in the picture by utilizing the VLM; 2) using text description of a picture environment to assist training of open target detection; and 3) in the reasoning process, taking a picture environment as an output label for conditional optimization open target detection. Through the optimization scheme, the performance of the original model can be maintained, and meanwhile, the recognition accuracy of the target label is improved by utilizing the open environment description capability of the multi-modal large language model. The method is suitable for research and practice in the visual language multi-mode field.

Description

Technical Field

[0001] The present invention relates to a target detection model optimization method, and more specifically to an open target detection model optimization method based on a multimodal large language model. Background Art

[0002] In object detection models, the accuracy of object labels is one of the key factors affecting detection results. However, in practical applications, objects that do not match the scene environment often appear. For example, it is relatively rare for a bus to appear in a kitchen environment, while fruits are more likely to appear.

[0003] Therefore, in the current prior art, there is an invention patent with patent number 202311051883.4, named A method for optimizing target detection labels based on image and text multimodality. The invention patent discloses that the predicted labels with the highest confidence of each target are compared with the correct labels, and then the predicted labels with the highest confidence that are the same as the correct labels are selected, and the predicted labels with the second highest confidence are set as wrong labels to achieve label optimization. However, the above optimization process does not perceive and understand the target and the current environment, so the accuracy of the labels optimized by this method is not very high. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an open target detection model optimization method based on a multimodal large language model to solve the problem that in the open target detection model, the model has limited perception and understanding of the target and the current environment, resulting in incorrect target labels.

[0005] To achieve the above object, the present invention provides the following technical solution: a method for optimizing an open object detection model based on a multimodal large language model, characterized in that it comprises the following steps:

[0006] Step 1: Use a multimodal large language model to describe the environment in the image;

[0007] Step 2: Use the text description of the image environment obtained in step 1 to assist in the training of open object detection;

[0008] Step 3: Then, during inference, the image context is used as a condition to optimize the output labels of open object detection.

[0009] As a further improvement of the present invention, the specific method of using the multimodal large language model to describe the environment in the picture in step one is to first perform environmental processing on the pictures in the training set as a pre-labeling of data processing, and then use the generation capability of the multimodal large language model to describe, reason and answer questions for the visual pictures.

[0010] As a further improvement of the present invention, the specific steps of using the text description of the image environment to assist the training of open object detection in step 2 are as follows:

[0011] Step 21: randomly select a batch of pictures and their paired text descriptions from the picture description database, and the batch size B is generally between 100 and 10,000;

[0012] Step 22: Use V to encode the images in all batches to obtain {v_i_1,...v_i_B};

[0013] Step 2 and 3: Use the pooling module P to extract features from {v_i_1,...v_i_B}. The extracted area range is the full image coordinates (0,0,W,H). W and H are the image sizes. Get {f_i_1,...f_i_B};

[0014] Step 24: Use the text module L to encode all the texts t enhanced in the above steps to obtain {f_t_1,...f_t_B};

[0015] Step 25: Calculate the loss function to maximize the probability of matching text and image features.

[0016] As a further improvement of the present invention, the specific steps of calculating the loss function in step 25 are as follows:

[0017] Step 251, calculate L1 using the following formula:

[0018]

[0019] Step 252, calculate L2 using the following formula:

[0020]

[0021] Step 253, calculate the final optimized loss function as (L1+L2) / 2;

[0022] Step 254: The training iteration number ends when it reaches the maximum target number.

[0023] As a further improvement of the present invention, the specific method of describing the visual picture in step one is to perceive the picture environment using the prompt "Describe the environment of the picture in the picture".

[0024] The beneficial effect of the present invention is that the accuracy of the target detection model can be significantly improved by using the perception of the image environment as an auxiliary condition for target detection. While maintaining the performance of the original model, the open environment description capability of the multimodal large language model can be used to improve the recognition accuracy of the target label. This method is suitable for research and practice in the field of multimodal visual language. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 A schematic diagram of the example described in step 1. DETAILED DESCRIPTION

[0026] The present invention will be further described below in detail with reference to the embodiments shown in the accompanying drawings.

[0027] Reference Figure 1 As shown, a method for optimizing an open object detection model based on a multimodal large language model in this embodiment includes the following steps:

[0028] S1. Use a multimodal large language model (VisionLargeLanguageModel, VLM) to describe the environment in the picture

[0029] In recent years, the development of VLM has achieved rapid technological progress. Using the generation ability of VLM, visual images can be described, reasoned and questioned. However, this ability is currently limited in combination with other types of models. Therefore, the method proposed in this scheme mainly uses the ability of VLM to enhance the recognition effect in the field of target detection. The VLM model here can be a closed source model such as Chatgpt, or an open source model such as Llava. In the first step, the images in the training set need to be processed by environment as a pre-labeling of data processing. For example, the prompt "Describe the environment of the picture in the picture" can be used to perceive the picture environment. The description of the picture can be used as an enhancement of the picture information.

[0030] S2. Using text descriptions of image environments to assist in training open object detection

[0031] Based on the environment construction of the training data set, the model training task can be partially adjusted. For the open target detection model, the general text query is used as the target pool for text recognition. This query generally comes from a large vocabulary, which contains possible targets. The target to be recognized may be a word in the vocabulary, or a target text label entered using a prompt. Using the text of the environment, each word t_0 in the text and vocabulary can be enhanced to t, which can enhance the understanding of the environment. On this basis, the target text label is a combination of the target and the environment description. The combination is: environment + target. Taking the above environment as an example, recognize the mouse in the picture, and the text label becomes: This picture shows the environment of an office desk. There is a keyboard, a laptop, a coffee cup, a mouse, a notebook and some paper on the desktop, and the wires are a bit messy. Overall, this is a typical desktop environment for work or study. Identify the mouse in the picture. On this basis, the steps of model pre-training become:

[0032] A batch of images and their paired text descriptions are randomly selected from the image description database, and the batch size B is generally between 100 and 10,000.

[0033] Use V to encode the images in all batches and obtain {v_i_1,...v_i_B}.

[0034] Use the pooling module P to extract features from {v_i_1,...v_i_B}. The extracted area range is the full image coordinates (0,0,W,H). W and H are the image sizes. Get {f_i_1,...f_i_B}

[0035] Use the text module L to encode all the texts t enhanced in the above steps to obtain {f_t_1,...f_t_B}.

[0036] The loss function is calculated using the following function to maximize the probability of matching text and image features:

[0037]

[0038] The final optimized loss function is (L1+L2) / 2

[0039] The number of training iterations ends when the maximum target number is reached, for example, 1 million iterations.

[0040] S3. During the inference process, the environment of the image is used as a condition to optimize the output label of open object detection.

[0041] The implementation process of this embodiment is as follows:

[0042] 1. Use the VLM model to describe the environment envi for the picture to be recognized;

[0043] 2. During reasoning, each word t_0 in the text and vocabulary is enhanced to t(t_0+envi=t);

[0044] 3. Use the normal object detection process for inference.

[0045] In summary, the present embodiment is an open target detection model optimization method based on a multimodal large language model. The method comprises the following steps: 1) using a multimodal large language model (VisionLargeLanguageModel, VLM) to describe the environment in the picture; 2) using the text description of the picture environment to assist in the training of open target detection; 3) during the reasoning process, the picture environment is used as a condition to optimize the output label of the open target detection. Through this optimization scheme, the open environment description capability of the multimodal large language model can be used to improve the recognition accuracy of the target label while maintaining the performance of the original model. This method is suitable for research and practice in the field of multimodal visual language.

[0046] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for optimizing an open object detection model based on a multimodal large language model, characterized in that: The steps include: Step 1: Use a multimodal large language model to describe the environment in the image; Step 2: Use the text description of the image environment obtained in step 1 to assist in the training of open object detection; Step 3: Then, during inference, the image context is used as a condition to optimize the output labels of open object detection.

2. The open object detection model optimization method based on a multimodal large language model according to claim 1, characterized in that: The specific method of using the multimodal large language model to describe the environment in the picture in step 1 is to first perform environmental processing on the pictures in the training set as a pre-labeling for data processing, and then use the generation capability of the multimodal large language model to describe, reason and answer questions for the visual pictures.

3. The open object detection model optimization method based on a multimodal large language model according to claim 1 or 2, characterized in that: The specific steps of using the text description of the image environment to assist the training of open object detection in step 2 are as follows: Step 21: randomly select a batch of pictures and their paired text descriptions from the picture description database, and the batch size B is generally between 100 and 10,000; Step 22: Use V to encode the images in all batches to obtain {v_i_1,...v_i_B}; Step 2 and 3: Use the pooling module P to extract features from {v_i_1,...v_i_B}. The extracted area range is the full image coordinates (0,0,W,H). W and H are the image sizes. Get {f_i_1,...f_i_B}; Step 24: Use the text module L to encode all the texts t enhanced in the above steps to obtain {f_t_1,...f_t_B}; Step 25: Calculate the loss function to maximize the probability of matching text and image features.

4. The open object detection model optimization method based on a multimodal large language model according to claim 3 is characterized in that: The specific steps for calculating the loss function in step 25 are as follows: Step 251, calculate L1 using the following formula: Step 252, calculate L2 using the following formula: Step 253, calculate the final optimized loss function as (L1+L2) / 2; Step 254: The training iteration number ends when it reaches the maximum target number.

5. The open object detection model optimization method based on a multimodal large language model according to claim 2, characterized in that: The specific method of describing the visual image in step one is to perceive the image environment using the prompt "Describe the environment of the picture in the picture".

Images