A target detection evaluation method based on a multi-modal model

Abstract

The present application relates to the technical field of image processing and target detection, and particularly relates to a target detection evaluation method based on a multi-modal model, which comprises the following steps: constructing a test image set, and fine-tuning and calibrating a multi-modal model group through the test image set; inputting the test image set and an image editing instruction set into the multi-modal model group to generate an edited image set; adding the edited image set to the test image set to construct a new test image set, and inputting the new test image set into a model under test to obtain a first inference result; inputting the new test image set, a test instruction set and the first inference result into the multi-modal model group to obtain a second inference result; inputting the first inference result and the second inference result into an inference result evaluation module to obtain an evaluation result; and repeating the above steps to average the evaluation result to obtain a final evaluation result. The present application improves the test efficiency of a target detection algorithm while ensuring the standardization and consistency of test data.

Description

Technical Field

[0001] This invention relates to the field of image processing and target detection technology, and in particular to a target detection evaluation method based on a multimodal model. Background Technology

[0002] Object detection is a common and important algorithmic task in the field of computer vision, and it has been widely applied in smart cities, e-commerce, and industrial quality inspection. Algorithm evaluation is a crucial stage in the experimentation, optimization, and production of object detection models. Scientific and comprehensive evaluation can not only quantify the model's performance metrics and control the quality of algorithm modules, but also identify algorithm weaknesses in a timely manner, improving the efficiency of optimization and iteration. However, existing algorithm evaluation methods heavily rely on the quality and quantity of labeled sample datasets, while manual annotation is costly, and the quality, standardization, and consistency of manually labeled data are difficult to guarantee.

[0003] For example, patent application CN114036036A discloses an automated test and analysis system, method, and terminal device based on automatic annotation, which relates to the field of image recognition technology. The system includes an automatic annotation module, a guidance module A, a guidance module B, a guidance module C, a guidance module D, an automatic summarization module, an automatic analysis module, and an analysis platform. The automatic annotation module processes the video and splits it into image sets frame by frame. The guidance module guides various configurations in the automated testing process. The automatic summarization module summarizes the data according to the summarization dimension, summary content, summary result output format, and summary result output path. The automatic analysis module automatically analyzes the annotation results.

[0004] A Chinese patent with authorization announcement number CN111027589B discloses a multi-score target detection algorithm evaluation system and method, including the following modules: an algorithm input module for inputting the algorithm to be tested; a test set preparation module for selecting a test set with the types of targets to be tested according to the algorithm to be tested; a comprehensive scoring module for weighting image quality score, image resolution score, target quality score, and target size score to obtain an overall comprehensive score; an application scenario scoring module for weighting scene image resolution score, scene target quality score, and scene target size score to obtain an application scenario score; and a free scoring module for generating corresponding test sets according to user testing requirements to complete targeted evaluation.

[0005] All of the above patents suffer from the problems described in the background section: existing target detection evaluation methods suffer from low coverage in verifying the generalization and robustness of the model due to limitations in the quantity and diversity of sample data. Furthermore, the cost of sample data annotation is high, and the quality and consistency of the annotated data also face significant challenges. Summary of the Invention

[0006] To overcome the defects and shortcomings of existing technologies, this invention provides a target detection evaluation method based on a multimodal model. By designing a test instruction set, it enables interactive editing of test data with a multimodal model group, automating the testing process while improving the testing capabilities in terms of model robustness and generalization.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: Construct a test image set and use the test image set to fine-tune and calibrate the multimodal model group; Input the test image set and image editing instruction set into the multimodal model group to generate the edited image set; The edited image set is added to the test image set to construct a new test image set. The new test image set is then input into the model under test to obtain the first inference result. The new test image set, test instruction set, and first inference result are input into the multimodal model group to obtain the second inference result; Input the first reasoning result and the second reasoning result into the reasoning result evaluation module to obtain the evaluation result; Repeat the above steps to average the evaluation results and obtain the final evaluation result.

[0008] As a preferred technical solution, the image editing instruction set is used to implement image interactive editing functions with text instructions, including guiding the multimodal model group to complete the attribute editing of a specified target detection instance and increasing or decreasing the number of target detection instances to be tested.

[0009] As a preferred technical solution, the multimodal model group is jointly composed of a language-visual model, a detection model, a segmentation model, and an image-oriented editing model. The language-visual model includes the CLIP model, the detection model includes the DINO model, the segmentation model includes the SAM model, and the image-oriented editing model includes the inpainting model.

[0010] As a preferred technical solution, the model under test is a target detection model to be tested and verified.

[0011] As a preferred technical solution, the first inference result includes a target detection box and a detection confidence score, and the first inference result is in the following format. ,in To detect the coordinates of the top left corner of the target, To detect the coordinates of the bottom right corner of the target, The confidence level of the target being detected.

[0012] As a preferred technical solution, the test instruction set is a text-based test instruction set that guides the multimodal model group to complete test inference and targeted image re-editing. The template for the test instruction set is as follows: ; Among them, test instructions This indicates that after guiding the multimodal model group to complete the target detection for the specified test category, the data is then sent to the segmentation model for fine-tuning of the target detection bounding boxes. (Test instruction) This indicates a test command that instructs the multimodal model group to complete the attribute editing of a specified target detection instance. This indicates how to increase or decrease the number of target detection instances to be tested in the multimodal model group.

[0013] As a preferred technical solution, the second reasoning result has the same format as the first reasoning result.

[0014] As a preferred technical solution, the evaluation results are evaluation indicators for the target detection task, including accuracy, recall, and mean precision.

[0015] As a preferred technical solution, the calculation process of the final evaluation result is expressed by the following formula: ; In the formula Indicates the first The results of this evaluation Indicates the number of repetitions. This indicates the final evaluation result.

[0016] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a target detection and evaluation method based on a multimodal model.

[0017] The present invention provides a controller comprising a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to implement a target detection and evaluation method based on a multimodal model.

[0018] Compared with the prior art, the present invention has the following advantages and beneficial effects: (1) The present invention designs an automated target detection algorithm evaluation process. In this process, the automated interaction with the multimodal model group is completed through a text-based test instruction set, so as to realize the automatic labeling and automatic evaluation of test data. This process does not require manual intervention, which can significantly improve test efficiency and ensure the standardization and consistency of test benchmark data.

[0019] (2) The multimodal model group in this invention has text-image multimodal interaction function and image-oriented editing generation function with semantic prompts. After adding the edited image generated by orientation to the test image set, the evaluation ability of the model in terms of generalization and robustness can be significantly improved, and the test coverage can be improved. Attached Figure Description

[0020] Other features, objects, and advantages of the invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the overall process of a target detection and evaluation method based on a multimodal model according to the present invention; Figure 2 This is a test schematic diagram of the image editing instruction set described in the target detection evaluation method based on a multimodal model of the present invention; Figure 3 This is a schematic diagram of the intelligent recommendation system based on streaming data according to the present invention. Detailed Implementation

[0021] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations thereof. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0022] Example 1

[0023] like Figure 1 As shown, this embodiment provides a target detection evaluation method based on a multimodal model, which specifically includes the following steps: S1: Construct a test image set and fine-tune and calibrate the multimodal model group using the test image set; The decoder of the SAM segmentation model can be fine-tuned by training a small number of labeled test samples, thereby improving the detection and segmentation performance of the fine-tuned SAM segmentation model on a customized test image set compared to the original SAM segmentation model.

[0024] S2: Input the test image set and image editing instruction set into the multimodal model group to generate the edited image set; The image editing instruction set is used to implement interactive image editing functions with text commands, including guiding the multimodal model group to complete attribute editing of specified target detection instances and increasing or decreasing the number of target detection instances to be tested, such as... Figure 2 As shown, for example, increasing the density of fireworks in the figure can improve the test results in terms of model robustness; The multimodal model group consists of a language-visual model, a detection model, a segmentation model, and an image-oriented editing model. The language-visual model includes the CLIP model, the detection model includes the DINO model, the segmentation model includes the SAM model, and the image-oriented editing model includes the inpainting model.

[0025] S3: Add the edited image set to the test image set to construct a new test image set, and input the new test image set into the model under test to obtain the first inference result; The model under test is the target detection model to be tested and verified. After receiving the input test image, it performs target detection inference and obtains the first inference result. The first inference result will serve as a location cue for the multimodal model group and as the data to be detected for the inference result evaluation module; The first inference result includes the target detection bounding box and the detection confidence score. The first inference result is in the following format. ,in To detect the coordinates of the top left corner of the target, To detect the coordinates of the bottom right corner of the target, The confidence level of the target being detected.

[0026] S4: Input the new test image set, test instruction set and first inference result into the multimodal model group to obtain the second inference result; The test instruction set consists of text-based test instructions that guide the multimodal model team in completing test inference and targeted image re-editing. The template for the test instruction set is as follows: ; Among them, test instructions This indicates that after guiding the multimodal model group to complete the target detection for the specified test category, the data is then sent to the segmentation model for fine-tuning of the target detection bounding boxes. (Test instruction) This indicates how to guide a multimodal model group to complete attribute editing for a specified object detection instance. For example, replacing the color of a pedestrian wearing yellow clothing in the original image with red can improve the model's generalization performance in tests. (Test instructions) This indicates how to increase or decrease the number of target detection instances to be tested in the multimodal model group.

[0027] S5: Input the first reasoning result and the second reasoning result into the reasoning result evaluation module to obtain the evaluation result; The second reasoning result has the same format as the first reasoning result; The evaluation results are the evaluation metrics for the target detection task, including accuracy, recall, and mean precision.

[0028] S6: Repeat the above steps to average the evaluation results and obtain the final evaluation result; The calculation process for the final evaluation result is expressed by the following formula: ; In the formula Indicates the first The results of this evaluation Indicates the number of repetitions. This indicates the final evaluation result.

[0029] Example 2

[0030] like Figure 3 As shown, this embodiment provides a target detection and evaluation system 20 based on a multimodal model, including: Test image set construction module 21 is used to construct a test image set and to fine-tune and calibrate the multimodal model group using the test image set. The image set generation module 22 is used to input the test image set and the image editing instruction set into the multimodal model group to generate the edit image set; The test model reasoning module 23 is used to add the edited image set to the test image set to construct a new test image set, and input the new test image set into the test model to obtain the first reasoning result; The multimodal model group inference module 24 is used to input the new test image set, test instruction set and first inference result into the multimodal model group to obtain the second inference result; The reasoning result evaluation module 25 is used to input the first reasoning result and the second reasoning result into the reasoning result evaluation module to obtain the evaluation result; The evaluation result output module 26 is used to repeat the above steps and calculate the average of the evaluation results to obtain the final evaluation result.

[0031] In this embodiment, the edited image set generation module 22 is used to input the test image set and the image editing instruction set into the multimodal model group to generate the edited image set, wherein, The image editing instruction set is used to implement interactive image editing functions with text instructions, including guiding the multimodal model group to complete the attribute editing of a specified target detection instance and increasing or decreasing the number of target detection instances to be tested; The multimodal model group consists of a language-visual model, a detection model, a segmentation model, and an image-oriented editing model. The language-visual model includes the CLIP model, the detection model includes the DINO model, the segmentation model includes the SAM model, and the image-oriented editing model includes the inpainting model.

[0032] In this embodiment, the test model inference module 23 is used to add the edited image set to the test image set to construct a new test image set, and input the new test image set into the test model to obtain a first inference result, wherein the test model is a target detection model to be tested and verified; the first inference result includes target detection boxes and detection confidence.

[0033] In this embodiment, the multimodal model group inference module 24 is used to input a new test image set, a test instruction set, and a first inference result into the multimodal model group to obtain a second inference result. The test instruction set consists of text-based test instructions that guide the multimodal model group to complete test inference and targeted image re-editing.

[0034] In this embodiment, the reasoning result evaluation module 25 is used to input the first reasoning result and the second reasoning result into the reasoning result evaluation module to obtain the evaluation result, wherein the evaluation result is the evaluation index of the target detection task, including accuracy, recall and mean precision.

[0035] In this embodiment, the evaluation result output module 26 is used to repeat the above steps and average the evaluation results to obtain the final evaluation result. The calculation process of the final evaluation result is expressed by the following formula: ; In the formula Indicates the first The results of this evaluation Indicates the number of repetitions. This indicates the final evaluation result.

[0036] The parameters and steps for implementing the corresponding functions of each unit module in the target detection evaluation system based on a multimodal model of the present invention described above can be referred to the parameters and steps in the embodiments of the target detection evaluation method based on a multimodal model described above, and will not be repeated here.

[0037] Example 3

[0038] This invention provides a computer-readable storage medium storing a computer program. When executed by a processor, this computer program implements the aforementioned target detection evaluation method based on a multimodal model. It should be noted that all computer programs for the multimodal model-based target detection evaluation method are implemented using Python. The test image set construction module, image set generation editing module, tested model inference module, multimodal model group inference module, inference result evaluation module, and evaluation result output module are all controlled by a remote server. The remote server has an Intel Xeon Gold 5118 CPU, an NVIDIA GTX 2080Ti 11GB GPU, an Ubuntu 18.04.2 operating system, PyTorch 1.7.0 deep learning framework, CUDA version 10.2, and uses cuDNN 7.6.5 for accelerated inference. The Intel Xeon Gold 5118 includes a memory and a processor. The memory stores the computer program, and the processor executes the computer program, enabling the Intel Xeon Gold 5118 to implement the multimodal model-based target detection evaluation method.

[0039] Those skilled in the art will know that this invention can be implemented as a system, method, or computer program product.

[0040] Therefore, this disclosure can be implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the invention can also be implemented as a computer program product in one or more computer-readable media containing computer-readable program code.

[0041] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.

[0042] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A target detection evaluation method based on a multimodal model set, characterized in that, Includes the following steps: Construct a test image set and use the test image set to fine-tune and calibrate the multimodal model group; Input the test image set and image editing instruction set into the multimodal model group to generate the edited image set; The edited image set is added to the test image set to construct a new test image set. The new test image set is input into the model under test to obtain the first inference result. The multimodal model group is jointly composed of a language-visual model, a detection model, a segmentation model and an image-oriented editing model. The language-visual model includes the CLIP model, the detection model includes the DINO model, the segmentation model includes the SAM model, and the image-oriented editing model includes the inpainting model. The new test image set, test instruction set, and first inference result are input into the multimodal model group to obtain the second inference result; Input the first reasoning result and the second reasoning result into the reasoning result evaluation module to obtain the evaluation result; Repeat the above steps to average the evaluation results and obtain the final evaluation result.

2. The target detection and evaluation method based on a multimodal model set according to claim 1, characterized in that, The image editing instruction set is used to implement interactive image editing functions with text instructions, including guiding the multimodal model group to complete the attribute editing of a specified target detection instance and increasing or decreasing the number of target detection instances to be tested.

3. The target detection and evaluation method based on a multimodal model set according to claim 1, characterized in that, The model under test is a target detection model to be tested and verified.

4. The target detection and evaluation method based on a multimodal model set according to claim 1, characterized in that, The first inference result includes the target detection box and the detection confidence.

5. The target detection and evaluation method based on a multimodal model set according to claim 1, characterized in that, The test instruction set consists of text-based test instructions that guide the multimodal model group to complete test inference and targeted image re-editing.

6. The target detection and evaluation method based on a multimodal model set according to claim 1, characterized in that, The evaluation results are the evaluation indicators for the target detection task, including accuracy, recall, and mean precision.

7. The target detection and evaluation method based on a multimodal model set according to claim 1, characterized in that, The calculation process for the final evaluation result is expressed by the following formula: ； In the formula Indicates the first The results of this evaluation Indicates the number of repetitions. This indicates the final evaluation result.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the target detection evaluation method based on a multimodal model group as described in any one of claims 1-7.

9. A controller comprising a memory and a processor, the memory for storing a computer program, characterized in that, When the processor executes the computer program, it implements the target detection and evaluation method based on a multimodal model set as described in any one of claims 1-7.

Images