Target detection optimization method and system, electronic equipment and storage medium

Abstract

The invention relates to the technical field of computer vision, and particularly discloses a target detection optimization method and system, electronic equipment and a storage medium, and the method comprises the steps: inputting a to-be-detected image into a trained target detection model, and obtaining a target detection frame of the to-be-detected image and a first confidence coefficient of the target detection frame; generating a background description text of the to-be-detected image by using a CO-STAR framework, and inputting the target detection frame and the background description text into a trained multi-modal large language model to obtain a second confidence coefficient of the target detection frame; and carrying out confidence coefficient fusion on the first confidence coefficient and the second confidence coefficient of the target detection frame to obtain a target confidence coefficient of the target detection frame. According to the method, the target detection result is rechecked by using the multi-modal large language model, new confidence judgment is given in a high-interpretability mode, the precision of an existing detection framework is remarkably improved, and thus optimization of the target detection result is realized.

Description

Background Art

[0002] At present, object detection models represented by the YOLO series are widely deployed in the industry, and many mature training frameworks and related data sets have been accumulated. However, this type of closed-set model lacks sufficient tuning capabilities for easily confused objects in complex detection scenarios.

[0003] First, the existing closed-set target detection models show obvious deficiencies when facing easily confused targets in complex detection scenarios. Due to the limitations of its model architecture, it lacks compatibility with these special cases, resulting in frequent missed detections and false detections, which seriously affects the accuracy and reliability of the detection results. Secondly, although open-set target detection models such as Grounding DINO have expanded the scope of detection to a certain extent, they still have a gap in detection accuracy compared with closed-set detection models and cannot meet the needs of high-precision detection in practical applications. Furthermore, the tuning of traditional target detection models often requires a large amount of data support, which not only increases the cost of data collection and processing, but also makes the model tuning process cumbersome and time-consuming. When faced with new false detection and missed detection scenarios, it is difficult for existing models to adapt and adjust quickly and effectively, resulting in the continuous optimization of detection results being hindered.

[0004] Therefore, it is urgent to provide a technical solution to solve the above problems. Summary of the invention

[0005] In order to solve the above technical problems, the present invention provides a target detection optimization method, system, electronic device and storage medium.

[0006] In a first aspect, the present invention provides a target detection optimization method, the technical solution of the method is as follows:

[0007] S1. Inputting the image to be tested into a trained target detection model to obtain a target detection frame of the image to be tested and a first confidence of the target detection frame;

[0008] S2. Generate background description text of the image to be tested using the CO-STAR framework, and input the target detection frame and the background description text into a trained multimodal large language model to obtain a second confidence level of the target detection frame;

[0009] S3. Confidence fusion is performed on the first confidence level and the second confidence level of the target detection frame to obtain the target confidence level of the target detection frame.

[0010] The beneficial effects of a target detection optimization method of the present invention are as follows:

[0011] The method of the present invention utilizes a multimodal large language model to recheck the target detection results, and provides new confidence judgments in a highly interpretable manner, thereby significantly improving the accuracy of the existing detection framework, thereby optimizing the target detection results.

[0012] Based on the above solution, a target detection optimization method of the present invention can also be improved as follows.

[0013] In an optional manner, the method further includes:

[0014] Based on multiple training samples and the background description texts corresponding to the training samples, an instruction fine-tuning dataset for training a multimodal large language model is generated;

[0015] Based on the instruction fine-tuning data set, the local low-rank adaptation matrix included in the multimodal large language model is trained to obtain the trained multimodal large language model.

[0016] In an optional manner, step S3 includes:

[0017] According to the confidence fusion formula, the confidence fusion is performed by combining the first confidence and the second confidence of the target detection frame to obtain the target confidence of the target detection frame; wherein the confidence fusion formula is: final =w1·T1+w2·T2; T final is the target confidence, w1 is the weight corresponding to the first confidence, w2 is the weight corresponding to the second confidence, T1 is the first confidence, T2 is the second confidence, w1+w2=1.

[0018] In an optional manner, the method further includes:

[0019] S4. When the error value between the review result of the image to be tested and the target confidence is greater than a threshold, the instruction fine-tuning data set of the trained multimodal large language model is adjusted and trained to obtain an optimized multimodal large language model, and the optimized multimodal large language model is used as the trained multimodal large language model, and the process returns to execute S2.

[0020] In a second aspect, the present invention provides a target detection optimization system, the technical solution of which is as follows:

[0021] It includes: a first generation module, a second generation module and an optimization module;

[0022] The first generating module is used to: input the image to be tested into the trained target detection model to obtain the target detection frame of the image to be tested and the first confidence of the target detection frame;

[0023] The second generation module is used to: generate background description text of the image to be tested by using the CO-STAR framework, and input the target detection frame and the background description text into the trained multimodal large language model to obtain a second confidence of the target detection frame;

[0024] The optimization module is used to: perform confidence fusion on the first confidence level and the second confidence level of the target detection frame to obtain the target confidence level of the target detection frame.

[0025] The beneficial effects of a target detection optimization system of the present invention are as follows:

[0026] The system of the present invention utilizes a multimodal large language model to recheck target detection results, and provides new confidence judgments in a highly interpretable manner, thereby significantly improving the accuracy of existing detection frameworks, thereby optimizing target detection results.

[0027] Based on the above solution, the target detection optimization system of the present invention can also be improved as follows.

[0028] In an optional manner, the method further includes: a training module; the training module is used to:

[0029] Based on multiple training samples and the background description texts corresponding to the training samples, an instruction fine-tuning dataset for training a multimodal large language model is generated;

[0030] Based on the instruction fine-tuning data set, the local low-rank adaptation matrix included in the multimodal large language model is trained to obtain the trained multimodal large language model.

[0031] In an optional manner, the optimization module is specifically used to:

[0032] According to the confidence fusion formula, the confidence fusion is performed by combining the first confidence and the second confidence of the target detection frame to obtain the target confidence of the target detection frame; wherein the confidence fusion formula is: final =w1·T1+w2·T2; T final is the target confidence, w1 is the weight corresponding to the first confidence, w2 is the weight corresponding to the second confidence, T1 is the first confidence, T2 is the second confidence, w1+w2=1.

[0033] In an optional manner, the method further includes: a review module; the review module is used to:

[0034] When the error value between the review result of the image to be tested and the target confidence is greater than a threshold, the instruction fine-tuning data set of the trained multimodal large language model is adjusted and trained to obtain an optimized multimodal large language model, and the optimized multimodal large language model is used as the trained multimodal large language model, and the second generation module is returned to be called.

[0035] In a third aspect, a technical solution of an electronic device of the present invention is as follows:

[0036] The invention comprises a memory, a processor and a program stored in the memory and running on the processor, and the processor implements the steps of the target detection optimization method of the present invention when executing the program.

[0037] In a fourth aspect, the present invention provides a computer-readable storage medium having the following technical solution:

[0038] Instructions are stored in the computer-readable storage medium. When the computer-readable storage medium reads the instructions, the computer-readable storage medium executes the steps of the target detection optimization method of the present invention.

[0039] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented according to the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the accompanying drawings. In the accompanying drawings:

[0041] Figure 1 A schematic diagram of a flow chart of an embodiment of a target detection optimization method of the present invention;

[0042] Figure 2 Schematic diagram of the principle optimized for target detection;

[0043] Figure 3 This is a diagram of the training process of a multimodal large language model;

[0044] Figure 4 A schematic diagram of the structure of an embodiment of a target detection optimization system of the present invention;

[0045] Figure 5 The figure is a schematic structural diagram of an embodiment of an electronic device of the present invention. DETAILED DESCRIPTION

[0046] Exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0047] Figure 1 A flow chart of an embodiment of a target detection optimization method provided by the present invention is shown, and the target detection optimization method can be executed by an electronic device such as a terminal device or a server. The terminal device can be any fixed or mobile terminal such as a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The server can be a single server or a server cluster composed of multiple servers. Any electronic device can implement the target detection optimization method by calling computer-readable instructions stored in a memory through a processor. Figure 1 As shown, the following steps are included:

[0048] S1. Input the image to be tested into a trained target detection model to obtain a target detection frame of the image to be tested and a first confidence of the target detection frame.

[0049] Among them, the image to be tested is an image that needs to be detected in any scene. The scene in this embodiment can be a scene such as a helmet detection at a production site, an electric bicycle detection in an elevator, etc. The target detection model defaults to the YOLOv8 network, and can also be selected according to actual conditions, without any restrictions. The number of target detection frames is determined according to the results of the target detection model, and the first confidence is the confidence of the target detection frame directly output by the target detection model, and each target detection frame corresponds to a first confidence.

[0050] It should be noted that the YOLOv8 network does not set or sets a lower confidence threshold to provide sufficient capacity for subsequent re-inspections.

[0051] S2. Generate background description text of the image to be tested by using the CO-STAR framework, and input the target detection box and the background description text into a trained multimodal large language model to obtain a second confidence of the target detection box.

[0052] Among them, Figure 2As shown in the figure, the CO-STAR framework is used to generate background description text of the image to eliminate hallucinations. The multimodal large language model has a powerful visual question-answering capability. It can infer the re-confidence score of each target detection box based on the target detection box containing the first confidence and the background description text. The second confidence that the multimodal large language model considers to be a false detection is set to 0, and a corresponding detailed analysis is given for each target detection box.

[0053] S3. Confidence fusion is performed on the first confidence level and the second confidence level of the target detection frame to obtain the target confidence level of the target detection frame.

[0054] In S3, specifically:

[0055] According to the confidence fusion formula, confidence fusion is performed in combination with the first confidence and the second confidence of the target detection frame to obtain the target confidence of the target detection frame.

[0056] Among them, the confidence fusion formula is: T final =w1·T1+w2·T2; T final is the target confidence, w1 is the weight corresponding to the first confidence, w2 is the weight corresponding to the second confidence, T1 is the first confidence, T2 is the second confidence, w1+w2=1.

[0057] In an optional manner, the method further includes:

[0058] Based on multiple training samples and the background description texts corresponding to the training samples, an instruction fine-tuning dataset for training a multimodal large language model is generated.

[0059] Based on the instruction fine-tuning data set, the local low-rank adaptation matrix included in the multimodal large language model is trained to obtain the trained multimodal large language model.

[0060] The background description text corresponding to the training sample is generated through the CO-STAR structure. The local low-rank adaptation matrix is ​​used to establish a bypass matrix in the multimodal large language model, and then the prompt word engineering (CO-STAR structure) is used to construct a few-sample fine-tuning dataset to train the local low-rank adaptation matrix. This tuning can be re-tuned with new false detection scenarios. While ensuring the detection accuracy, this iterative framework significantly improves the detection effect of the closed-set target detection model for boundary tasks that are difficult to identify, thereby realizing a "semi-open set" detection.

[0061] like Figure 3 As shown in the figure, for the training process of the multimodal large language model, specifically:

[0062] 1) First, select specific few-shot scenarios, such as helmet detection at production sites, electric bicycle detection in elevators, and other scene data. According to the pre-designed few-shot training quantity, take 20 samples (20shot) as an example, 1 / 5, i.e. 4, are correctly detected target samples, then 2 / 5, i.e. 8, are missed target samples, and finally 2 / 5 are misdetected non-target samples. Construct the training samples in the instruction fine-tuning dataset in a ratio of 1:2:2.

[0063] 2) Use the CO-STAR structure to construct prompt words (background description text), as shown below:

[0064] ①Context: Provides background information related to the task, such as the scene of the detection task, and the monitoring at the x intersection in x city in x province.

[0065] ②Objective: Define the task that you want LLM to focus on, that is, give a detailed description of the correctly detected data (correct target detection box), and give a detailed analysis of the cause of the error for the incorrectly detected data (falsely detected target detection box).

[0066] ③Style: Specify the writing style you want LLM to use. Here, it is set to be complete, rigorous, and step-by-step analysis, etc.

[0067] ④Tone: Set the attitude of the response. It should be neutral and contain no emotions.

[0068] ⑤ Audience: Determine the target audience for the response, ensuring it is appropriate and understandable in the required context. This is set here to provide data for fine-tuning the large model.

[0069] ⑥Response: Provide the format of the response. This ensures that LLM outputs the format required by downstream tasks. Here, fine-tune the required JSON multi-round dialogue format output according to the instructions.

[0070] Through the steps in 2), the background description text corresponding to the training samples in the instruction fine-tuning dataset is constructed.

[0071] 3) After obtaining the instruction fine-tuning data set, the local low-rank adaptation matrix contained in the multimodal large language model is trained. In this embodiment, the multimodal large language model uses "Pu Yu Ling Bi 2.5" by default.

[0072] In an optional manner, the method further includes:

[0073] S4. When the error value between the review result of the image to be tested and the target confidence is greater than a threshold, the instruction fine-tuning data set of the trained multimodal large language model is adjusted and trained to obtain an optimized multimodal large language model, and the optimized multimodal large language model is used as the trained multimodal large language model, and the process returns to execute S2.

[0074] The threshold is set according to the actual situation and is not limited here.

[0075] Specifically, when the re-inspection capability of the multimodal large language model needs to be further optimized, the final test results can be manually reviewed, and the problem can be located in combination with the re-inspection output of the multimodal large language model, and the multimodal large language model can be fine-tuned and trained again.

[0076] In this embodiment, in order to verify the fusion and enhancement effect of the technical solution of this embodiment with the current commonly used closed set target detection algorithm in real scenes, this embodiment designs and deploys two classic model baselines, namely YOLOv8 and FasterRCNN, where FasterRCNN is deployed using the mmDetection framework. Then, through the technical solution of this embodiment, the multimodal large language model is integrated into the detection process for verification. In order to be close to the actual production scene, this verification is based on seven common traffic and production safety monitoring scenes, including safety helmets, smoking, making phone calls, fire extinguishers, electric vehicles entering the ladder and reflective clothing. Each scene contains 100-300 test samples. As shown in Table 1 below, the average accuracy rate with a confidence threshold of 50 and the average accuracy rate with a threshold of 50-95 are taken as evaluation indicators at two levels. It can be seen from Table 1 that the target detection framework reconstructed using this embodiment performs better in both indicators and two baselines.

[0077] Table 1:

[0078] Model mAP50-95 mAP50 YOLOv8 0.716 0.917 YOLOv8+ multimodal large language model 0.751 0.944 Faster RCNN 0.689 0.898 Faster RCNN + Multimodal Large Language Model 0.724 0.922

[0079] In order to further verify that the multimodal large language model can effectively eliminate the illusion of a large model under the training process of this embodiment, and has excellent error correction capabilities for various types of missed detection and false detection samples. This verification designed common false detection data based on seven types of detection scenarios. In order to verify the ability improvement effect of few sample training, 5-50 pieces of training data were selected for each category, and 200 pieces of verification data were selected for each category. For the JSON data set of instruction fine-tuning, the prompt word engineering was used for description generation, and then the details were manually optimized. At the same time, in order to verify the optimality of the local fine-tuning method of this embodiment, the low-rank adaptive fine-tuning using the original version and the low-rank adaptive fine-tuning fused with the adapter were set as control groups. As shown in Table 2 below, the evaluation criterion for this verification is the error correction rate, which is calculated as follows: error correction rate = corrected sample / total number of false detection samples * 100%. Combined with Table 2, it can be seen that with the increase of training samples, the error correction rate continues to increase, and the improvement is most significant in the range of 5-20, and there is a slight improvement in the range of 20-50.

[0080] Table 2:

[0081]

[0082] The technical solution of this embodiment utilizes the rich pre-training knowledge and VQA capabilities of the multimodal large language model to perform fast model instruction fine-tuning with a small number of samples, and obtain a model version positioned in a specific scenario, and then integrates the positioned multimodal large language model into the re-inspection process of target detection, replacing 95% of the workload of traditional manual inspection in the form of VQA, thereby significantly optimizing the time consumption of this process and improving its problem tracing capability and intelligence.

[0083] The technical solution of this embodiment uses a multimodal large language model to recheck the target detection results, and gives new confidence judgments in a highly interpretable manner, which significantly improves the accuracy of the existing detection framework, thereby optimizing the target detection results.

[0084] Figure 4 FIG. 2 is a schematic diagram showing a structure of an embodiment of a target detection optimization system 200 provided by the present invention. Figure 4 As shown, the system 200 includes: a first generation module 210, a second generation module 220 and an optimization module 230;

[0085] The first generating module 210 is used to: input the image to be tested into the trained target detection model to obtain the target detection frame of the image to be tested and the first confidence of the target detection frame;

[0086] The second generating module 220 is used to: generate background description text of the image to be tested by using the CO-STAR framework, and input the target detection frame and the background description text into the trained multimodal large language model to obtain a second confidence of the target detection frame;

[0087] The optimization module 230 is used to: perform confidence fusion on the first confidence level and the second confidence level of the target detection frame to obtain the target confidence level of the target detection frame.

[0088] In an optional manner, the method further includes: a training module; the training module is used to:

[0089] Based on multiple training samples and the background description texts corresponding to the training samples, an instruction fine-tuning dataset for training a multimodal large language model is generated;

[0090] Based on the instruction fine-tuning data set, the local low-rank adaptation matrix included in the multimodal large language model is trained to obtain the trained multimodal large language model.

[0091] In an optional manner, the optimization module 230 is specifically used to:

[0092] According to the confidence fusion formula, the confidence fusion is performed by combining the first confidence and the second confidence of the target detection frame to obtain the target confidence of the target detection frame; wherein the confidence fusion formula is: final =w1·T1+w2·T2; T final is the target confidence, w1 is the weight corresponding to the first confidence, w2 is the weight corresponding to the second confidence, T1 is the first confidence, T2 is the second confidence, w1+w2=1.

[0093] In an optional manner, the method further includes: a review module; the review module is used to:

[0094] When the error value between the review result of the image to be tested and the target confidence is greater than a threshold, the instruction fine-tuning data set of the trained multimodal large language model is adjusted and trained to obtain an optimized multimodal large language model, and the optimized multimodal large language model is used as the trained multimodal large language model, and the second generation module is returned to be called.

[0095] The technical solution of this embodiment uses a multimodal large language model to recheck the target detection results, and gives new confidence judgments in a highly interpretable manner, which significantly improves the accuracy of the existing detection framework, thereby optimizing the target detection results.

[0096] The above-mentioned parameters and steps for each module to implement corresponding functions in the target detection optimization system 200 of this embodiment can refer to the parameters and steps in the embodiment of the target detection optimization method above, and will not be repeated here.

[0097] like Figure 5 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above-mentioned target detection optimization methods, specifically:

[0098] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements any target detection optimization method provided in the above embodiments. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the electronic device 300 may also include other components for implementing device functions, which will not be described in detail here.

[0099] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that the computer implements any of the above-mentioned target detection optimization methods.

[0100] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0101] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above-mentioned target detection optimization methods.

[0102] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0103] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0104] Any combination of one or more computer-readable media can be used. Computer-readable media can be computer-readable signal media or computer-readable storage media. Computer-readable storage media can be, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0105] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A target detection optimization method, characterized in that: include: S1. Inputting the image to be tested into a trained target detection model to obtain a target detection frame of the image to be tested and a first confidence of the target detection frame; S2. Generate background description text of the image to be tested using the CO-STAR framework, and input the target detection frame and the background description text into a trained multimodal large language model to obtain a second confidence level of the target detection frame; S3. Confidence fusion is performed on the first confidence level and the second confidence level of the target detection frame to obtain the target confidence level of the target detection frame.

2. The target detection optimization method according to claim 1, characterized in that: Also includes: Based on multiple training samples and the background description texts corresponding to the training samples, an instruction fine-tuning dataset for training a multimodal large language model is generated; Based on the instruction fine-tuning data set, the local low-rank adaptation matrix included in the multimodal large language model is trained to obtain the trained multimodal large language model.

3. The target detection optimization method according to claim 1, characterized in that: Step S3 includes: According to the confidence fusion formula, the confidence fusion is performed by combining the first confidence and the second confidence of the target detection frame to obtain the target confidence of the target detection frame; wherein the confidence fusion formula is: final =w1·T1+w2·T2; T final is the target confidence, w1 is the weight corresponding to the first confidence, w2 is the weight corresponding to the second confidence, T1 is the first confidence, T2 is the second confidence, w1+w2=1.

4. The target detection optimization method according to any one of claims 1 to 3, characterized in that: Also includes: S4. When the error value between the review result of the image to be tested and the target confidence is greater than a threshold, the instruction fine-tuning data set of the trained multimodal large language model is adjusted and trained to obtain an optimized multimodal large language model, and the optimized multimodal large language model is used as the trained multimodal large language model, and the process returns to execute S2.

5. A target detection optimization system, characterized in that: include: A first generation module, a second generation module and an optimization module; The first generating module is used to: input the image to be tested into the trained target detection model to obtain the target detection frame of the image to be tested and the first confidence of the target detection frame; The second generation module is used to: generate background description text of the image to be tested by using the CO-STAR framework, and input the target detection frame and the background description text into the trained multimodal large language model to obtain a second confidence of the target detection frame; The optimization module is used to: perform confidence fusion on the first confidence level and the second confidence level of the target detection frame to obtain the target confidence level of the target detection frame.

6. The target detection optimization system according to claim 5, characterized in that: Also includes: Training modules; The training module is used to: Based on multiple training samples and the background description texts corresponding to the training samples, an instruction fine-tuning dataset for training a multimodal large language model is generated; Based on the instruction fine-tuning data set, the local low-rank adaptation matrix included in the multimodal large language model is trained to obtain the trained multimodal large language model.

7. The target detection optimization system according to claim 5, characterized in that: The optimization module is specifically used for: According to the confidence fusion formula, the confidence fusion is performed by combining the first confidence and the second confidence of the target detection frame to obtain the target confidence of the target detection frame; wherein the confidence fusion formula is: final =w1·T1+w2·T2; T final is the target confidence, w1 is the weight corresponding to the first confidence, w2 is the weight corresponding to the second confidence, T1 is the first confidence, T2 is the second confidence, w1+w2=1.

8. The target detection optimization system according to any one of claims 5 to 7, characterized in that: Also includes: Review module; the review module is used to: When the error value between the review result of the image to be tested and the target confidence is greater than a threshold, the instruction fine-tuning data set of the trained multimodal large language model is adjusted and trained to obtain an optimized multimodal large language model, and the optimized multimodal large language model is used as the trained multimodal large language model, and the second generation module is returned to be called.

9. An electronic device, characterized in that: The electronic device includes a processor, the processor is coupled to a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the target detection optimization method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the computer-readable storage medium implements the target detection optimization method according to any one of claims 1 to 4.

Images