Training set construction method and device, equipment and medium

Abstract

The embodiment of the invention provides a training set construction method and device, equipment and a medium, and is used for solving the problem that a model cannot be effectively trained due to the fact that sample labeling cannot be accurately and effectively performed in related technologies. The electronic equipment inputs a first image to be labeled and a pre-stored image labeled with a target object into a pre-trained visual prompt detection model, and labeling of the first image to be labeled is achieved through the visual prompt model; and inputting the second image, a pre-stored positive sample image labeled with a correct detection frame and a scoring standard into a multi-modal large model, and scoring the labeling quality of the labeled second image through the multi-modal large model, so that the labeling accuracy is improved, and the accuracy of labeling is improved based on the labeled sample. The accuracy and efficiency of model training can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a training set construction method, device, equipment and medium. Background Art

[0002] Small object detection models usually need to be customized for each specific task and require a large amount of labeled data to achieve the desired effect. This not only leads to high data annotation costs, but also limits the scalability and migration capabilities of the model.

[0003] In order to train the model accurately and effectively, the data is manually labeled and then input into the model for training. Although this method performs well in processing speed, it still requires manual labeling, and the limitations of model generalization will also affect the accuracy of labeling. Summary of the invention

[0004] The embodiments of the present application provide a training set construction method, apparatus, device and medium for solving the problem in the related art that samples cannot be accurately and effectively labeled, and thus the model cannot be effectively trained.

[0005] In a first aspect, an embodiment of the present application provides a method for constructing a training set, the method comprising:

[0006] Inputting the first image to be annotated and the pre-saved image annotated with the target object into a pre-trained visual cue detection model, and obtaining a second image output by the visual cue detection model and annotated with the detection frame where the target object is located;

[0007] Inputting the second image, a pre-saved positive sample image marked with a correct detection frame, and a scoring standard into the multimodal large model, and obtaining a score corresponding to the second image output by the multimodal large model;

[0008] If the score of the second image is higher than a preset score, the second image is added to the training set.

[0009] Furthermore, after adding the second image to the training set, the method further includes:

[0010] Adjusting parameters of a preset type of a large open vocabulary object detection model based on each image in the training set;

[0011] A batch of unlabeled images are labeled based on the adjusted open vocabulary object detection large model, and the labeled images are added to the training set.

[0012] Furthermore, the method further comprises:

[0013] Based on the labeled image, the parameters of the preset type of the open vocabulary object detection large model are adjusted again.

[0014] Furthermore, the method further comprises:

[0015] Performing data enhancement processing on the images in the training set; wherein the data enhancement processing includes: rotation, cropping, and color jittering.

[0016] Furthermore, after determining that the score of the second image is higher than a preset score and before adding the second image to the training set, the method further includes:

[0017] Sending the second image and the score of the second image to a preset device to evaluate whether the quality of the annotation of the second image meets the standard;

[0018] If an instruction indicating that the preset device has passed the review is received, the subsequent step of adding the second image to the training set is performed.

[0019] Furthermore, the method further comprises:

[0020] If an instruction carrying expert knowledge is received and returned by a preset device, the expert knowledge is saved.

[0021] Furthermore, the step of inputting the second image, the pre-saved positive sample image marked with a correct detection frame, and the scoring criteria into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model includes:

[0022] If expert knowledge is pre-saved, the second image, the pre-saved positive sample image marked with the correct detection box, the expert knowledge, and the scoring criteria are input into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model.

[0023] In a second aspect, an embodiment of the present application further provides a training set construction device, the device comprising:

[0024] A processing module is used to input the first image to be annotated and the pre-saved image annotated with the target object into a pre-trained visual cue detection model, and obtain a second image output by the visual cue detection model and annotated with the detection frame where the target object is located; input the second image, the pre-saved positive sample image annotated with the correct detection frame, and the scoring standard into the multimodal large model, and obtain the score corresponding to the second image output by the multimodal large model;

[0025] An adding module is used to add the second image to the training set if the score of the second image is higher than a preset score.

[0026] Furthermore, the processing module is also used to adjust the preset type of parameters of the open vocabulary object detection large model based on each image in the training set; annotate a batch of unlabeled images based on the adjusted open vocabulary object detection large model, and add the labeled images to the training set.

[0027] Furthermore, the processing module is also used to adjust the parameters of the preset type of the open vocabulary object detection large model again based on the labeled image.

[0028] Furthermore, the processing module is also used to perform data enhancement processing on the images in the training set; wherein the data enhancement processing includes: rotation, cropping, and color jittering.

[0029] Furthermore, the adding module is also used to send the second image and the score of the second image to a preset device to evaluate whether the quality of the second image annotation meets the standard; if an instruction of review approval is returned by the preset device, the subsequent step of adding the second image to the training set is executed.

[0030] Furthermore, the processing module is further configured to save the expert knowledge if an instruction carrying the expert knowledge is received and returned by a preset device.

[0031] Furthermore, the processing module is specifically used to input the second image, the pre-saved positive sample image marked with the correct detection frame, the expert knowledge, and the scoring criteria into the multimodal large model if expert knowledge is pre-saved, and obtain the score corresponding to the second image output by the multimodal large model.

[0032] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0033] A memory for storing program instructions;

[0034] The processor is used to call the program instructions stored in the memory and execute the steps included in any of the above-mentioned methods for constructing a training set according to the obtained program instructions.

[0035] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes a method for constructing a training set as described in any one of the above items.

[0036] In an embodiment of the present application, the electronic device inputs a first image to be annotated and a pre-saved image annotated with a target object into a pre-trained visual cue detection model, and obtains a second image annotated with a detection frame output by the visual cue detection model; the second image, a pre-saved positive sample image annotated with a correct detection frame, and a scoring standard are input into a multimodal large model, and a score corresponding to the second image output by the multimodal large model is obtained; if the score of the second image is higher than a preset score, the second image is added to a training set. In an embodiment of the present application, the annotation of the first image to be annotated is achieved through the visual cue model, and the annotation quality of the annotated second image is evaluated through the multimodal large model, thereby improving the accuracy of the annotation, and based on the annotated samples, the accuracy and efficiency of the model training can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0038] Figure 1 A process diagram of a training set construction method provided in an embodiment of the present application;

[0039] Figure 2 A detailed process diagram of constructing a training set provided in an embodiment of the present application;

[0040] Figure 3 A detailed schematic diagram of a process for constructing a training set provided in an embodiment of the present application;

[0041] Figure 4 A structural diagram of a training set construction device provided in an embodiment of the present application;

[0042] Figure 5 A structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0044] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0045] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0046] The words "connected", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "plurality" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and subsequent associated objects are in an "or" relationship. The terms "first", "second", "third" and the like involved in this application are merely used to distinguish similar objects, and do not represent a specific ordering of objects.

[0047] In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the fact that they can be implemented by ordinary technicians in this field. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0048] In order to solve the problem in the related art that samples cannot be accurately and effectively labeled, and thus the model cannot be effectively trained, the embodiments of the present application provide a training set construction method, apparatus, device and medium.

[0049] The training set construction method includes: the electronic device inputs a first image to be labeled and a pre-saved image labeled with a target object into a pre-trained visual cue detection model, and obtains a second image output by the visual cue detection model and labeled with a detection frame where the target object is located; the second image, a pre-saved positive sample image labeled with a correct detection frame, and a scoring standard are input into a multimodal large model, and a score corresponding to the second image output by the multimodal large model is obtained; if the score of the second image is higher than a preset score, the second image is added to the training set.

[0050] Embodiment 1:

[0051] Figure 1 A process diagram of a training set construction method provided in an embodiment of the present application, the process includes the following steps:

[0052] S101: Input a first image to be annotated and a pre-saved image annotated with a target object into a pre-trained visual cue detection model, and obtain a second image output by the visual cue detection model and annotated with a detection frame where the target object is located.

[0053] The sample set construction method provided in the embodiment of the present application is applied to an electronic device, which may be a smart device such as a PC or a server.

[0054] In order to efficiently and accurately construct the training set, the electronic device locally stores a pre-trained visual cue detection model and pre-stores images of labeled target objects. These images can also be called visual cue images. For easy distinction, the images can be called first images. These images provide rich visual examples for the visual cue detection model. Specifically, the first images are carefully labeled by business personnel using point selection or target box marking technology after clarifying the category to be detected, ensuring the accuracy and pertinence of the annotation. Among them, the first image can be an image taken from a surveillance video or an industrial camera, etc.

[0055] In order to further improve the quality and construction efficiency of the training set, the electronic device inputs the first image and the pre-stored visual cue image into a pre-trained visual cue detection model. The model uses its powerful image recognition and processing capabilities to perform in-depth analysis of the input image, automatically mark the location of the target object, and generate an image with a detection frame. For easy distinction, the image is called the second image. Moreover, visual cue detection models rely on visual similarity rather than language relevance, allowing them to be applied to new objects not encountered during training. The visual cue detection model can be T-Rex.

[0056] It should be noted that there can be more than one first image input into the visual cue detection model, and the categories of the target objects marked in the multiple first images can also be different. The second image has been accurately identified and labeled by the visual cue detection model, and not only contains the accurate location information of the target object, but also retains the rich details of the original image.

[0057] S102: Input the second image, a pre-saved positive sample image marked with a correct detection frame, and a scoring standard into a multimodal large model to obtain a score corresponding to the second image output by the multimodal large model.

[0058] In order to further improve the quality and construction efficiency of the training set, the electronic device also locally stores a multimodal large model and a positive sample image annotated with a correct detection frame. After acquiring the second image, the electronic device can input the second image, the pre-saved positive sample image annotated with a correct detection frame, and the scoring criteria into the multimodal large model.

[0059] That is to say, the second image constructed by the visual cue model is used as the sample to be detected. The electronic device constructs corresponding visual instructions based on the second image, the pre-saved positive sample image annotated with the correct detection box, and the scoring criteria, so as to use the multimodal large model to determine whether the annotation of the second image is accurate according to the positive sample image and its correctly annotated category.

[0060] This process can be optimized in the following way: the electronic device pre-stores an instruction template, which has set placeholders for inserting the second image, the positive sample image and the scoring criteria. Then, the second image, the positive sample image with the correct detection frame annotated, and the corresponding scoring criteria are accurately filled into the corresponding positions of the instruction template. In this way, the multimodal large model can accurately judge the annotation of the second image based on this complete and formatted information.

[0061] The scoring criteria may be "evaluate whether the annotations in the image below are correct, and score based on the quality of the annotations, with a score range of 0 to 5, where 0 means the annotation is completely wrong and 5 means the annotation box perfectly covers the target object."

[0062] The electronic device obtains the output of the multimodal large model, which is a score for the annotation quality of the annotation box in the second image.

[0063] In a possible implementation, the multimodal large model may be GPT-4o, Qwen-VL, etc. It should be noted that the multimodal large model has strong image and text semantic understanding and instruction following capabilities. In the embodiment of the present application, the second image is filtered with the help of the multimodal large model.

[0064] The instruction template input into the multimodal large model may be: As a data annotation review administrator, the image above contains the detected annotation information. Based on the image information above, please evaluate whether the annotation in the image below is correct and score it based on the quality of the annotation. The score range is 0 to 5, where 0 means the annotation is completely wrong and 5 means the annotation box perfectly covers the target object. Please reply in the following JSON format: {"score":0}, .

[0065] It should be noted that It is a placeholder and will be replaced with the actual image when actually used.

[0066] In a possible implementation, the constructed positive sample image also includes the correct category of the annotated object, where the category can also be called a label.

[0067] S103: If the score of the second image is higher than a preset score, adding the second image to a training set.

[0068] After obtaining the score of the second classmate output by the multimodal large model, the electronic device can determine whether the score of the second image is higher than the preset score. If the score of the second image is higher than the preset score, the second image is added to the training set. If the score of the second image is not higher than the preset score, it is determined that the annotation quality of the second image is low. In view of the poor annotation quality of the second image, if it is used for model training, it will inevitably lead to a decrease in model recognition accuracy. Therefore, the electronic device will automatically eliminate the second image and prevent it from being included in the training data set.

[0069] In order to construct the training set, the electronic device may also add the unlabeled image corresponding to the second image to the corresponding training set, and save the corresponding relationship between the second image and the corresponding unlabeled image.

[0070] The embodiment of the present application is equivalent to providing a method for constructing a training set for an industrial cold-start rare target detection model.

[0071] In an embodiment of the present application, the electronic device inputs a first image to be annotated and a pre-saved image annotated with a target object into a pre-trained visual cue detection model, and obtains a second image annotated with a detection frame output by the visual cue detection model; the second image, a pre-saved positive sample image annotated with a correct detection frame, and a scoring standard are input into a multimodal large model, and a score corresponding to the second image output by the multimodal large model is obtained; if the score of the second image is higher than a preset score, the second image is added to a training set. In an embodiment of the present application, the annotation of the first image to be annotated is achieved through the visual cue model, and the annotation quality of the annotated second image is evaluated through the multimodal large model, thereby improving the accuracy of the annotation, and based on the annotated samples, the accuracy and efficiency of the model training can be improved.

[0072] Embodiment 2:

[0073] In order to accurately construct a training set, based on the above embodiment, in the embodiment of the present application, after adding the second image to the training set, the method further includes:

[0074] Adjusting parameters of a preset type of a large open vocabulary object detection model based on each image in the training set;

[0075] A batch of unlabeled images are labeled based on the adjusted open vocabulary object detection large model, and the labeled images are added to the training set.

[0076] Since the generalization ability of the visual cue detection model is relatively poor, and the efficiency of constructing a training set by cooperating with the visual cue detection model and the multimodal large model is low, the electronic device can also annotate a batch of unlabeled images through an open vocabulary object detection large model, thereby improving the efficiency of constructing a training set. Among them, the open vocabulary object detection large model can be a grounding dino for detecting objects based on text cues.

[0077] It should be noted that there is a large gap between the training data of the open source target detection large model and the closed source scenario. When directly applied to the actual scenario, the effect often cannot meet the business needs. Therefore, in order to improve the accuracy of the annotation of the open vocabulary target detection large model, the electronic device can first fine-tune the open vocabulary target detection large model. Specifically, the electronic device can adjust the parameters of the open vocabulary target detection large model based on each image in the training set. In the actual adjustment process, by freezing the main parameters and only fine-tuning a small number of parameters, that is, only adjusting the parameters of the preset types, a more accurate recognition effect can be achieved. During the fine-tuning process, the optimizer with weight decay (Adam Weight Decay, AdamW) and the learning rate scheduling strategy can be used to improve the training effect.

[0078] After obtaining the adjusted open vocabulary object detection large model, batches of unlabeled images can be labeled based on the adjusted open vocabulary object detection large model, and the labeled images can be added to the training set.

[0079] In the embodiment of the present application, a fine-tuned open vocabulary object detection large model is used to assist in the annotation of a large number of unlabeled data sets, thereby constructing a training set. Using a small amount of labeled data, the open source open vocabulary object detection large model is fine-tuned. The open vocabulary large model combines the text model with a larger visual backbone network to expand the model size, and is trained using a large amount of public data, which has a strong zero-sample object detection capability.

[0080] In order to construct the training set, the electronic device may also add the unlabeled images corresponding to the labeled images to the corresponding training set, and save the corresponding relationship between the labeled images and the corresponding unlabeled images.

[0081] The embodiment of the present application utilizes the efficient training characteristics of the open source large model under a small amount of annotated data to design a method for constructing a training set for a cold-start rare target detection model. This method can quickly construct a target detection large model, i.e., an open vocabulary target detection large model, without or with only a small amount of annotated data, to achieve rapid cold start and significantly reduce the cost of manual annotation.

[0082] Figure 2 A detailed process diagram of constructing a training set provided in an embodiment of the present application, the process includes the following steps:

[0083] S201: Inputting a first image to be annotated and a pre-saved image annotated with a target object into a pre-trained visual cue detection model.

[0084] S202: Obtain a second image output by the visual cue detection model and marked with a detection frame where the target object is located.

[0085] S203: Input the second image, the pre-saved positive sample image annotated with the correct detection frame, and the scoring criteria into the multimodal large model.

[0086] S204: Obtain a score corresponding to the second image output by the multimodal large model.

[0087] S205: If the score of the second image is higher than the preset score, the second image is added to the training set.

[0088] S206: Adjusting parameters of a preset type of the open vocabulary object detection large model based on each image in the training set.

[0089] S207: Annotate a batch of unlabeled images based on the adjusted open vocabulary object detection large model, and add the labeled images to the training set.

[0090] Traditional methods usually rely on the following methods to achieve fast cold start: Data augmentation: Expand the training data set through data augmentation technology to improve the generalization ability of the model. Transfer learning: Use the pre-trained model to transfer the knowledge learned on the large-scale data set to the target task to reduce the dependence on a few samples. Meta-learning: Through the meta-learning method, the model can quickly adapt to new tasks, thereby improving the efficiency and effect of cold start. Multi-task learning: Train the model on multiple related tasks at the same time to share knowledge and features and improve the performance of the model in the case of few samples. However, if the data augmentation strategy is not appropriate, it may cause the model to overfit to the augmented data rather than the original data. When the domain of the pre-trained model is too different from the domain of the target task, the transfer effect may not be ideal. Meta-learning algorithms are usually complex and difficult to implement and debug. There may be conflicts between different tasks, making it difficult for the model to achieve optimal performance on all tasks at the same time. In the initial research stage, researchers mainly focused on data labeling under small model learning. In order to achieve automatic labeling, they manually labeled a small amount of data, then input it into the small model for training, and then used the model's prediction results as the final labeling. Although this method performs well in processing speed, it still requires manual labeling, and the limitations of model generalization will also affect the accuracy of labeling. This application uses open source large models to assist in data labeling, reduce the number of data labeling, improve the quality of data labeling, and thus improve the effect of cold start.

[0091] Embodiment 3:

[0092] In order to improve the accuracy of the large model annotation of the open vocabulary object detection, on the basis of the above embodiments, in the embodiment of the present application, the method further includes:

[0093] Based on the labeled image, the parameters of the preset type of the open vocabulary object detection large model are adjusted again.

[0094] Since the quality of the training set cannot guarantee that all annotations are correct, electronic devices can adopt an open vocabulary target detection large model and data iterative evolution method to improve the accuracy of the open vocabulary target detection large model annotations.

[0095] Specifically, after acquiring the annotated image, the electronic device adjusts the parameters of the preset type of the open vocabulary object detection large model again based on the annotated image. In the embodiment of the present application, the electronic device uses a large number of training sets to fine-tune the open vocabulary object detection large model again, which not only expands the amount of data, but also can serve as the noise of the open vocabulary object detection large model, thereby improving the robustness of the open vocabulary object detection large model.

[0096] Among them, the process of repeatedly fine-tuning the open vocabulary object detection large model can be repeated 2 to 3 times to obtain an open vocabulary object detection large model with better performance and a high-quality large-scale training set.

[0097] In order to accurately and effectively construct a training set, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0098] Performing data enhancement processing on the images in the training set; wherein the data enhancement processing includes: rotation, cropping, and color jittering.

[0099] In an embodiment of the present application, the electronic device can perform data enhancement processing on the images in the training set, wherein the data enhancement technology includes random cropping, rotation, color jittering, etc., so as to increase the diversity of the data.

[0100] Depending on the scenario, if it is an edge scenario, a small model needs to be used to run online. Electronic devices can use this training set to train small models such as YOLO. The YOLO model has a high detection speed and low computing resource requirements, which is suitable for running on edge devices. If it is a cloud scenario, the training set can be used to quantize and compress the large target detection model, such as 8-bit integer (INT8) quantization to improve the model reasoning speed. TensorRT or Open Neural Network Exchange Runtime (ONNX Runtime) can be used for model optimization and acceleration.

[0101] Embodiment 4:

[0102] In order to improve the accuracy of annotation, based on the above embodiments, in the embodiment of the present application, after determining that the score of the second image is higher than the preset score, before adding the second image to the training set, the method further includes:

[0103] Sending the second image and the score of the second image to a preset device to evaluate whether the quality of the annotation of the second image meets the standard;

[0104] If an instruction indicating that the preset device has passed the review is received, the subsequent step of adding the second image to the training set is performed.

[0105] Since the accuracy of the scores determined by the multimodal large model is limited, the accuracy of the scores determined by the multimodal large model alone is not high. Therefore, in an embodiment of the present application, the scores determined by the multimodal large model can be reviewed based on business experts.

[0106] Specifically, the electronic device may send the second image and the score of the second image to the preset device so that the business expert can score the quality of the second image annotation. The electronic device may also send the category to which the corresponding target object belongs to the preset device.

[0107] The business expert can review the quality of the annotation of the second image based on the second image and the score of the second image. If the business expert passes the review, the preset button, such as the "pass" button, can be selected on the preset page corresponding to the preset device. At this time, the preset device can send the review pass instruction to the electronic device. If the electronic device receives the review pass instruction returned by the preset device, it is determined that the quality of the annotation in the second image meets the standards, and the electronic device can execute the subsequent step of adding the second image to the training set.

[0108] Embodiment 5:

[0109] In order to improve the accuracy of multimodal large model annotation, based on the above embodiments, in the embodiment of the present application, the method further includes:

[0110] If an instruction carrying expert knowledge is received and returned by a preset device, the expert knowledge is saved.

[0111] In an embodiment of the present application, a business expert reviews the score output by the multimodal large model and the corresponding annotated second image. If quality problems occur, the expert knowledge can be entered on the preset page of the preset device and a preset button, such as a "Submit" button, can be clicked. The preset device can then send an instruction carrying the expert knowledge to the electronic device. If the electronic device receives an instruction carrying the expert knowledge returned by the preset device, the expert knowledge is saved.

[0112] In a possible implementation, if quality problems occur, the electronic device can also return a labeling instruction to the preset device so that the business expert can add more sample images to the visual prompt model to improve the quality of the data. The business expert can add more sample images from different angles and under different lighting conditions.

[0113] In order to improve the accuracy of the score determined by the multimodal large model, on the basis of the above embodiments, in an embodiment of the present application, the second image, the pre-saved positive sample image marked with a correct detection frame, and the scoring standard are input into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model, including:

[0114] If expert knowledge is pre-saved, the second image, the pre-saved positive sample image marked with the correct detection box, the expert knowledge, and the scoring criteria are input into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model.

[0115] In order to improve the accuracy of the score determined by the multimodal large model, if expert knowledge is pre-saved, the electronic device can input the second image, the pre-saved positive sample image marked with the correct detection frame, the expert knowledge, and the scoring criteria into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model.

[0116] Figure 3 A schematic diagram of a detailed process for constructing a training set provided in an embodiment of the present application.

[0117] Depend on Figure 3 It can be seen that different categories of visual cue images can be input into the visual cue detection model, and the visual cue detection model automatically annotates the images to be annotated to obtain small-scale annotated data, which is then filtered by the multimodal large model and business experts. The open vocabulary object detection large model is fine-tuned based on the filtered small-scale annotated data, and the large-scale annotated data is annotated based on the fine-tuned open vocabulary object detection large model. The model and data are improved together, and the small object detection model is guided by the large-scale annotated data.

[0118] Embodiment 6:

[0119] Based on the same inventive concept, the present application embodiment provides a training set construction device, please refer to Figure 4 , the device comprises:

[0120] Processing module 401 is used to input the first image to be annotated and the pre-saved image annotated with the target object into a pre-trained visual cue detection model, and obtain the second image output by the visual cue detection model and annotated with the detection frame where the target object is located; input the second image, the pre-saved positive sample image annotated with the correct detection frame, and the scoring standard into the multimodal large model, and obtain the score corresponding to the second image output by the multimodal large model;

[0121] The adding module 402 is configured to add the second image to a training set if the score of the second image is higher than a preset score.

[0122] In a possible implementation, the processing module 401 is also used to adjust the parameters of a preset type of the open vocabulary object detection large model based on each image in the training set; annotate a batch of unlabeled images based on the adjusted open vocabulary object detection large model, and add the labeled images to the training set.

[0123] In a possible implementation, the processing module 401 is further configured to adjust the parameters of the preset type of the open vocabulary object detection large model again based on the labeled image.

[0124] In a possible implementation, the processing module 401 is further used to perform data enhancement processing on the images in the training set; wherein the data enhancement processing includes: rotation, cropping, and color jittering.

[0125] In a possible implementation, the adding module 402 is further used to send the second image and the score of the second image to a preset device to evaluate whether the quality of the second image annotation meets the requirements; if an instruction indicating that the review is passed is returned by the preset device, the subsequent step of adding the second image to the training set is executed.

[0126] In a possible implementation manner, the processing module 401 is further configured to save the expert knowledge if an instruction carrying the expert knowledge returned by a preset device is received.

[0127] In a possible implementation, the processing module 401 is specifically used to, if expert knowledge is pre-stored, input the second image, a pre-stored positive sample image marked with a correct detection frame, the expert knowledge, and a scoring standard into a multimodal large model to obtain a score corresponding to the second image output by the multimodal large model.

[0128] Embodiment 7:

[0129] Based on the same inventive concept, the present application embodiment provides an electronic device, which can implement the function of constructing the training set in the previous text. Please refer to Figure 5 The device includes a processor 501, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504.

[0130] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 performs the following steps:

[0131] Inputting the first image to be annotated and the pre-saved image annotated with the target object into a pre-trained visual cue detection model, and obtaining a second image output by the visual cue detection model and annotated with the detection frame where the target object is located;

[0132] Inputting the second image, a pre-saved positive sample image marked with a correct detection frame, and a scoring standard into the multimodal large model, and obtaining a score corresponding to the second image output by the multimodal large model;

[0133] If the score of the second image is higher than a preset score, the second image is added to the training set.

[0134] In a possible implementation, after adding the second image to a training set, the method further includes:

[0135] Adjusting parameters of a preset type of a large open vocabulary object detection model based on each image in the training set;

[0136] A batch of unlabeled images are labeled based on the adjusted open vocabulary object detection large model, and the labeled images are added to the training set.

[0137] In a possible implementation, the method further includes:

[0138] Based on the labeled image, the parameters of the preset type of the open vocabulary object detection large model are adjusted again.

[0139] In a possible implementation, the method further includes:

[0140] Performing data enhancement processing on the images in the training set; wherein the data enhancement processing includes: rotation, cropping, and color jittering.

[0141] In a possible implementation, after determining that the score of the second image is higher than a preset score and before adding the second image to a training set, the method further includes:

[0142] Sending the second image and the score of the second image to a preset device to evaluate whether the quality of the annotation of the second image meets the standard;

[0143] If an instruction indicating that the preset device has passed the review is received, the subsequent step of adding the second image to the training set is performed.

[0144] In a possible implementation, the method further includes:

[0145] If an instruction carrying expert knowledge is received and returned by a preset device, the expert knowledge is saved.

[0146] In a possible implementation, inputting the second image, a pre-saved positive sample image marked with a correct detection frame, and a scoring standard into a multimodal large model, and obtaining a score corresponding to the second image output by the multimodal large model includes:

[0147] If expert knowledge is pre-saved, the second image, the pre-saved positive sample image marked with the correct detection box, the expert knowledge, and the scoring criteria are input into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model.

[0148] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0149] The communication interface 502 is used for communication between the electronic device and other devices.

[0150] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0151] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0152] Embodiment 8:

[0153] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program executable by an electronic device. When the program runs on the electronic device, the electronic device implements the following steps when executing:

[0154] Inputting the first image to be annotated and the pre-saved image annotated with the target object into a pre-trained visual cue detection model, and obtaining a second image output by the visual cue detection model and annotated with the detection frame where the target object is located;

[0155] Inputting the second image, a pre-saved positive sample image marked with a correct detection frame, and a scoring standard into the multimodal large model, and obtaining a score corresponding to the second image output by the multimodal large model;

[0156] If the score of the second image is higher than a preset score, the second image is added to the training set.

[0157] In a possible implementation, after adding the second image to a training set, the method further includes:

[0158] Adjusting parameters of a preset type of a large open vocabulary object detection model based on each image in the training set;

[0159] A batch of unlabeled images are labeled based on the adjusted open vocabulary object detection large model, and the labeled images are added to the training set.

[0160] In a possible implementation, the method further includes:

[0161] Based on the labeled image, the parameters of the preset type of the open vocabulary object detection large model are adjusted again.

[0162] In a possible implementation, the method further includes:

[0163] Performing data enhancement processing on the images in the training set; wherein the data enhancement processing includes: rotation, cropping, and color jittering.

[0164] In a possible implementation, after determining that the score of the second image is higher than a preset score and before adding the second image to a training set, the method further includes:

[0165] Sending the second image and the score of the second image to a preset device to evaluate whether the quality of the annotation of the second image meets the standard;

[0166] If an instruction indicating that the preset device has passed the review is received, the subsequent step of adding the second image to the training set is performed.

[0167] In a possible implementation, the method further includes:

[0168] If an instruction carrying expert knowledge is received and returned by a preset device, the expert knowledge is saved.

[0169] In a possible implementation, inputting the second image, a pre-saved positive sample image marked with a correct detection frame, and a scoring standard into a multimodal large model, and obtaining a score corresponding to the second image output by the multimodal large model includes:

[0170] If expert knowledge is pre-saved, the second image, the pre-saved positive sample image marked with the correct detection box, the expert knowledge, and the scoring criteria are input into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model.

[0171] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0172] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0173] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0174] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0175] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A training set construction method, characterized in that: The method comprises: Inputting the first image to be annotated and the pre-saved image annotated with the target object into a pre-trained visual cue detection model, and obtaining a second image output by the visual cue detection model and annotated with the detection frame where the target object is located; Inputting the second image, a pre-saved positive sample image marked with a correct detection frame, and a scoring standard into the multimodal large model, and obtaining a score corresponding to the second image output by the multimodal large model; If the score of the second image is higher than a preset score, the second image is added to the training set.

2. The method according to claim 1, characterized in that After adding the second image to the training set, the method further comprises: Adjusting parameters of a preset type of a large open vocabulary object detection model based on each image in the training set; A batch of unlabeled images are labeled based on the adjusted open vocabulary object detection large model, and the labeled images are added to the training set.

3. The method according to claim 2, characterized in that The method further comprises: Based on the labeled image, the parameters of the preset type of the open vocabulary object detection large model are adjusted again.

4. The method according to claim 2 or 3, characterized in that: The method further comprises: Performing data enhancement processing on the images in the training set; wherein the data enhancement processing includes: rotation, cropping, and color jittering.

5. The method according to claim 1, characterized in that After determining that the score of the second image is higher than a preset score and before adding the second image to the training set, the method further includes: Sending the second image and the score of the second image to a preset device to evaluate whether the quality of the annotation of the second image meets the standard; If an instruction indicating that the preset device has passed the review is received, the subsequent step of adding the second image to the training set is performed.

6. The method according to claim 5, characterized in that The method further comprises: If an instruction carrying expert knowledge is received and returned by a preset device, the expert knowledge is saved.

7. The method according to claim 1, characterized in that The step of inputting the second image, the pre-saved positive sample image marked with a correct detection frame, and the scoring standard into the multimodal large model, and obtaining the score corresponding to the second image output by the multimodal large model, comprises: If expert knowledge is pre-saved, the second image, the pre-saved positive sample image marked with the correct detection box, the expert knowledge, and the scoring criteria are input into the multimodal large model to obtain the score corresponding to the second image output by the multimodal large model.

8. A training set construction device, characterized in that: The device comprises: A processing module is used to input the first image to be annotated and the pre-saved image annotated with the target object into a pre-trained visual cue detection model, and obtain a second image output by the visual cue detection model and annotated with the detection frame where the target object is located; input the second image, the pre-saved positive sample image annotated with the correct detection frame, and the scoring standard into the multimodal large model, and obtain the score corresponding to the second image output by the multimodal large model; An adding module is used to add the second image to the training set if the score of the second image is higher than a preset score.

9. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in the method for constructing a training set according to any one of claims 1 to 7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method for constructing a training set as described in any one of claims 1 to 7.

Images