An AI large model-based semi-automatic labeling and semi-automatic auditing method

Abstract

The application discloses a kind of based on AI big model's semi-automatic labeling and semi-automatic auditing method, it is related to image processing, target detection and artificial intelligence application field.The application includes: first step: using a big model and unsupervised labeling technology has obtained pre-labeling model, obtains preliminary preselection label frame;Second step: training a labeling effect auditing network, for excluding target frame that does not meet the requirements;Third step: based on the first two steps obtained target frame pre-extraction and target frame confidence analysis two network models, further labeling is carried out on the pre-labeled label screened by artificial, perfect label;Subsequently, based on SAM semantic segmentation big model for perfect label result, obtain target contour, to further screen target frame using label frame detection model, and the labeling result is audited, and the final labeling result is formed after auditing and perfecting.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, target detection and artificial intelligence application, and particularly relates to a semi-automatic labeling and semi-automatic auditing method based on AI large model. BACKGROUND

[0002] Data labeling is a key link in the field of artificial intelligence. In recent years, with the rapid development of deep learning and machine learning technology, the demand for high-quality labeled data has been increasing. The global data labeling market continues to expand, and it is expected to maintain a high growth rate in the next few years. The current sample labeling work has become the longest time-consuming and highest cost link in the field of artificial intelligence. Among many data labeling tasks, complex tasks such as target box detection positioning and semantic segmentation have relatively high costs. For high-precision requirements, multiple audits are needed to ensure sample quality, which further increases costs. According to the relevant information available on the Internet, the unit price of a relatively basic image labeling task is between 0.01 yuan and 0.5 yuan. For more complex labeling work, the unit price may reach several yuan or even more. The labeling cost of a data set containing 10,000 images may reach tens of thousands of yuan. How to effectively improve the sample labeling speed and quality has become a key proposition in the field of artificial intelligence. Reducing the workload of sample labeling and checking errors generated during the labeling process is a key.

[0003] The current data labeling industry is experiencing a process from extensive growth to refinement and intelligent development. At the same time, it is also seeking to reduce the cost of labor-intensive work and seeking technical innovation to improve the economic benefits and sustainable development capacity of the overall industry. To meet the needs of industry development and solve industry development problems, the present application aims to address the high cost and long time consumption of model labeling and auditing. It designs a semi-automatic labeling and semi-automatic auditing method based on AI large model to intelligentize the sample labeling link. While labeling pictures, it captures the background information around the labeling target. By referring to the booming large model and unsupervised technology today, it uses self-supervised methods and traditional segmentation methods to pre-label the target. After labeling is completed, it checks the labeled boxes to reduce the workload of manual auditing. SUMMARY

[0004] In view of the deficiencies of the prior art, the present application provides a semi-automatic labeling and semi-automatic auditing method based on AI large model to reduce the workload of sample labeling and improve the efficiency of artificial intelligence deployment. The specific process is shown in Figure 1 By improving the pre-selected frame and auditing link content in the manual labeling process, the workload of manual labeling is reduced. The method mainly includes the following steps:

[0005] An AI large model-based semi-automatic labeling and semi-automatic auditing method, comprising the following steps:

[0006] S01: Establish a label frame pre-extraction model to provide initial labeling prompts for the labeling process;

[0007] S02: Based on a self-supervised learning method, a target detection frame confidence detection model is trained;

[0008] S03: According to the models obtained in S01 and S02, input the image to the label frame pre-extraction model to obtain the label frame of the image, and input the label frame slice to the target detection frame confidence detection model to obtain the label frame confidence, provide initial audit suggestions and preliminary screening labels, and after manual labeling, according to the manual labeling result and the image features, the label frame confidence is calculated again, the label score ranking is provided, and the corrected data set result is obtained.

[0009] Further, in S01, comprising:

[0010] S101: Based on the pre-processed results of the input image, obtain the target prompt frame and confidence of the image based on multiple deep target detection large models;

[0011] S102: Based on the obtained target prompt frame, perform target frame maximum value suppression to obtain the final foreground target detection frame;

[0012] S103: Based on the side length and area information of the target detection frame, screen out obviously inconsistent detection targets.

[0013] Further, in step S02, comprising:

[0014] S201: Based on various types of target detection data sets, first generate target frame image slices;

[0015] S202: Based on the target frame image slices and label frame coordinate information obtained in S201, generate a reference confidence of the detection frame combined with the feature information, calculate the IOU as the target frame reference confidence, and obtain the usable training data set;

[0016] S203: Based on the image slices, label frames and reference confidence in S202, perform network training, wherein the network input is the image slice and the label frame, and the output is the reference confidence, and the audit network model is trained;

[0017] In the training process of S203, the target frame coordinate information is introduced: the image obtains a feature map F after passing through the basic network framework, the feature map F is cropped based on the target frame, and the feature vectors of equal length are obtained by downsampling the feature map before and after cropping. The two feature vectors are combined and transmitted into the network as new features.

[0018] Further, in S03, comprising:

[0019] S301: input the target detection frame obtained in S01 and the original image, and generate a target slice;

[0020] S302: based on the segmentation large model, obtain the contour information of the target in the image based on the target detection frame and the target slice, filter out the target frame that does not meet the requirements based on the contour information, and filter out the unqualified labeling sample based on the target detection frame confidence detection model trained in S02;

[0021] S303: in the screened image and labeling frame, artificial correction is performed, and category information is supplemented;

[0022] S304: check by the target detection frame confidence detection model trained in S02, and save into the warehouse to form a new data set after no error.

[0023] Compared with the prior art, the beneficial effects of the present application are:

[0024] In view of the problems of multiple target labeling tasks, high cost and long time-consuming of auditing in deep learning task, a semi-automatic labeling and semi-automatic auditing method based on AI large model is proposed, compared with the prior art process, the present application has the following advantages:

[0025] 1. By pre-labeling algorithm, the possible target in the image is extracted, the workload of manual labeling is reduced, and the sample labeling efficiency is improved.

[0026] 2. The reference confidence can be obtained by calculating the matching degree of the sample frame and its background, the sample labeling auxiliary auditing suggestion is provided, and the workload required for auditing is reduced. BRIEF DESCRIPTION OF DRAWINGS

[0027] Figure 1 is the basic method flow diagram in the embodiment of the present application;

[0028] Figure 2 is the initial labeling structure diagram in the embodiment of the present application;

[0029] Figure 3 is the network diagram of the auditing module in the embodiment of the present application;

[0030] Figure 4 is the specific implementation flowchart of the embodiment of the present application

[0031] Figure 5 is the pre-labeling and scoring effect diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0032] The application will be further clarified by the following drawings and specific embodiments, which are intended to be illustrative only and not limiting of the scope of the application. Any modifications of the application, obvious to those skilled in the art, falling within the scope of the claims appended hereto, are intended to be within the scope of the application.

[0033] An AI large model-based semi-automatic labeling and semi-automatic auditing method, specifically as follows,

[0034] Step 1, in the pre-labeling stage, the possible target bounding boxes of the image are selected in advance after inputting the image; this process involves a large model and unsupervised labeling technology, mainly using the pre-selected boxes generated by the target detection large model during operation, and through the integration and screening of these pre-selected boxes, the preliminary pre-selected labeling boxes are obtained.

[0035] Step 2, train a labeling effect auditing network to filter out target boxes that do not meet the requirements; the specific operation is to generate target box image slices based on the target detection dataset, and then based on the self-supervised learning method and target box image slices. Based on the target box slices, the target box information known in advance is combined to add coordinate perturbations to the real target box, simulate the target box input in the real situation, and take the target box before adding perturbations as the reference value to train a model that detects the confidence of the target detection box;

[0036] Step 3, based on the two network models of target box pre-extraction and target box confidence analysis obtained in the previous two steps, and based on the basic steps shown in Figure 1 , the two models are used to assist labeling. First, generate pre-labeling labels through step 1, and manually further label and perfect the labels on the pre-labeled labels screened; then, based on the perfect labeling results, use the labeling box detection model obtained in step 2 to audit the labeling results, and after auditing and perfecting, form the final labeling results.

[0037] The following is a more specific embodiment, as shown in Figure 1 , the specific process is as follows:

[0038] S01: Based on a large model and unsupervised labeling technology, pre-labeling prompts are realized in the labeling process.

[0039] S101: Input an image, after preprocessing, based on a deep unsupervised target detection large model (such as GroudingDINO, YOLOv8) to obtain the detection box and confidence of the image.

[0040] S102: The pre-labeled box results of multiple models are integrated, and a non-maximum suppression operation (NMS) is performed on the labeled boxes according to the confidence. Specifically, the extracted boxes are sorted according to the confidence, and for the target boxes with an overlapping area greater than a certain threshold, only the one with higher confidence is taken. The judgment of the overlapping range is made by IOU (degree of overlap).

[0041] S103: For the target extraction box after maximum suppression, the target box obviously larger than the to-be-determined target is removed according to the area, side length and other factors. In this patent, the target box with an area greater than the full Figure 1 / 4 area of the region is removed in one application.

[0042] S02: The target frame audit network is used to screen out the detection frame that does not actually contain the target. Next, the data set construction, network loss function construction, network structure and training process of the target frame audit network are introduced.

[0043] S201: First, the data set construction method is introduced. This method is based on a target detection data set to generate target frame image slices, and based on a self-supervised learning method for training to obtain a target detection frame confidence detection model for judging whether the detection frame contains a target and whether the detection frame better wraps the target range. The specific structure and process are shown in Figure 3

[0044] S202: Data set construction. In this paper, the target frame image slices are generated based on the historical labeled target detection data set. The specific operation is as follows: fill the image with a black border of 1 / 4 of the original size, then loop through all the target frames in the image, and expand the target frame range. The expanded range is 1 / 4 of the original target frame range to generate image slices and new detection frame coordinates. The slice image and image target frame position are obtained. During training, the input image and the perturbed image target frame position are input. The target frame perturbation includes adding an offset to the target frame center coordinates, adjusting the target frame height and adjusting the target frame width.

[0045] ​S203: Loss function design, the image slice and detection box coordinate information obtained in S201 are input into the network for training. Assuming that the obtained image and detection box are Pair1, an offset is added to the detection box to obtain Pair2, then the same enhancement transformation is performed on Pair1 and Pair2, and then the IOU of the target box after the enhancement is calculated as the target box reference confidence, and the specific formula is shown in formula (1). The confidence range is between 0 and 1. After the target box with the added offset and the original image are input into the network, a network predicted confidence score result is obtained, and a MSE loss function is constructed between the score and the above-mentioned reference confidence, and the MSE loss function is shown in formula (2), wherein the output confidence result is Pre_i, and the reference confidence result is Ref_i.

[0046]

[0047] S204: Network structure and training process introduction, finally, the propagation path of data in the network and the model structure details are introduced, and the network is trained based on the target slice image, the labeled box and the reference confidence obtained in S201, wherein the network input is the target slice image and the labeled box, and the output is the reference confidence; specifically: the image is input into the basic feature extraction network (such as ResNet34 network) to obtain 1 / 4, 1 / 8, 1 / 16 and 1 / 32 size down-sampling feature maps, which are respectively up-sampled to the original image size for splicing to obtain a feature map F. The feature map F is cropped based on the target box, and the feature maps before and after cropping are down-sampled to obtain equal-length feature vectors, which are combined and input into 4 fully connected layers to obtain the final confidence score. According to the network structure, the data prepared in S202 is input, the network parameters and hyperparameters are set, and then the training is performed, the output result is compared with the confidence score, and the model for target box review is obtained after the training is completed.

[0048] S03: Based on the two steps, the target box pre-extraction and the target box confidence analysis network models obtained in the two steps are used for target detection actual auxiliary labeling work, the feature coincidence degree is calculated to obtain the labeled box confidence, the initial review suggestion is provided, the label is preliminarily selected, and after manual labeling, the labeling review module is provided according to the manual labeling result and the image features, the label score ranking is provided, and the auxiliary review is provided;

[0049] The specific mode is:

[0050] S301: input the original image, get the initial target frame and the original image through S01, and then generate the target slice according to the target frame. Based on the segmentation large model (such as SAM), the target contour information in the target frame is obtained, and the contour information is compared with the information of the target frame. For the results with large difference between the contour information and the target frame, the contour information is removed. The contour information comparison includes the intersection and union ratio of the contour envelope and the target frame, the center point Euclidean distance and the contour area. After removing the target frame that does not meet the requirements, the final pre-labeling result is obtained.

[0051] S302: input the target detection frame and the slice image batch obtained by slicing the image into S02 to obtain the confidence detection model trained in S02 to obtain the corresponding confidence ranking. According to the ranking, unqualified labeling samples are removed, such as Figure 3 As shown in the figure, the initial labeling frame and the confidence ranking are shown.

[0052] S303: in the screened image and labeling frame, manual correction is carried out, and the category information is supplemented;

[0053] S304: finally, for the completed labeling result, the model trained in S02 is used for the final check, and after confirming that there is no error, it is saved into the warehouse to obtain a new target detection sample.

Claims

1. A semi-automatic annotation and semi-automatic review method based on AI large model, characterized by: The following steps are involved: S01: Establish a pre-extraction model for annotation boxes to provide initial annotation prompts for the annotation process; S02: Based on the self-supervised learning method, the target detection box confidence detection model is trained; S03: Based on the model obtained in S01 and S02, the image is input into the annotation box pre-extraction model to obtain the image annotation box. After the annotation box is sliced, it is input into the target detection box confidence detection model to obtain the annotation box confidence, provide initial review suggestions and preliminary screening labels, and after manual annotation, recalculate the annotation box confidence based on the manual annotation results and image features, provide label score ranking, and obtain the corrected dataset results; In step S02, it includes: S201: Based on various public target detection datasets, first generate target frame image slices; S202: Obtain the target frame image slice and the annotation frame coordinate information in S201, generate the reference confidence of the detection frame in combination with the feature information, and calculate the IOU as the reference confidence of the target frame, thereby obtaining a usable training dataset; S203: Performing network training based on the image slices, annotation boxes, and reference confidences in S202, wherein the network inputs are the image slices and annotation boxes, and the output is the reference confidence, and an audit network model is obtained through training; In the training process of S203, the target frame coordinate information is introduced: after the image passes through the basic network framework, the feature map F is obtained, the feature map F is cropped based on the target frame, and the feature maps before and after cropping are downsampled to obtain feature vectors of equal length. The two feature vectors are merged and passed into the network as new features.

2. The semi-automatic annotation and semi-automatic review method based on AI large model according to claim 1 is characterized in that: In S01, including: S101: Based on the preprocessed result of the input image, the target prompt box and confidence level of the image are obtained based on multiple deep object detection models; S102: Based on the obtained target prompt frame, perform target frame maximum suppression to obtain the final foreground target detection frame; S103: Based on the side length and area information of the target detection frame, filter out the detection targets that are obviously not in compliance.

3. The semi-automatic annotation and semi-automatic review method based on AI large model according to claim 1 is characterized in that: In S03, including: S301: Input the target detection frame and original image obtained in S01 to generate a target slice; S302: Based on the large segmentation model, the target detection box and the target slice are used as the basis to obtain the contour information of the target in the image. Based on the contour information, the target boxes that do not meet the requirements are filtered out. Then, based on the target detection box confidence detection model trained in S02, unqualified labeled samples are filtered out. S303: Perform manual correction on the filtered images and annotation frames, and supplement category information; S304: Check the target detection box confidence detection model trained in S02, and save it into the database to form a new data set after it is correct.

Images