Semi-automatic labeling and semi-automatic auditing method based on AI large model

Abstract

The invention discloses a semi-automatic labeling and semi-automatic auditing method based on an AI large model, and relates to the field of image processing, target detection and artificial intelligence application. Comprising the following steps: 1, obtaining a pre-labeling model by applying a large model and an unsupervised labeling technology, and obtaining a preliminary pre-selected labeling box; 2, training a labeling effect auditing network for screening out target frames which do not meet requirements; a third step of obtaining two network models of target frame pre-extraction and target frame confidence analysis based on the previous two steps, and manually performing further labeling on the screened pre-labeling labels to perfect the labels; and then, for the perfect label result, obtaining a target contour based on an SAM semantic segmentation large model, further screening a target frame according to the target contour, and auditing the labeling result by adopting a labeling frame detection model, and forming a final labeling result after the auditing is perfect.

Description

Technical Field

[0001] The present invention relates to the fields of image processing, target detection and artificial intelligence applications, and in particular to a semi-automatic labeling and semi-automatic review method based on an AI large model. Background Art

[0002] Data labeling is a key link in the field of artificial intelligence. In recent years, with the rapid development of deep learning and machine learning technologies, the demand for high-quality labeled data has been growing. The global data labeling market continues to expand, and it is expected to maintain a high growth rate in the next few years. At present, sample labeling has become the longest and most expensive link in the field of artificial intelligence. Among the many data labeling tasks, complex tasks such as target box detection and positioning, semantic segmentation, etc. are relatively expensive. For high-precision requirements, multiple audits are required to ensure the quality of samples, which further increases the cost. According to relevant information that can be queried online, the unit price of a relatively basic image labeling task is between 0.01 yuan and 0.5 yuan, while for more complex labeling work, the unit price may reach several yuan or even higher. The labeling cost of a data set containing 10,000 images may reach tens of thousands of yuan. How to effectively improve the speed and quality of sample labeling has become a key proposition in the field of artificial intelligence, and reducing the workload of sample labeling and checking the errors generated during the labeling process are the key.

[0003] The current data labeling industry is undergoing a process of development from extensive growth to intensive and intelligent development. At the same time, it is also seeking to reduce the cost of labor-intensive work and seek technological innovation to enhance the overall economic benefits and sustainable development capabilities of the industry. In order to meet the needs of industry development and solve industry development problems, the present invention focuses on the intelligentization of the sample labeling link to address the problems of high cost and time-consuming model labeling and review, and designs a semi-automatic labeling and semi-automatic review method based on an AI large model. While labeling the picture, the background information around the labeled target is captured. Drawing on the current booming large models and unsupervised technology, self-supervision methods and traditional segmentation methods are used to pre-label the target. After the labeling is completed, the labeled box is checked in combination with the background information to reduce the workload of manual review. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention provides a semi-automatic labeling and semi-automatic review method based on AI large model, which reduces the workload of sample labeling and improves the efficiency of AI deployment. The specific process is as follows: Figure 1 As shown in the figure, by improving the pre-selection box and review process in the manual annotation process, the workload of manual annotation is reduced, which mainly includes the following steps:

[0005] A semi-automatic annotation and semi-automatic review method based on an AI large model, comprising the following steps:

[0006] S01: Establish a pre-extraction model for annotation boxes to provide initial annotation hints for the annotation process;

[0007] S02: Based on the self-supervised learning method, the target detection box confidence detection model is trained;

[0008] S03: Based on the model obtained in S01 and S02, the image is input into the annotation box pre-extraction model to obtain the annotation box of the image. After the annotation box is sliced, it is input into the target detection box confidence detection model to obtain the annotation box confidence, provide initial review suggestions and preliminary screening labels, and after manual labeling, calculate the annotation box confidence again based on the manual labeling results and image features, provide label score ranking, and obtain the corrected data set results.

[0009] Furthermore, in S01, it includes:

[0010] S101: obtaining a target prompt box and a confidence level of the image based on a plurality of deep target detection models according to the result of the preprocessing of the input image;

[0011] S102: Based on the obtained target prompt frame, maximum suppression of the target frame is performed to obtain a final foreground target detection frame;

[0012] S103: Screen out detection targets that are obviously not in compliance based on the side length and area information of the target detection frame.

[0013] Furthermore, in step S02, it includes:

[0014] S201: Based on various public target detection datasets, first generate target frame image slices;

[0015] S202: Obtain the target frame image slice and the annotation frame coordinate information in S201, generate the reference confidence of the detection frame in combination with the feature information, calculate the IOU as the reference confidence of the target frame, and thus obtain a usable training data set;

[0016] S203: Performing network training based on the image slices, annotation boxes, and reference confidences in S202, wherein the network input is the image slices and annotation boxes, and the output is the reference confidence, and the audit network model is obtained through training;

[0017] In the training process of S203, the target frame coordinate information is introduced: the image obtains a feature map F after passing through the basic network framework, the feature map F is cropped based on the target frame, the feature maps before and after cropping are downsampled to obtain feature vectors of equal length, and the two feature vectors are merged and passed into the network as new features.

[0018] Furthermore, in S03, it includes:

[0019] S301: input the target detection frame and the original image obtained in S01 to generate a target slice;

[0020] S302: Based on the large segmentation model, the target detection frame and the target slice are used as the basis to obtain the contour information of the target in the image, and the target frames that do not meet the requirements are screened based on the contour information, and then the unqualified labeled samples are screened out based on the target detection frame confidence detection model trained in S02;

[0021] S303: Performing manual correction on the filtered images and annotation frames, and supplementing category information;

[0022] S304: Check the target detection box confidence detection model trained in S02, and save it into the database to form a new data set after it is correct.

[0023] Compared with the prior art, the present invention has the following beneficial effects:

[0024] Aiming at the problems of many deep learning task labeling tasks, high costs, and long review time, a semi-automatic labeling and semi-automatic review method based on AI large model is proposed. Compared with the existing technical process, the present invention has the following advantages:

[0025] 1. Through the pre-labeling algorithm, possible targets in the image are extracted to reduce the workload of manual labeling and improve the efficiency of sample labeling.

[0026] 2. By calculating the degree of match between the sample frame and its background, the reference confidence level can be obtained, and sample annotation auxiliary audit suggestions can be provided to reduce the workload required for auditing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of a basic method flow in an embodiment of the present invention;

[0028] Figure 2 is a schematic diagram of an initial marking structure in an embodiment of the present invention;

[0029] Figure 3 A schematic diagram of a network of an audit module in an embodiment of the present invention;

[0030] Figure 4 The specific implementation flow chart of the audit in the embodiment of the present invention is as follows

[0031] Figure 5 This is a diagram of the pre-marking and scoring effects in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention is further explained below in conjunction with the accompanying drawings and specific embodiments. It should be understood that these examples are only used to illustrate the present invention and are not used to limit the scope of the present invention. After reading the present invention, various equivalent forms of modifications to the present invention by those skilled in the art all fall within the scope defined by the claims attached to this application.

[0033] A semi-automatic annotation and semi-automatic review method based on AI big model, as follows:

[0034] Step 1: In the pre-labeling stage, after the image is input, possible targets in the image are pre-selected. This process involves a large model and unsupervised labeling technology, which mainly uses the pre-selected boxes generated by the large target detection model during operation. By integrating and screening these pre-selected boxes, a preliminary pre-selected labeling box is obtained.

[0035] Step 2: Train a labeling effect review network to filter out target boxes that do not meet the requirements. The specific operation is to generate target box image slices based on the target detection dataset, and then use the self-supervised learning method and target box image slices as the basis. Based on the target box slices, combined with the pre-known target box information, add coordinate perturbations to the real target box, simulate the target box input in the real situation, and use the target box before adding perturbations as the reference value for training to obtain a model for detecting the confidence of the target detection box.

[0036] Step 3: Based on the first two steps, two network models for target box pre-extraction and target box confidence analysis are obtained. Figure 1 The basic steps shown are based on the two models to assist in labeling. First, the pre-labeled labels are generated through step 1, and the pre-labeled labels are further labeled manually to improve the labels; then, for the improved label results, the labeling box detection model obtained in step 2 is used to review the labeling results, and the final labeling results are formed after review and improvement.

[0037] The following is a more specific embodiment. Figure 1 As shown, the specific process is:

[0038] S01: Based on large models and unsupervised annotation technology, pre-annotation prompts are realized in the annotation process;

[0039] S101: Input an image, and after preprocessing, obtain the detection box and confidence of the image based on a large deep unsupervised object detection model (such as GroudingDINO, YOLOv8).

[0040] S102: Integrate the pre-labeled box results of multiple models, and perform non-maximum suppression (NMS) on the labeled boxes according to the confidence level. Specifically, sort the extracted boxes according to the confidence level, and for target boxes with overlapping areas larger than a certain threshold, only take the results with higher confidence levels. The overlapping range is determined by IOU (overlap).

[0041] S103: For the target extraction frame after maximum value suppression, the target frame that is obviously larger than the target to be determined is screened and removed according to factors such as area and side length. For example, in one application of this patent, the target frame with an area larger than the entire target is removed. Figure 1 / 4 area target box.

[0042] S02: Based on the target box review network, the detection boxes that do not actually contain the target are screened out. Next, the dataset construction, network loss function construction, network structure and training process of the target box review network are introduced.

[0043] S201: First, we introduce the dataset construction method. This method generates target frame image slices based on the target detection dataset, and trains based on the self-supervised learning method to obtain the target detection frame confidence detection model, which is used to determine whether the detection frame contains the target and whether the detection frame covers the target range well. The specific structure and process are as follows: Figure 3 As shown;

[0044] S202: Dataset construction. This paper generates target frame image slices based on various target detection datasets with historical annotations. The specific operations are as follows: fill the image with a black border of 1 / 4 the size of the original image, and then loop through all the target frames in the image, expand the target frame range, and generate image slices and new detection frame coordinates with a range of 1 / 4 of the original target frame range. The sliced ​​image and image target frame position are obtained. The image and the perturbed image target frame position are input during training. The target frame perturbation includes adding an offset to the target frame center coordinate, adjusting the target frame height, and adjusting the target frame width.

[0045] S203: Loss function design. For S201, the image slice and detection frame coordinate information are obtained and input into the network for training. Assuming that the obtained image and detection frame are Pair1, an offset is added to the detection frame to obtain Pair2. Then, the same enhancement transformation is performed on Pair1 and Pair2. Then, the IOU of the enhanced Pair1 and Pair2 target frames is calculated as the reference confidence of the target frame. The specific formula is shown in formula (1). The confidence range is between 0 and 1. After the target frame with the offset added and the original image are input into the network, a network predicted confidence score result will be obtained. An MSE loss function is constructed between this score and the above reference confidence. The MSE loss function is shown in formula (2), where the output confidence result is Pre_i, and the reference confidence result is Ref_i;

[0046]

[0047] S204: Introduction to network structure and training process. Finally, the data propagation path in the network and the details of the model structure are introduced. The network training is performed based on the target slice image, annotation box and reference confidence obtained in S201, where the network input is the target slice image and annotation box, and the output is the reference confidence. Specifically: the image is subjected to a basic feature extraction network (such as a ResNet34 network) to obtain downsampled feature maps of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 sizes, which are upsampled to the original image size for splicing to obtain feature map F. Feature map F is cropped based on the target frame, and the feature maps before and after cropping are downsampled to obtain feature vectors of equal length. The two feature vectors are merged and passed through four fully connected layers to obtain the final confidence score. According to the network structure, the data prepared in S202 is input, and after the network parameters and hyperparameters are set, training can be performed. The output result is compared with the confidence score. After training, a model for target frame review is obtained.

[0048] S03: Based on the previous two steps, we obtain two network models, namely, target box pre-extraction and target box confidence analysis, to perform actual auxiliary labeling for target detection, calculate feature conformity to obtain label box confidence, provide initial review suggestions, preliminarily select labels, and after manual labeling, provide a label review module based on the manual labeling results and image features, provide label score ranking, and assist in review;

[0049] The specific method is:

[0050] S301: Input the original image, obtain the initial target frame and the original image through S01, and then generate the target slice according to the target frame. Obtain the target contour information in the target frame based on the segmentation model (such as SAM), compare the contour information with the target frame information, and eliminate the results with large differences between the contour information and the target frame. The contour information comparison includes the intersection and union ratio of the contour envelope and the target frame, the Euclidean distance of the center point, and the contour area. After eliminating the target frame that does not meet the requirements, the final pre-labeling result is obtained.

[0051] S302: Input the target detection frame and sliced ​​images obtained by slicing the image into the confidence detection model trained in S02 in batches to obtain the corresponding confidence ranking, and remove unqualified labeled samples according to the ranking, such as Figure 3 The figure shows the initial annotation boxes and confidence rankings;

[0052] S303: Performing manual correction on the filtered images and annotation frames, and supplementing category information;

[0053] S304: Finally, the completed labeling results are checked based on the model trained in S02, and after confirmation, they are saved and stored to obtain a new target detection sample.

Claims

1. A semi-automatic annotation and semi-automatic review method based on AI big model, characterized in that: The following steps are involved: S01: Establish a pre-extraction model for annotation boxes to provide initial annotation hints for the annotation process; S02: Based on the self-supervised learning method, the target detection box confidence detection model is trained; S03: Based on the model obtained in S01 and S02, the image is input into the annotation box pre-extraction model to obtain the annotation box of the image. After the annotation box is sliced, it is input into the target detection box confidence detection model to obtain the annotation box confidence, provide initial review suggestions and preliminary screening labels, and after manual labeling, calculate the annotation box confidence again based on the manual labeling results and image features, provide label score ranking, and obtain the corrected data set results.

2. According to claim 1, a semi-automatic annotation and semi-automatic review method based on an AI large model is characterized in that: In S01, it includes: S101: obtaining a target prompt box and a confidence level of the image based on a plurality of deep target detection models according to the result of the preprocessing of the input image; S102: Based on the obtained target prompt frame, maximum suppression of the target frame is performed to obtain a final foreground target detection frame; S103: Screen out detection targets that are obviously not in compliance based on the side length and area information of the target detection frame.

3. According to the semi-automatic labeling and semi-automatic review method based on AI big model in claim 1, it is characterized in that: In step S02, it includes: S201: Based on various public target detection datasets, first generate target frame image slices; S202: Obtain the target frame image slice and the annotation frame coordinate information in S201, generate the reference confidence of the detection frame in combination with the feature information, calculate the IOU as the reference confidence of the target frame, and thus obtain a usable training data set; S203: Performing network training based on the image slices, annotation boxes, and reference confidences in S202, wherein the network input is the image slices and annotation boxes, and the output is the reference confidence, and the audit network model is obtained through training; In the training process of S203, the target frame coordinate information is introduced: the image obtains a feature map F after passing through the basic network framework, the feature map F is cropped based on the target frame, the feature maps before and after cropping are downsampled to obtain feature vectors of equal length, and the two feature vectors are merged and passed into the network as new features.

4. According to claim 1, a semi-automatic annotation and semi-automatic review method based on AI big model is characterized in that: In S03, it includes: S301: input the target detection frame and the original image obtained in S01 to generate a target slice; S302: Based on the large segmentation model, the target detection frame and the target slice are used as the basis to obtain the contour information of the target in the image, and the target frames that do not meet the requirements are screened based on the contour information, and then the unqualified labeled samples are screened out based on the target detection frame confidence detection model trained in S02; S303: Performing manual correction on the filtered images and annotation frames, and supplementing category information; S304: Check the target detection box confidence detection model trained in S02, and save it into the database to form a new data set after it is correct.

Images