A data labeling method, apparatus, device, and medium

Abstract

The application relates to the technical field of image processing, in particular to a data labeling method and device, equipment and medium, which are used for improving data labeling efficiency. The method comprises the following steps: determining the category of a target in a first image based on a plurality of first images in a first data set; determining whether a second data set exists, the second data set comprising a plurality of second images with labels, the labels being used for indicating the position of the target in the second images and / or the category of the target, and the category of the target in the second images being the same as that in the first images; if the second data set exists, inputting the first image into a trained target detection model for prediction to obtain the label of the first image, and the trained target detection model being obtained by training the target detection model through the second data set. The method uses the second data set of similar scenes to quickly obtain the trained target detection model, and the label of an unlabeled image is predicted through the model, so that the data labeling efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, specifically to a data annotation method, apparatus, device, and medium. Background Technology

[0002] Object detection (OD) has wide applications across various industries. It involves locating, selecting bounding boxes, and performing preliminary type identification of objects in an image. Currently, object detection is primarily achieved through data-driven deep learning methods; therefore, the size and quality of the labeled dataset are crucial to the detection performance of the object detection model.

[0003] To ensure the generalization ability of object detection models, expanding the data scale is essential. Traditional brute-force annotation methods rely entirely on manual annotation by personnel, a time-consuming and labor-intensive process. Existing naive semi-automatic annotation methods first manually annotate a portion of the dataset, then train a usable model using this manually annotated data. For example, if the model's accuracy and recall are both above 65%, it is considered usable. This usable model is then used to predict the remaining unlabeled data. However, to achieve a usable model, a large amount of data still needs to be manually annotated, resulting in low annotation efficiency for the entire dataset. Summary of the Invention

[0004] This application provides a data annotation method, apparatus, device, and medium to improve data annotation efficiency.

[0005] Firstly, this application provides a data annotation method, including:

[0006] Based on multiple first images in the first dataset, determine the category of the target in the first image;

[0007] Determine whether a second dataset exists; the second dataset includes multiple labeled second images, the labels indicating the location of a target in the second image and / or the category of the target, the category of the target in the second image being the same as the category of the target in the first image;

[0008] If a second dataset exists, the first image is input into the trained object detection model for prediction to obtain the label of the first image; wherein, the trained object detection model is obtained by training the object detection model using the second dataset.

[0009] In this embodiment, the target detection model is trained by making full use of the labeled second images in the second dataset. The trained target detection model can be obtained quickly. The trained target detection model can be used to predict the unlabeled first images in the first dataset, thereby quickly obtaining the labels of the first images. No manual annotation is required, which reduces the workload of manual annotation and improves the efficiency of data annotation.

[0010] In one possible embodiment, before inputting the first image into a trained object detection model for prediction and obtaining the label of the first image, the method further includes:

[0011] Feature extraction is performed on any second image to obtain the features of the second image, and prediction is performed on the features of the second image to obtain the prediction result of the second image.

[0012] Feature extraction is performed on any first image to obtain the features of the first image;

[0013] The parameters of the target detection model are adjusted based on the first error between the features of any first image and the features of any second image, and the second error between the prediction result of any second image and the corresponding label.

[0014] The training of the target detection model is obtained by continuing until the maximum number of iterations is reached.

[0015] In this embodiment, the parameters of the object detection model are adjusted by the first error between the image features of different datasets, so that the first image of the first dataset and the second image of the second dataset maintain a consistent distribution in the latent feature space, thereby enabling the object detection model trained on the second dataset to have better transfer ability (also known as generalization ability) on the first dataset.

[0016] In one possible embodiment, the first image is input into a trained object detection model for prediction to obtain a label for the first image, including:

[0017] If the trained object detection model does not reach the usability index value, then manually labeled partial images in the first dataset are obtained; wherein, the first dataset includes the partial images and the remaining unlabeled images;

[0018] Based on the aforementioned partial images, the parameters of the trained target detection model are adjusted until the adjusted target detection model reaches the usability index value.

[0019] The remaining images are input into the adjusted target detection model for prediction to obtain the labels of the remaining images.

[0020] In this embodiment, the parameters of the trained target detection model are adjusted by manually annotating a small number of images, so that the trained target detection model can reach the usability index value, which can reduce the prediction error of the subsequent target detection model.

[0021] In one possible embodiment, after determining whether a second dataset exists, the method further includes:

[0022] If the second dataset does not exist, then the manually labeled partial images in the first dataset are obtained; wherein, the first dataset includes the partially labeled images and the remaining unlabeled images;

[0023] The remaining images are input into the trained target detection model for prediction to obtain the labels of the remaining images; wherein, the trained target detection model is obtained by training the target detection model using the partial images and the remaining images.

[0024] In one possible embodiment, the remaining image is input into the trained object detection model for prediction to obtain the label of the remaining image, including:

[0025] If the trained target detection model does not reach the usability index value, then perform Fourier transform on the partial image to obtain the first frequency map and the first phase map, and perform Fourier transform on the remaining image to obtain the second frequency map and the second phase map.

[0026] The mean amplitude-frequency map is obtained by averaging the first amplitude-frequency map and the second amplitude-frequency map, and the inverse Fourier transform is performed on the mean amplitude-frequency map and the first phase map to obtain the enhanced partial image.

[0027] Based on the enhanced partial image, the trained target detection model is adjusted until the adjusted target detection model reaches the usability index value.

[0028] The remaining images are input into the adjusted target detection model for prediction to obtain the labels of the remaining images.

[0029] In this embodiment, considering that labeled and unlabeled data are completely independent and identically distributed, but in reality, labeled data only represents a small portion, and there are always discrepancies in their distributions, feature enhancement is performed on the labeled portion by mixing the amplitude-frequency maps of the labeled portion of the images and the remaining unlabeled images. The enhanced portion of the images contains some features of the remaining unlabeled images, but the main features remain unchanged, making the amplitude-frequency of the labeled data closer to that of the unlabeled data, while not losing the category and location semantics of the object detection. As a data augmentation method, this can also improve the generalization ability of the trained object detection model.

[0030] In one possible embodiment, the remaining image is input into the adjusted target detection model for prediction to obtain the label of the remaining image, including:

[0031] The remaining images are divided into multiple image subsets;

[0032] When i equals 1, each image in the i-th image subset is input into the adjusted target detection model for prediction, and the label of each image in the i-th image subset is obtained.

[0033] In one possible embodiment, after dividing the remaining image into multiple image subsets, the method further includes:

[0034] When i is greater than or equal to 2, obtain the manually corrected images with correct labels in the (i-1)th image subset, and the index values ​​corresponding to the (i-1)th image subset;

[0035] If the index value corresponding to the (i-1)th image subset does not reach the saturation index value, then the (i-1)th incremental model is trained based on the manually corrected image with the correct label to obtain the i-th incremental model; wherein, when i equals 2, the (i-1)th incremental model is the adjusted target detection model, and the saturation index value is greater than the usability index value.

[0036] Each image in the i-th image subset is input into multiple incremental models for prediction, thereby obtaining multiple prediction results for each image in the i-th image subset; wherein, the multiple incremental models include the (i-1)-th incremental model and the i-th incremental model;

[0037] The labels of each image in the i-th image subset are obtained by weighted summation of multiple prediction results for each image.

[0038] In this embodiment, reinforcement learning is used, and each time only manually corrected images are used to train the (i-1)th incremental model of the previous stage to obtain the ith incremental model, which reduces the training time of the model and further improves the efficiency of data annotation.

[0039] In one possible embodiment, before inputting each image in the i-th image subset into multiple incremental models for prediction and obtaining multiple prediction results for each image in the i-th image subset, the method further includes:

[0040] Sample images are extracted from the partial images, and the sample images are respectively input into the multiple incremental models for prediction to obtain multiple prediction results for the sample images;

[0041] Based on multiple prediction results and corresponding labels of the sample image, multiple index values ​​corresponding to the sample image are obtained;

[0042] The labels of each image in the i-th image subset are obtained by weighted summation of multiple prediction results for each image, including:

[0043] Based on the multiple index values, the multiple prediction results of each image in the i-th image subset are weighted and summed to obtain the label of each image in the i-th image subset.

[0044] In this embodiment, the indicators of each underfitting incremental model are evaluated on the labeled image, and the estimated indicator values ​​are used as weight coefficients. When predicting unlabeled images, the outputs of each incremental model are weighted and voted on using the weight coefficients as the final output. This can reduce the impact of the underfitting state of the incremental model and make full use of the incremental models obtained at each stage.

[0045] Secondly, this application provides a data annotation device, comprising:

[0046] The determination module is used to determine the category of the target in the first image based on multiple first images in the first dataset;

[0047] The determining module is used to determine whether a second dataset exists; the second dataset includes multiple labeled second images, the labels being used to indicate the location of the target in the second image and / or the category of the target, the category of the target in the second image being the same as the category of the target in the first image;

[0048] The module is configured to, if a second dataset does not exist, input the first image into the trained object detection model for prediction and obtain the label of the first image; wherein the trained object detection model is obtained by training the object detection model using the second dataset.

[0049] In one possible embodiment, the obtaining module is specifically used for:

[0050] Before inputting the first image into the trained target detection model for prediction and obtaining the label of the first image, feature extraction is performed on any second image to obtain the features of the second image, and prediction is performed on the features of the second image to obtain the prediction result of the second image.

[0051] Feature extraction is performed on any first image to obtain the features of the first image;

[0052] The parameters of the target detection model are adjusted based on the first error between the features of any first image and the features of any second image, and the second error between the prediction result of any second image and the corresponding label.

[0053] The training of the target detection model is obtained by continuing until the maximum number of iterations is reached.

[0054] In one possible embodiment, the obtaining module is specifically used for:

[0055] If the trained object detection model does not reach the usability index value, then manually labeled partial images in the first dataset are obtained; wherein, the first dataset includes the partial images and the remaining unlabeled images;

[0056] Based on the aforementioned partial images, the parameters of the trained target detection model are adjusted until the adjusted target detection model reaches the usability index value.

[0057] The remaining images are input into the adjusted target detection model for prediction to obtain the labels of the remaining images.

[0058] In one possible embodiment, the obtaining module is further configured to:

[0059] After determining whether a second dataset exists, if the second dataset does not exist, then the manually labeled portion of the images in the first dataset is obtained; wherein, the first dataset includes the labeled portion of the images and the remaining unlabeled images;

[0060] The remaining images are input into the trained target detection model for prediction to obtain the labels of the remaining images; wherein, the trained target detection model is obtained by training the target detection model using the partial images and the remaining images.

[0061] In one possible embodiment, the obtaining module is specifically used for:

[0062] If the trained target detection model does not reach the usability index value, then perform Fourier transform on the partial image to obtain the first frequency map and the first phase map, and perform Fourier transform on the remaining image to obtain the second frequency map and the second phase map.

[0063] The mean amplitude-frequency map is obtained by averaging the first amplitude-frequency map and the second amplitude-frequency map, and the inverse Fourier transform is performed on the mean amplitude-frequency map and the first phase map to obtain the enhanced partial image.

[0064] Based on the enhanced partial image, the trained target detection model is adjusted until the adjusted target detection model reaches the usability index value.

[0065] The remaining images are input into the adjusted target detection model for prediction to obtain the labels of the remaining images.

[0066] In one possible embodiment, the obtaining module is specifically used for:

[0067] The remaining images are divided into multiple image subsets;

[0068] When i equals 1, each image in the i-th image subset is input into the adjusted target detection model for prediction, and the label of each image in the i-th image subset is obtained.

[0069] In one possible embodiment, after dividing the remaining image into multiple image subsets, the method further includes:

[0070] When i is greater than or equal to 2, obtain the manually corrected images with correct labels in the (i-1)th image subset, and the index values ​​corresponding to the (i-1)th image subset;

[0071] If the index value corresponding to the (i-1)th image subset does not reach the saturation index value, then the (i-1)th incremental model is trained based on the manually corrected image with the correct label to obtain the i-th incremental model; wherein, when i equals 2, the (i-1)th incremental model is the adjusted target detection model, and the saturation index value is greater than the usability index value.

[0072] Each image in the i-th image subset is input into multiple incremental models for prediction, thereby obtaining multiple prediction results for each image in the i-th image subset; wherein, the multiple incremental models include the (i-1)-th incremental model and the i-th incremental model;

[0073] The labels of each image in the i-th image subset are obtained by weighted summation of multiple prediction results for each image.

[0074] In one possible embodiment, the obtaining module is further configured to:

[0075] Before inputting each image in the i-th image subset into multiple incremental models for prediction and obtaining multiple prediction results for each image in the i-th image subset, sample images are extracted from the partial images, and the sample images are input into the multiple incremental models for prediction to obtain multiple prediction results for the sample images.

[0076] Based on multiple prediction results and corresponding labels of the sample image, multiple index values ​​corresponding to the sample image are obtained;

[0077] The labels of each image in the i-th image subset are obtained by weighted summation of multiple prediction results for each image, including:

[0078] Based on the multiple index values, the multiple prediction results of each image in the i-th image subset are weighted and summed to obtain the label of each image in the i-th image subset.

[0079] Thirdly, this application provides an electronic device, comprising:

[0080] Memory, used to store program instructions;

[0081] A processor is configured to invoke program instructions stored in the memory and execute the method described in any one of the first aspects according to the obtained program instructions.

[0082] Fourthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the method described in any one of the first aspects. Attached Figure Description

[0083] Figure 1 A graph showing the relationship between data size and accuracy provided in the embodiments of this application;

[0084] Figure 2 This is a flowchart of a naive semi-automatic annotation method in existing technology;

[0085] Figure 3 This is a graph showing the time consumption percentage of each stage in the existing naive semi-automatic annotation method.

[0086] Figure 4 This application provides an illustration of an data annotation method for a specific embodiment of this application.

[0087] Figure 5 A flowchart of a data annotation method provided in this application embodiment Figure 1 ;

[0088] Figure 6 A schematic diagram of the structure of a target detection model provided in an embodiment of this application. Figure 1 ;

[0089] Figure 7 A flowchart of a transfer alignment training algorithm provided in an embodiment of this application;

[0090] Figure 8 A schematic diagram of the structure of a target detection model provided in an embodiment of this application. Figure 2 ;

[0091] Figure 9 A schematic diagram of a domain mixing method provided in an embodiment of this application;

[0092] Figure 10 A flowchart of a weighted voting algorithm provided in an embodiment of this application;

[0093] Figure 11 A flowchart of a data annotation method provided in this application embodiment Figure 2 ;

[0094] Figure 12 A flowchart of data annotation method 1 provided in the embodiments of this application;

[0095] Figure 13 A flowchart of data annotation method 2 provided in the embodiments of this application;

[0096] Figure 14 A time consumption percentage diagram for each stage of the data annotation method 1 provided in the embodiments of this application;

[0097] Figure 15 A structural diagram of a data annotation device provided in an embodiment of this application;

[0098] Figure 16 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0099] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0100] Before introducing the data annotation method provided in the embodiments of this application, the background technology of the embodiments of this application will be described in detail for ease of understanding.

[0101] The performance of an object detection model can be represented by metrics such as accuracy and recall. Taking accuracy as an example, please refer to... Figure 1, which is a relationship curve graph between the data scale and the accuracy rate provided by the embodiments of this application. It can be seen that as the data scale increases, the accuracy rate of the target detection model significantly improves. When the data scale reaches the available scale, the accuracy rate reaches 65%, and at this time, the target detection model has reached the available level, that is, it has certain usability. When the data scale reaches the saturation scale, the accuracy rate reaches 95%, and the subsequent improvement of the accuracy rate of the target detection model is no longer obvious.

[0102] Please refer to Figure 2 , which is a flowchart of the naive semi-automatic annotation method in the prior art. The following combines Figure 2 to introduce the specific steps of the naive semi-automatic annotation method. The process starts, and first, S201 is executed.

[0103] S201. Initial annotation stage S0: Divide the remaining dataset into N parts and initialize i = 1.

[0104] In this stage, it is necessary to manually annotate part of the data in the dataset to be annotated to ensure that the target detection model reaches the available level. Among them, the remaining dataset refers to the unannotated data in the dataset to be annotated. Divide the remaining dataset into D1, D2 ….. D ,

[0108] ,

[0113] ,

[0109] ,

[0106] ,

[0111] ,

[0107] , i ,

[0112] ,

[0110] , , , i , , , i , , , and other N data subsets.

[0105] S202. Model training stage S1 i .

[0106] In this stage, the target detection model is trained with the annotated part of the data to obtain a target detection model at the available level.

[0107] S203. Prediction stage S2 i : Predict the i-th data subset.

[0108] In this stage, the data in the i-th data subset D i is predicted by the target detection model at the available level to obtain the annotated data.

[0109] S204. Correction stage S3 i : Manual inspection and correction.

[0110] In this stage, it is necessary to interact with the annotator to manually annotate and correct or add "difficult cases" such as misdetection, false detection, and missed detection in each image.

[0111] S205. Judge whether i is less than N.

[0112] N represents the total number of divided data subsets. If i < N, after executing S206, continue to execute S202. If i ≥ N, the process ends, and the annotation work of the entire dataset to be annotated is completed.

[0113] S206, i = i + 1.

[0114] For the same amount of annotation tasks, although the naive semi-automatic annotation method reduces the overall time consumption compared to the brute-force annotation method, there is still a lot of annotation work that needs to be done manually before the data scale reaches a usable scale. Therefore, the initial annotation stage S0 takes a long time, and the model training stage S1 requires continuous retraining of the model, which is also a time-consuming process.

[0115] Please refer to Figure 3 The figure shows the time consumption percentage of each stage in the existing naive semi-automatic annotation method. The initial annotation stage (S0) accounts for 31.4% of the time, the model training stage (S1) accounts for 36.9%, the prediction stage (S2) accounts for 7.3%, and the correction stage (S3) accounts for 24.4%. The time consumption of the prediction stage (S2) and the correction stage (S3) is difficult to compress, while the combined time consumption of the initial annotation stage (S0) and the model training stage (S1) is close to 70%, indicating significant room for optimization.

[0116] To improve data annotation efficiency, this application provides a data annotation method that can be executed by an annotation device, which can be implemented through a terminal or a server. The terminal can be a mobile terminal, a fixed terminal, or a portable terminal, such as a smart camera, mobile phone, multimedia computer, multimedia tablet, desktop computer, laptop computer, or tablet computer. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms, but is not limited to these.

[0117] The following is a brief introduction to the application scenarios to which the technical solutions of the embodiments of this application are applicable. Please refer to... Figure 4 This diagram illustrates an application scenario of a data annotation method provided in this application. The application scenario may include a dataset 401 to be annotated and an annotation device 402. After acquiring the dataset 401, the annotation device 402 annotates the images in the dataset 401 to obtain an annotated dataset. The specific process by which the annotation device 402 annotates the dataset 401 will be described below.

[0118] The application scenarios have been introduced above; the following will combine... Figure 4 The application scenarios shown are as follows: Figure 4 This section uses the data annotation method performed by annotation device 402 as an example to illustrate the process. Please refer to... Figure 5 The flowchart of a data annotation method provided in this application embodiment Figure 1 .

[0119] S501. Based on multiple first images in the first dataset, determine the category of the target in the first image.

[0120] The new dataset to be labeled can be called the first dataset. The first dataset includes multiple unlabeled first images. The first dataset can be collected by other camera devices and sent to the labeling device, or it can be collected by the labeling device itself, for example, if the labeling device is a device with camera capabilities. After obtaining the first dataset, the labeling device determines the category of the target in the first image. For example, the labeler inputs the category of the target to be labeled in the first dataset as pedestrian.

[0121] S502. Determine if a second dataset exists.

[0122] After obtaining the first dataset, the annotation device searches for a second dataset of the same category based on the target category in the first image. The second dataset can be downloaded from online resources or saved by the annotation device after completing a previous annotation task. The second dataset includes multiple labeled images, with labels indicating the location and / or category of the targets within the second images. The location of a target in the second image includes the bounding box containing the target, and may also include the positional information of the bounding box. Target categories include, for example, pedestrians, motor vehicles, non-motorized vehicles, and faces. The second dataset can also be called a transferable dataset, where the target category in the second image is the same as the target category in the first image; for example, the target category in both the second and first images is "face."

[0123] S503. If a second dataset exists, input the first image into the trained object detection model for prediction to obtain the label of the first image.

[0124] If a second dataset exists, the labeling device can train the object detection model based on the second dataset to obtain the trained object detection model. The first image is then input into the trained object detection model for prediction to obtain the label of the first image.

[0125] Specifically, the annotation equipment contains pre-established target detection models; please refer to [the relevant documentation]. Figure 6 The diagram below illustrates the structure of a target detection model provided in this application embodiment. Figure 1The object detection model includes a backbone module 601, a prediction (neck, head) module 602, and a domain aligner module 603. The first and second datasets use the same backbone module 601 for feature extraction; in other words, the backbone module 601 extracts features from both the first and second images. The prediction module 602 obtains features from the second image and outputs the prediction result for the second image. The domain aligner module 603 obtains features from both the first and second images and calculates the differences between the features extracted by the backbone module 601 for the different datasets.

[0126] The following is combined Figure 6 The object detection model shown illustrates the process of training the object detection model using the labeled device based on the second dataset.

[0127] S1.1. Extract features from any second image to obtain the features of any second image, and predict the features of any second image to obtain the prediction result of any second image.

[0128] The annotation device inputs any second image into the backbone module 601 for feature extraction and outputs the features of any second image. It also inputs the features of any second image into the prediction module 602 for prediction and outputs the prediction result of any second image.

[0129] S1.2. Extract features from any first image to obtain the features of any first image.

[0130] The annotation device inputs any first image into the backbone module 601 for feature extraction and outputs the features of any first image.

[0131] S1.3 Adjust the parameters of the target detection model based on the first error between the features of any first image and the features of any second image, and the second error between the prediction result of any second image and the corresponding label.

[0132] The annotation device can input the features of any first image and the features of any second image into the domain alignment module 603 to calculate the feature difference, and output a first error between the features of the first image and the features of the second image. The annotation device can also calculate a second error between the prediction result of any second image and the corresponding label, and adjust the parameters of the backbone module 601 and the prediction module 602 according to the first error and the second error, and adjust the parameters of the domain alignment module 603 according to the second error.

[0133] S1.4 Continue until the maximum number of iterations is reached to obtain the trained object detection model.

[0134] Until the maximum number of iterations is reached, the annotation device uses the backbone module 601 and the prediction module 602 as the trained object detection model. It should be noted that the source domain, i.e., the second image and its corresponding label in the second dataset, can drive the object detection model to learn a better feature representation on the second dataset. To further ensure that the model trained on the second dataset has good transfer capability on the target domain, i.e., the first dataset, it is necessary to ensure that the data from the second dataset and the data from the first dataset maintain a consistent distribution in the latent feature space. A direct approach is to use the difference in probability distribution functions as the loss function for the alignment process, thereby achieving domain alignment. Common methods for calculating distribution differences include KL divergence and JS divergence.

[0135] The first error in the embodiments of this application can be the JS divergence. The formula for calculating the JS divergence is as follows:

[0136]

[0137] Where p(x) is the probability density function of the source domain, i.e., the second dataset, and q(x) is the probability density function of the target domain, i.e., the first dataset.

[0138] Since the probability density functions of the first and second datasets are unknown, the difference in data distribution between the two datasets cannot be directly calculated. However, both datasets have large sample sizes, and their equivalent forms of JS divergence can be implicitly calculated through sampling.

[0139] Assuming that the first and second datasets share a latent feature space, the equivalent form of the JS divergence within that space is as follows:

[0140]

[0141] Among them, P s (x) is the probability density function of the source domain, i.e., the second dataset, P t (x) is the probability density function of the target domain, i.e., the first dataset.

[0142] To estimate the JS divergence, an objective function V(D) is defined to be optimized. aligner D backbone )as follows:

[0143] V(D aligner D backbone )=∫ x p s (x)logD aligner (D backbone (x))+p t (x)log(1-Daligner (D backbone (x)))dx

[0144] Among them, D aligner The domain alignment module 603 discussed earlier is a neural network that is ultimately normalized to [0,1] using a function similar to softmax, thus representing an implicit probabilistic model. D backbone (x) represents the characteristic output of the backbone module 601 discussed above. P s (x) and P t The meaning of (x) is as discussed above and will not be repeated here.

[0145] Sampling can transform the integral into the expectation expression:

[0146]

[0147] in, This represents the expected value of the source domain, i.e., the second dataset. This represents the expected value of the target domain, i.e., the first dataset. D aligner and D backbone The meaning of (x) is as discussed above and will not be repeated here.

[0148] The expected value can be obtained by repeatedly sampling from the first and second datasets, thus obtaining V(D). aligner D backbone Estimates of JS divergence and V(D). aligner D backbone There is a correlation, keep D backbone Fixed, specific, for V(D) aligner D backbone To find the maximum value, it's easy to get:

[0149]

[0150] in, V(D) represents aligner D backbone The maximum value of p) s (D backbone (x) represents the features of the second image sampled from the source domain, i.e., the second dataset, p t (D backbone (x) represents the features of the first image sampled from the target domain, i.e., the first dataset.

[0151] At this point, the JS divergence can be derived from... To indicate:

[0152]

[0153] in, and D backbone The meaning of "is as discussed above and will not be repeated here."

[0154] Therefore, as long as the backbone module 0601 is fixed and the domain alignment module 603 is updated in the feature space, the maximized V(D) can be calculated. aligner D backbone The JS divergence can then be estimated. The JS divergence is used as a loss function to guide the alignment of the distributions of the second and first datasets in the feature space.

[0155] To reduce the differences between features extracted by the backbone module from different datasets, this application provides a transfer alignment training algorithm. Its core idea is to align the feature space at the feature level extracted by the backbone module. Please refer to... Figure 7 This is a flowchart of a transfer alignment training algorithm provided in an embodiment of this application. First, the process begins, executing S701.

[0156] S701. Set the maximum number of iterations I, initialize i = 0, set the number of JS divergence iterations M, and align the weights a.

[0157] S702. Obtain Xs_i from the source domain and Xt_i from the target domain.

[0158] The annotation device can sample a second image Xs_i from the source domain, i.e., the second dataset, and sample a first image Xt_i from the target domain, i.e., the first dataset.

[0159] S703. Calculate Loss_od_i from Xs_i and Ys_i, and calculate Loss_JS_pos_i from Xs_i and Xt_i. Update the backbone module and prediction module using stochastic gradient backpropagation based on Loss_i = Loss_od_i + a*Loss_JS_pos_i.

[0160] The annotation device can calculate Loss_i based on Loss_JS_pos_i and Loss_od_i, and update the backbone module 601 and prediction module 602 based on the stochastic gradient backpropagation of Loss_i. The formula for calculating Loss_i is as follows:

[0161] Loss_i=Loss_od_i+a*Loss_JS_pos_i

[0162] Where Loss_JS_pos_i represents the first error between the features of the second image Xs_i and the features of the first image Xt_i, Loss_od_i represents the second error between the prediction result of the second image Xs_i and the corresponding label, and a represents the alignment weight.

[0163] S704. Initialize m = 0.

[0164] S705. Sample Xs_j from the source domain and Xt_j from the target domain, calculate Loss_JS_neg for backpropagation, and freeze the backbone module while only updating the domain alignment module.

[0165] The annotation device can resample a second image Xs_j from the source domain, i.e., the second dataset, and a first image Xt_j from the target domain, i.e., the first dataset. Input the second image Xs_j and the first image Xt_j into the backbone module 602 to extract features, and then input the features of the second image Xs_j and the first image Xt_j into the domain alignment module 603 for feature difference calculation to obtain Loss_JS_neg. Only update the parameters of the domain alignment module 603 according to Loss_JS_neg.

[0166] S706. Determine whether M is less than m.

[0167] M is the set number of JS divergence iterations. If M < m, execute S708. If M ≥ m, execute S707 and then continue to execute S705.

[0168] S707. m = m + 1.

[0169] S708. Determine whether i is greater than I.

[0170] The annotation device can determine whether the iteration number i of the backbone module 601 and the prediction module 602 is greater than the maximum iteration number I. If i > I, the process ends. If i ≤ I, execute S709 and then continue to execute S702.

[0171] S709. i = i + 1.

[0172] After S701 - S709, when i > I, the training ends, and the annotation device can use the backbone module 601 and the prediction module 602 as the trained object detection model.

[0173] In the embodiment of the present application, every time the backbone module 601 and the prediction module 602 are iterated once, the domain alignment module 603 is iterated m times, and this process repeats in turn. Iterating the domain alignment module 603 m times can be used to calculate the JS divergence more accurately, so as to ensure that the first error loss_JS_pos on which one iteration of the object detection model corresponding to the backbone module 601 and the prediction module 602 depends has the effect of guiding feature alignment, and a usable object detection model can be obtained more quickly, thus greatly reducing the time consumption in the training stage.

[0174] The above describes how to obtain a trained object detection model based on the second dataset. Furthermore, after obtaining the trained object detection model, the labeling device can directly input all the first images in the first dataset into the trained object detection model for prediction to obtain the labels of the first images.

[0175] Considering that the trained object detection model may not have reached a usability level and may have a large prediction error, in one possible embodiment, the annotation device may first determine whether the trained object detection model has reached a usability metric value. Based on the determination result, it may decide whether to fine-tune the trained object detection model. Here, the usability metric value refers to the value of a preset metric when the object detection model reaches a usability level. A usability level refers to the model having a certain level of usability. The preset metric can be precision, recall, the mean of precision and recall, the product of precision and recall, etc. For example, the usability metric value could be a precision of 65%, or a mean of precision and recall of 65%, or a precision of 65% and a recall of 65%.

[0176] Specifically, the annotation device can divide the second dataset into a training set and a test set. The object detection model is trained using the second images from the training set, and then tested using the second images from the test set to obtain the model's metric value. If the model's metric value is greater than or equal to the usability metric value, the trained object detection model is considered to have reached the usability metric value; if the model's metric value is less than the usability metric value, the trained object detection model is considered to have not reached the usability metric value.

[0177] For example, the usability metric is defined as an accuracy of 65% and a recall of 65%. A labeling device tests the trained object detection model to obtain its accuracy and recall. If both accuracy and recall are greater than 65%, the model is considered to have met the usability metric; otherwise, it is considered not to have met the usability metric.

[0178] In one possible embodiment, if the trained object detection model fails to reach the usability metric, the annotation device can acquire manually annotated portions of the images in the first dataset. For example, annotators manually annotate a portion of the images in the first dataset and input them into the annotation device. Thus, the first dataset includes manually annotated images and the remaining unannotated images. Based on these images, the annotation device adjusts the parameters of the trained object detection model and continues to test whether the adjusted model reaches the usability metric until it does. Then, the remaining images are input into the adjusted model for prediction to obtain labels for the remaining images.

[0179] In one possible embodiment, if the trained object detection model reaches the usability index value, then there is no need for manual image labeling. The labeling device can directly input all the first images sequentially into the trained object detection model for prediction to obtain the labels of the first images.

[0180] It should be noted that S503 is optional. If the second dataset does not exist, S503 is not executed. In this case, the annotation device can acquire the manually annotated portion of the images in the first dataset. For example, annotators manually annotate a portion of the images in the first dataset and input them into the annotation device. Thus, the first dataset includes the partial images and the remaining unannotated images. The remaining images are then input into the trained object detection model for prediction to obtain the labels for the remaining images. The trained object detection model is obtained by training on the partial and remaining images.

[0181] Please refer to Figure 8 The diagram below illustrates the structure of a target detection model provided in this application embodiment. Figure 2 The object detection model includes a backbone module 801, a prediction (neck / head) module 802, and an autoencoder module 803. The backbone module 801 extracts image features. The prediction module 802 acquires image features and outputs the predicted result. The autoencoder module 803 is an autoencoder network used to reconstruct the image based on its features and output the reconstructed image.

[0182] There are two methods for training object detection models based on partial and remaining images, which will be discussed below. Figure 8 The structure of the target detection model shown will be introduced separately.

[0183] The first method involves updating the backbone module 801 and the autoencoder module 803 using the remaining unlabeled images, and then updating the backbone module 801 and the prediction module 802 using the manually labeled partial images.

[0184] Specifically, the annotation device can input any remaining image into the backbone module 801 for feature extraction to obtain the features of that remaining image. The features of that remaining image are then input into the autoencoder module 803 for encoding to obtain the reconstructed image. The third error between the remaining image and the reconstructed image, i.e., the loss, is then used to determine the final image. recovery Adjust the parameters of the backbone module 801 and the self-encoding module 803.

[0185] Furthermore, the annotation device inputs any partial image into the backbone module 801 for feature extraction, obtaining the features of that partial image. These features are then input into the prediction module 802 for prediction, yielding the prediction result for that partial image. The fourth error, or Loss, between the prediction result of that partial image and its corresponding label is then used. od The parameters of the backbone module 801 and the prediction module 802 are adjusted until preset conditions are met, such as reaching the maximum number of iterations. The annotation device then uses the backbone module 801 and the prediction module 802 as the trained target detection model.

[0186] The second approach involves first updating the backbone module 801 and the autoencoder module 803 using the unlabeled remaining images, then using a domain fusion method on the manually labeled partial images and the unlabeled remaining images to obtain enhanced partial images, and finally using the enhanced partial images to update the backbone module 801 and the prediction module 802.

[0187] In this embodiment, a special domain blending method is introduced to enhance the features of the labeled image. Please refer to [link / reference]. Figure 9 The image below is a schematic diagram of a domain mixing method provided in an embodiment of this application. Figure 9 This section introduces the specific steps of the domain mixing method.

[0188] The annotation device performs a Fast Fourier Transform (FFT) on the manually annotated portion of the image to obtain a first frequency map and a first phase map. It also performs an FFT on the remaining unannotated image to obtain a second frequency map and a second phase map. The first and second frequency maps are then averaged to obtain a mean frequency map. Finally, an Inverse Fast Fourier Transform (IFFT) is performed on the mean frequency map and the first phase map to obtain the enhanced portion of the image.

[0189] Specifically, the manually labeled partial image consists of multiple images. The labeling device performs FFT channel-by-channel on each partial image and represents it as an amplitude-frequency map and a phase map, thus obtaining the first amplitude-frequency map and the first phase map of each partial image. The labeling device then performs FFT channel-by-channel on a randomly sampled remaining image, thus obtaining the second amplitude-frequency map and the second phase map of the randomly sampled remaining image. Keeping the first phase map of each partial image unchanged, the labeling device adds the first amplitude-frequency map of each partial image completely or partially to the second amplitude-frequency map of the corresponding channel of the randomly sampled remaining image, takes the average, and replaces the first amplitude-frequency map of each partial image with the average amplitude-frequency map. IFFT is then used to obtain the enhanced partial image.

[0190] Since the amplitude-frequency map reflects the energy proportion of different frequency components in the overall image, and the phase map reflects the spatial relative positions of various shapes and features in the image, small changes in the amplitude-frequency map will not cause significant visual changes to the image. In this embodiment, data enhancement can be achieved by performing domain mixing on the amplitude-frequency map.

[0191] The domain fusion method has been introduced above; the following section combines... Figure 8 The process of training the object detection model in the second method discussed above is introduced. The annotation device can input any remaining image into the backbone module 801 for feature extraction to obtain the features of that remaining image. The features of that remaining image are then input into the autoencoder module 803 for encoding to obtain the reconstructed image. The third error between the remaining image and the reconstructed image, i.e., the loss, is then calculated. recovery Adjust the parameters of the backbone module 801 and the self-encoding module 803.

[0192] Furthermore, the annotation device inputs any one of the enhanced partial images into the backbone module 801 for feature extraction, obtaining the features of that enhanced partial image. These features are then input into the prediction module 802 for prediction, obtaining the prediction result for that enhanced partial image. The fourth error, or Loss, between the prediction result of any enhanced partial image and its corresponding label is then used. od The parameters of the backbone module 801 and the prediction module 802 are adjusted until preset conditions are met, such as reaching the maximum number of iterations. The annotation device then uses the backbone module 801 and the prediction module 802 as the trained target detection model.

[0193] In one possible embodiment, if the trained object detection model does not reach the usability metric, the labeling device can use a domain fusion method to obtain enhanced partial images. Based on the enhanced partial images, the trained object detection model is adjusted until the adjusted model reaches the usability metric. The remaining images are then input into the adjusted model for prediction to obtain labels for the remaining images. The meaning of the usability metric and the process of obtaining the enhanced partial images are explained above and will not be repeated here.

[0194] The above describes how the annotation device obtains the trained object detection model and the adjusted object detection model in two cases: with and without a second dataset. Next, we will introduce the process of inputting the remaining images into the adjusted object detection model for prediction.

[0195] In one possible embodiment, the annotation device divides the remaining images into multiple image subsets and uses incremental learning to annotate and predict the images in each image subset. The incremental learning process is described below.

[0196] When i equals 1, each image in the i-th image subset is input into the adjusted target detection model for prediction to obtain the label of each image in the i-th image subset. When i is greater than or equal to 2, the process of predicting the label of each image in the i-th image subset is as follows:

[0197] S2.1 Obtain manually corrected images with correct labels from the (i-1)th image subset, and the corresponding index values ​​for the (i-1)th image subset.

[0198] After the labeling device predicts the labels for each image in the (i-1)th image subset, the labelers can manually check the accuracy of the prediction results for each image in the (i-1)th image subset and correct any incorrect predictions to ensure that the labels for each image in the (i-1)th image subset are correct. Based on the number of manually corrected images in the (i-1)th image subset and the total number of images in the (i-1)th image subset, the labeling device determines the corresponding metric value for the (i-1)th image subset, such as precision or recall.

[0199] S2.2 If the index value corresponding to the (i-1)th image subset does not reach the saturation index value, then the (i-1)th incremental model is trained based on the manually corrected image with the correct label to obtain the i-th incremental model.

[0200] Considering that the cost of incremental learning exceeds the cost of manual error correction when labeled data reaches a certain scale, a saturation metric is introduced. The saturation metric refers to the value of a preset indicator when the object detection model reaches a saturation level. Preset indicators include accuracy, recall, the mean of accuracy and recall, and the product of accuracy and recall. The saturation metric is greater than the usability metric. For example, if the usability metric is 65% accuracy, the saturation metric is 97.5% accuracy. When the model's accuracy is greater than 65%, the model is considered to have a certain degree of usability. When the model's accuracy is greater than 97.5%, incremental learning is considered unnecessary, and the prediction results contain only a very small number of biases. When i equals 2, the (i-1)th incremental model, i.e., the first incremental model, refers to the adjusted object detection model discussed earlier.

[0201] S2.3. Input each image in the i-th image subset into multiple incremental models for prediction, and obtain multiple prediction results for each image in the i-th image subset. Among them, the multiple incremental models include the (i-1)-th incremental model and the i-th incremental model.

[0202] S2.4. Perform a weighted summation of the multiple prediction results for each image in the i-th image subset to obtain the label of each image in the i-th image subset.

[0203] For example, the first image subset contains 100 images. After inputting them into the adjusted object detection model, 10 predictions are obtained. Annotators check and find 20 incorrect predictions. These 20 predictions are manually corrected, resulting in an accuracy of 80% for the first image subset and a saturation index of 97.5%. The annotation equipment trains the adjusted object detection model (the first incremental model) based on these 20 manually corrected images to obtain the second incremental model. Image A from the second image subset is then input into both the first and second incremental models, yielding two predictions for image A. These two predictions are weighted and summed to obtain the label for image A, and so on.

[0204] In the model training phase, the naive semi-automatic annotation method requires retraining the model periodically. However, in this embodiment, reinforcement learning is used, which trains the incremental model of the previous stage using only new samples each time, and only for 1 or 2 epochs. The purpose is to maintain the underfit of the model to the new samples and reduce the training time of the model.

[0205] To reduce the impact of underfitting, this application provides a weighted voting algorithm that sums the prediction results from multiple enhanced models using weighted methods. This algorithm is described below.

[0206] The annotation device extracts sample images from a subset of images, inputs each sample image into multiple incremental models for prediction, and obtains multiple prediction results for the sample images. Based on the multiple prediction results and corresponding labels, it obtains multiple index values ​​for the sample images. Based on these index values, it performs a weighted sum of the multiple prediction results for each image in the i-th image subset to obtain the label for each image in the i-th image subset. Here, the index values ​​refer to the values ​​of preset indicators, such as precision, recall, the mean of precision and recall, and the product of precision and recall.

[0207] Please refer to Figure 10 The following is a flowchart of a weighted voting algorithm provided in an embodiment of this application. First, the process begins, executing S1001.

[0208] S1001. Determine the model evaluation metric as metric, and initialize k=1.

[0209] Model evaluation metrics, such as the mean of accuracy and recall, are denoted as metric.

[0210] S1002, Extract some images from the labeled images, denoted as S.

[0211] Annotated images refer to all annotated images in the first dataset, including manually annotated images and manually corrected images.

[0212] S1003. Use S to evaluate the k-th incremental model and obtain the metric. k .

[0213] The extracted sample image S is input into the k-th incremental model for prediction. The metric value of the k-th incremental model is evaluated based on whether the predicted result of image S matches the label of image S. k The first incremental model refers to the adjusted target detection model discussed earlier.

[0214] S1004. Determine if k is greater than i.

[0215] i represents the number of incremental models that have been trained. If k > i, then execute S1006; if k ≤ i, then execute S1005 and then continue to execute S1002.

[0216] S1005, k = k + 1.

[0217] S1006. Calculate the voting weight for each incremental model.

[0218] a k =metric k / ∑metric

[0219] Among them, a k The metric represents the voting weight of the k-th incremental model. k Let ∑metric represent the index value of the k-th incremental model, and let ∑metric represent the sum of the index values ​​of all incremental models.

[0220] S1007. Calculate the weighted summation of multiple incremental models.

[0221] Out=∑a k ×out k = a1×out1+…+a k ×out k

[0222] Among them, a k Represents the voting weight of the k-th incremental model, out k represents the prediction result of the k-th incremental model, and Out represents the weighted sum of the prediction results of the k incremental models.

[0223] S1008. Output the target detection result after non-maximum suppression.

[0224] After the annotation device obtains the weighted summation results of multiple incremental models, it can use non-maximum suppression to output the final target detection result, i.e., the image label. Non-maximum suppression is within the scope of existing technology and will not be elaborated here.

[0225] After S1008 is executed, the process ends.

[0226] In summary, the annotation device in this application employs different data annotation methods depending on the presence or absence of a second dataset. Please refer to... Figure 11 The present application provides a data annotation method flowchart. Figure 2 The following is combined with Figure 11 This paper introduces the overall process of the data annotation method provided in the embodiments of this application.

[0227] The process begins by executing S1101.

[0228] S1101. Determine the category of the target to be labeled in the first dataset.

[0229] The first dataset refers to a brand new dataset that needs to be labeled.

[0230] S1102. Determine if a second dataset exists.

[0231] If it exists, proceed to step S1103. If it does not exist, proceed to step S1104. For the meaning of the second dataset and how to obtain it, please refer to the previous discussion; it will not be repeated here.

[0232] S1103. Use the data annotation method 1 proposed in this application.

[0233] Please refer to the data annotation method 1 proposed in this application. Figure 12 The content of the discussion.

[0234] S1104. Use the data annotation method 2 proposed in this application.

[0235] Please refer to the data annotation method 2 proposed in this application. Figure 13 The content of the discussion.

[0236] It should be noted that only one of S1103 and S1104 needs to be executed. After executing S1103 or S1104, the process ends.

[0237] Please refer to Figure 12 The above is a flowchart of the data annotation method 1 provided in this application embodiment. The specific steps of the data annotation method 1 proposed in this application are described below with reference to page 12.

[0238] S1201. Use the transfer alignment training algorithm to obtain the trained target detection model.

[0239] The annotation device, based on the second dataset, uses a transfer alignment training algorithm to train the object detection model, resulting in the trained object detection model. Please refer to the previous text for the steps of the transfer alignment algorithm. Figure 7 The content of the discussion will not be repeated here.

[0240] S1202. Determine whether the availability index value has been reached.

[0241] The annotation device determines whether the trained target detection model has reached the usability index value. The meaning of the usability index value and the specific judgment process are explained in the preceding text and will not be repeated here. If the usability index value is not reached, proceed to step S1203; if the usability index value is reached, proceed to step S1204.

[0242] S1203, Fine-tuning model of the labeled image.

[0243] The annotation device acquires manually annotated portions of the images in the first dataset and adjusts the parameters of the trained object detection model based on these portions until the adjusted object detection model reaches the usability index value.

[0244] S1204. Divide the remaining image into N parts and let i = 1.

[0245] The remaining images refer to the unlabeled images in the first dataset, excluding a subset of images.

[0246] S1205, Prediction Phase S2 i : Use a weighted voting algorithm for the i-th subset of images.

[0247] Please refer to the previous text for the steps of the weighted voting algorithm. Figure 10 The content of the discussion will not be repeated here.

[0248] S1206, Correction Phase S3 i Manual inspection and correction.

[0249] Annotators manually correct or add annotations for "difficult cases" such as false positives, false negatives, and missed positives in each image of the i-th image subset. Annotators can also highlight target boxes with low confidence, as the uncertainty of these boxes often leads to box bias and classification errors. For example, marking low-confidence target boxes or using specific colors to display target boxes can help annotators quickly locate suspicious targets.

[0250] S1207. Determine whether the saturation index value has been reached.

[0251] For the meaning of the saturation index value and the judgment process, please refer to the content described above, which will not be elaborated here. If the saturation index value is not reached, execute S1208; if the saturation index value is reached, directly execute S1209.

[0252] S1208. Model training stage S1 i : Use the incremental learning method to train on the (i - 1)-th incremental model with new samples for 1 - 2 epochs to obtain the i-th incremental model.

[0253] The new samples refer to the images with correct labels corrected manually in the correction stage. For the process of incremental learning, please refer to the content described above, which will not be elaborated here.

[0254] S1209. Judge whether i is greater than or equal to N.

[0255] N is the total number of image subsets. If i < N, after executing S1210, continue to execute S1205. If i ≥ N, the process ends.

[0256] S1210. i = i + 1.

[0257] When there is no second dataset, at this time, data annotation method 1 cannot be used, but the first dataset can still be used for optimization. Please refer to Figure 13 , which is the flowchart of data annotation method 2 provided in the embodiments of this application. The following combines Figure 13 to introduce the specific steps of data annotation method 2.

[0258] S1301. Use the unsupervised method to obtain a trained object detection model.

[0259] The annotation device uses unlabeled images to train the backbone module and the auto-encoding module of the object detection model.

[0260] S1302. Label some images, use the domain mixing method to obtain enhanced partial images, and fine-tune the model with the enhanced partial images.

[0261] For the steps of the domain mixing method, please refer to the Figure 9 content described above, which will not be elaborated here.

[0262] S1303. Judge whether the usability index value is reached.

[0263] If the usability index value is not reached, continue to execute S1302. If the usability index value is reached, execute S1304. For the meaning of the usability index value, please refer to the content described above, which will not be elaborated here.

[0264] S1304. Divide the remaining images into N parts and set i = 1.

[0265] The remaining images refer to the unlabeled images in the first dataset except for some images.

[0266] S1305, Prediction stage S2 i : Use the weighted voting algorithm for the i-th image subset.

[0267] For the weighted voting algorithm, please refer to the content described above. Figure 9 It will not be elaborated here.

[0268] S1306, Correction stage S3 i : Manual inspection and correction.

[0269] Annotators manually label and correct or add "difficult cases" such as misdetections, false detections, and missed detections in each image of the i-th image subset.

[0270] S1307, Determine whether the saturation index value is reached.

[0271] If the saturation index value is not reached, execute S1308. If the saturation index value is reached, directly execute S1309. For the meaning of the saturation index value, please refer to the content described above. It will not be elaborated here.

[0272] S1308, Model training stage S1 i : Use the incremental learning method to train on the (i - 1)-th incremental model with new samples for 1 - 2 epochs to obtain the i-th incremental model.

[0273] New samples refer to the images with correct labels manually corrected in the correction stage. For the process of incremental learning, please refer to the content described above. It will not be elaborated here.

[0274] S1309, Determine whether i is greater than or equal to N.

[0275] N is the total number of image subsets. If i < N, after executing S1310, continue to execute S1305. If i ≥ N, the process ends.

[0276] S1310, i = i + 1.

[0277] Taking the annotation tasks of pedestrians, motor vehicles, non-motor vehicles, and faces with 50,000 images as an example, the time consumption of different annotation methods is shown in Table 1.

[0278] Table 1

[0279]

[0280] Here, "person-hours" refers to the number of people multiplied by the number of hours. It is evident that although the time consumed by the naive semi-automatic annotation method is already reduced compared to the brute-force annotation method, the total time consumed can be further reduced by using the data annotation method 1 or data annotation method 2 proposed in this application.

[0281] To examine the key stages for reducing time consumption, the time consumption of each stage for different annotation methods is shown in Table 2.

[0282] Table 2

[0283]

[0284] As can be seen, the data annotation methods 1 and 2 proposed in this application reduce the time spent on repeated training in the model training stage S1 through incremental learning and weighted voting algorithms. Therefore, the time spent in the model training stage S1 is less than that of the naive semi-automatic annotation method.

[0285] Please refer to Figure 14 This is a time consumption percentage diagram for each stage of a data annotation method 1 provided in this application embodiment. Combined with... Figure 3 and Figure 14 As can be seen, compared to naive semi-automatic annotation methods, the data annotation method 1 proposed in this application significantly reduces the time required for the model to reach a usable level by employing a transfer alignment training algorithm in the initial annotation stage So. Therefore, the time consumption of the So stage is reduced from [previous stage]. Figure 3 The percentage decreased from 31.4% to Figure 14 24.8%.

[0286] In summary, the data annotation method provided in this application, when faced with a completely new dataset to be labeled, uses different data annotation methods to accelerate the annotation process based on whether there is available labeled data for similar scenarios. If a second dataset exists, it is fully utilized in the initial annotation stage. Through a transfer alignment training algorithm, a target detection model usable for the first dataset is quickly obtained, effectively reducing the annotation workload. If no second dataset is available, compared to the naive semi-automatic annotation method, only a smaller amount of manual data annotation is required. A usable prediction model can be obtained by using a domain fusion method to augment the labeled data, effectively reducing the annotation workload. Throughout the annotation process, incremental learning eliminates the need for repeated model retraining, reducing training time. Furthermore, a weighted voting algorithm ensures that multiple incremental models are fully utilized, effectively improving dataset annotation efficiency and reducing manpower and time investment.

[0287] Based on the same inventive concept, this application also provides a data annotation device, which is disposed in the annotation equipment discussed above. Please refer to... Figure 15The device includes:

[0288] The determination module 1501 is used to determine the category of the target in the first image based on multiple first images in the first dataset;

[0289] The determination module 1501 is also used to determine whether a second dataset exists; the second dataset includes multiple labeled second images, the labels being used to indicate the location of the target in the second image and / or the category of the target, the category of the target in the second image being the same as the category of the target in the first image;

[0290] The module 1502 is used to input the first image into the trained object detection model for prediction if the second dataset does not exist, and obtain the label of the first image; wherein the trained object detection model is obtained by training the object detection model with the second dataset.

[0291] In one possible embodiment, the obtaining module 1502 is specifically used for:

[0292] Before inputting the first image into the trained object detection model for prediction and obtaining the label of the first image, feature extraction is performed on any second image to obtain the features of any second image, and prediction is performed on the features of any second image to obtain the prediction result of any second image.

[0293] Feature extraction is performed on any first image to obtain the features of any first image;

[0294] The parameters of the target detection model are adjusted based on the first error between the features of any first image and the features of any second image, and the second error between the prediction result of any second image and the corresponding label.

[0295] The training continues until the maximum number of iterations is reached, at which point the trained object detection model is obtained.

[0296] In one possible embodiment, the obtaining module 1502 is specifically used for:

[0297] If the trained object detection model does not reach the usability index value, then manually labeled images from the first dataset are obtained; the first dataset includes some images and the remaining unlabeled images.

[0298] Based on a subset of images, the parameters of the trained object detection model are adjusted until the adjusted object detection model reaches the usability index value.

[0299] The remaining images are input into the adjusted object detection model for prediction to obtain the labels for the remaining images.

[0300] In one possible embodiment, the obtaining module 1502 is further configured to:

[0301] After determining whether a second dataset exists, if no second dataset exists, then the manually labeled images in the first dataset are obtained; wherein, the first dataset includes some images and the remaining unlabeled images;

[0302] The remaining images are input into the trained object detection model for prediction to obtain the labels of the remaining images; the trained object detection model is obtained by training the object detection model with the partial images and the remaining images.

[0303] In one possible embodiment, the obtaining module 1502 is specifically used for:

[0304] If the trained object detection model does not reach the usability index value, then perform Fourier transform on some images to obtain the first frequency map and the first phase map, and perform Fourier transform on the remaining images to obtain the second frequency map and the second phase map.

[0305] The mean amplitude-frequency map is obtained by averaging the first and second amplitude-frequency maps, and the inverse Fourier transform is performed on the mean amplitude-frequency map and the first phase map to obtain the enhanced partial image.

[0306] Based on the enhanced partial images, the trained target detection model is adjusted until the adjusted target detection model reaches the usability index value.

[0307] The remaining images are input into the adjusted object detection model for prediction to obtain the labels for the remaining images.

[0308] In one possible embodiment, the obtaining module 1502 is specifically used for:

[0309] Divide the remaining images into multiple image subsets;

[0310] When i equals 1, each image in the i-th image subset is input into the adjusted target detection model for prediction, and the labels of each image in the i-th image subset are obtained.

[0311] In one possible embodiment, after dividing the remaining image into multiple image subsets, the method further includes:

[0312] When i is greater than or equal to 2, obtain the manually corrected images with the correct labels in the (i-1)th image subset, and the index values ​​corresponding to the (i-1)th image subset;

[0313] If the index value corresponding to the (i-1)th image subset does not reach the saturation index value, then the (i-1)th incremental model is trained based on the manually corrected image with the correct label to obtain the i-th incremental model; where, when i equals 2, the (i-1)th incremental model is the adjusted target detection model, and the saturation index value is greater than the usability index value.

[0314] Each image in the i-th image subset is input into multiple incremental models for prediction, resulting in multiple prediction results for each image in the i-th image subset; wherein, the multiple incremental models include the (i-1)-th incremental model and the i-th incremental model;

[0315] The labels of each image in the i-th image subset are obtained by weighted summation of multiple prediction results for each image.

[0316] In one possible embodiment, the obtaining module 1502 is further configured to:

[0317] Before inputting each image in the i-th image subset into multiple incremental models for prediction and obtaining multiple prediction results for each image in the i-th image subset, sample images are extracted from a subset of images, and the sample images are input into multiple incremental models for prediction to obtain multiple prediction results for the sample images.

[0318] Based on multiple prediction results and corresponding labels of the sample image, obtain multiple index values ​​corresponding to the sample image;

[0319] The weighted sum of multiple prediction results for each image in the i-th image subset is used to obtain the labels for each image in the i-th image subset, including:

[0320] Based on multiple index values, the multiple prediction results of each image in the i-th image subset are weighted and summed to obtain the label of each image in the i-th image subset.

[0321] Figure 15 The device can also be used to implement the data annotation methods discussed above, which will not be elaborated here.

[0322] Based on the same inventive concept, this application provides an electronic device, which is equivalent to the labeling device discussed above. Please refer to... Figure 16 The device includes:

[0323] Memory 1602 is used to store program instructions;

[0324] Processor 1601 is used to call program instructions stored in memory 1602 and execute the data annotation method described above according to the obtained program instructions. Processor 1601 can also implement the aforementioned... Figure 15 The function of the data annotation device is discussed.

[0325] Processor 1601 can be a central processing unit (CPU), a digital processing unit, or a graphics processor, or a combination thereof. Memory 1602 can be volatile memory, such as random-access memory (RAM); memory 1602 can also be non-volatile memory, such as read-only memory, flash memory, hard disk drive (HDD), or solid-state drive (SSD); or memory 1602 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures, accessible by a computer, but is not limited thereto. Memory 1602 can be a combination of the above-mentioned memories.

[0326] Based on the same inventive concept, embodiments of this application provide a computer-readable storage medium storing a computer program. The computer program includes program instructions, which, when executed by a computer, cause the computer to perform the data annotation method as described above. Since the principles and methods of solving the problem described above are similar, the implementation of the computer-readable storage medium can be referred to the implementation of the method, and repeated details will not be elaborated further.

[0327] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0328] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0329] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0330] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0331] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data annotation method, characterized in that, include: Based on multiple first images in the first dataset, determine the category of the target in the first image; Determine if a second dataset exists; The second dataset includes multiple labeled second images, the labels indicating the location of a target in the second image and / or the category of the target, wherein the category of the target in the second image is the same as the category of the target in the first image; If the second dataset exists, feature extraction is performed on any second image to obtain the features of the second image, and prediction is performed on the features of the second image to obtain the prediction result of the second image. Feature extraction is performed on any first image to obtain the features of the first image; Based on the first error between the features of any first image and the features of any second image, and the second error between the prediction result of any second image and the corresponding label, the parameters of the target detection model are adjusted so that the feature distribution of the data in the first image set and the data in the second dataset is consistent in the latent feature space. Continue until the maximum number of iterations is reached to obtain the trained object detection model; The first image is input into the trained object detection model for prediction to obtain the label of the first image; wherein the trained object detection model is obtained by training the object detection model using the second dataset.

2. The method as described in claim 1, characterized in that, The first image is input into the trained object detection model for prediction to obtain the label of the first image, including: If the trained object detection model does not reach the usability index value, then manually labeled partial images in the first dataset are obtained; wherein, the first dataset includes the partial images and the remaining unlabeled images; Based on the aforementioned partial images, the parameters of the trained target detection model are adjusted until the adjusted target detection model reaches the usability index value. The remaining images are input into the adjusted target detection model for prediction to obtain the labels of the remaining images.

3. The method as described in claim 1, characterized in that, After determining whether a second dataset exists, the method further includes: If the second dataset does not exist, then the manually labeled partial images in the first dataset are obtained; wherein, the first dataset includes the partially labeled images and the remaining unlabeled images; The remaining images are input into the trained target detection model for prediction to obtain the labels of the remaining images; wherein, the trained target detection model is obtained by training the target detection model using the partial images and the remaining images.

4. The method as described in claim 3, characterized in that, The remaining images are input into the trained object detection model for prediction to obtain the labels of the remaining images, including: If the trained target detection model does not reach the usability index value, then perform Fourier transform on the partial image to obtain the first frequency map and the first phase map, and perform Fourier transform on the remaining image to obtain the second frequency map and the second phase map. The mean amplitude-frequency map is obtained by averaging the first amplitude-frequency map and the second amplitude-frequency map, and the inverse Fourier transform is performed on the mean amplitude-frequency map and the first phase map to obtain the enhanced partial image. Based on the enhanced partial image, the trained target detection model is adjusted until the adjusted target detection model reaches the usability index value. The remaining images are input into the adjusted target detection model for prediction to obtain the labels of the remaining images.

5. The method as described in claim 2 or 4, characterized in that, The remaining images are input into the adjusted target detection model for prediction to obtain the labels of the remaining images, including: The remaining images are divided into multiple image subsets; When i equals 1, each image in the i-th image subset is input into the adjusted target detection model for prediction, and the label of each image in the i-th image subset is obtained.

6. The method as described in claim 5, characterized in that, After dividing the remaining image into multiple image subsets, the method further includes: When i is greater than or equal to 2, obtain the manually corrected images with the correct labels in the (i-1)th image subset, and the index values ​​corresponding to the (i-1)th image subset; If the index value corresponding to the (i-1)th image subset does not reach the saturation index value, then the (i-1)th incremental model is trained based on the manually corrected image with the correct label to obtain the i-th incremental model; wherein, when i equals 2, the (i-1)th incremental model is the adjusted target detection model, and the saturation index value is greater than the usability index value. Each image in the i-th image subset is input into multiple incremental models for prediction, thereby obtaining multiple prediction results for each image in the i-th image subset; wherein, the multiple incremental models include the (i-1)-th incremental model and the i-th incremental model; The labels of each image in the i-th image subset are obtained by weighted summation of multiple prediction results for each image.

7. The method as described in claim 6, characterized in that, Before inputting each image in the i-th image subset into multiple incremental models for prediction and obtaining multiple prediction results for each image in the i-th image subset, the method further includes: Sample images are extracted from the partial images, and the sample images are respectively input into the multiple incremental models for prediction to obtain multiple prediction results for the sample images; Based on multiple prediction results and corresponding labels of the sample image, multiple index values ​​corresponding to the sample image are obtained; The labels of each image in the i-th image subset are obtained by weighted summation of multiple prediction results for each image, including: Based on the multiple index values, the multiple prediction results of each image in the i-th image subset are weighted and summed to obtain the label of each image in the i-th image subset.

8. A data annotation device, characterized in that, include: The determination module is used to determine the category of the target in the first image based on multiple first images in the first dataset; The determining module is further configured to determine whether a second dataset exists; the second dataset includes multiple labeled second images, the labels indicating the location of a target in the second image and / or the category of the target, the category of the target in the second image being the same as the category of the target in the first image; if the second dataset exists, feature extraction is performed on any second image to obtain the features of the any second image, and prediction is performed on the features of the any second image to obtain the prediction result of the any second image; feature extraction is performed on any first image to obtain the features of the any first image; based on the first error between the features of the any first image and the features of the any second image, and the second error between the prediction result of the any second image and the corresponding label, the parameters of the target detection model are adjusted so that the feature distribution of the data in the first image set and the data in the second dataset is consistent in the latent feature space; Continue until the maximum number of iterations is reached to obtain the trained object detection model; The module is used to input the first image into the trained object detection model for prediction and obtain the label of the first image; wherein the trained object detection model is obtained by training the object detection model using a second dataset.

9. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the method of any one of claims 1-7 according to the obtained program instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images