A data labeling method and device, and a storage medium

Abstract

The application relates to the technical field of data processing, and discloses a data labeling method and device and a storage medium, which comprise the following steps: determining a to-be-labeled data set and a semantic embedding vector result of sample data labeling; performing sample value analysis and sample screening operations according to the to-be-labeled data set and the semantic embedding vector result, and obtaining a candidate pseudo-label list; performing a pseudo-label determination operation according to the candidate pseudo-label list, and obtaining a pseudo-label set; performing multi-view processing and special sample processing operations according to the pseudo-label set, and obtaining an effective sample data set; performing multi-level labeling quality evaluation operations according to the effective sample data set, and obtaining a quality evaluation result; and determining a final data labeling result according to the quality evaluation result. It can be seen that the application can improve the comprehensiveness of the data labeling processing mode, and further improve the labeling accuracy and reliability of the data, and improve the labeling efficiency and convenience of the data.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data annotation method, apparatus, and storage medium. Background Technology

[0002] With the widespread application of artificial intelligence technology in fields such as computer vision, natural language processing, and speech recognition, labeled data has become a core foundational resource supporting the training of deep learning models. Therefore, how to perform high-quality data labeling has become a major research focus.

[0003] Currently, data annotation is still mainly done manually, requiring staff to analyze and annotate large amounts of data one by one based on their subjective judgment. Even for the same data, different staff may produce different annotation results. Therefore, manual annotation methods generally suffer from low efficiency, high cost, and insufficient consistency. In addition, there are also data annotation methods based on active learning in existing technologies. However, active learning still relies heavily on manual annotation. Therefore, existing data annotation methods suffer from low data annotation accuracy and low annotation efficiency. Summary of the Invention

[0004] This invention provides a data annotation method, apparatus, and storage medium, which can improve the accuracy and reliability of data annotation, as well as the efficiency and convenience of data annotation.

[0005] To address the aforementioned technical problems, the first aspect of this invention discloses a data annotation method, the method comprising: Determine the set of data to be labeled and its semantic embedding vector results that need to be labeled, wherein the set of data to be labeled includes at least one sample data to be labeled; Based on the dataset to be labeled and the semantic embedding vector results, perform sample value analysis and sample screening operations to obtain a candidate pseudo-label list; Based on the candidate pseudo-label list, perform a pseudo-label determination operation to obtain a pseudo-label set; Based on the pseudo-label set, perform multi-view processing and special sample processing operations to obtain a valid sample data set; Based on the set of valid sample data, perform a multi-level annotation quality assessment operation to obtain the quality assessment results; Based on the quality assessment results, the final data labeling results are determined.

[0006] As an optional implementation, in a first aspect of the invention, the quality assessment result includes a review candidate set, which includes at least one low-quality sample data. And, determining the final data annotation result based on the quality assessment result includes: Based on the low-quality sample data, determine the second nearest neighbor sample information, and based on the second nearest neighbor sample information, determine the highest frequency labeling category that meets the preset highest occurrence frequency condition; Based on the annotation information of the low-quality sample data and the most frequent annotation category, determine whether the low-quality sample data meets the preset automatic correction category conditions; When it is determined that the low-quality sample data meets the automatic correction category conditions, the automatic correction category operation is performed on the low-quality sample data according to the highest frequency labeling category to obtain the automatically corrected sample data, which is used as the first verification sample data. When it is determined that the low-quality sample data does not meet the automatic correction category conditions, the low-quality sample data is identified as the first review sample data; Based on the first review sample data, the final data annotation result is determined.

[0007] As an optional implementation, in the first aspect of the present invention, determining the final data annotation result based on the first verification sample data includes: Determine the second confidence level result and the second uncertainty result of the first review sample data, and based on the second confidence level result and the second uncertainty result, determine whether the first review sample data meets the preset manual review conditions; When it is determined that the first review sample data does not meet the manual review conditions, an update operation is performed on the pseudo-label set based on the first review sample data to obtain the updated pseudo-label set; When it is determined that the first review sample data meets the manual review conditions, a manual review and annotation operation is performed on the first review sample data to obtain the second review sample data; the second review sample data is added to the pre-constructed labeled data set to obtain the updated labeled data set. Based on the updated set of pseudo-labels, determine whether the preset label convergence condition is met. When it is determined that the current annotation convergence condition is met, the final data annotation result is determined based on the updated pseudo-label set and the updated labeled data set.

[0008] As an optional implementation, in the first aspect of the present invention, the semantic embedding vector result includes the semantic embedding vector of each of the labeled sample data; And, based on the dataset to be labeled and the semantic embedding vector results, the step of performing sample value analysis and sample screening operations to obtain a candidate pseudo-label list includes: For each of the sample data to be labeled, the probability distribution prediction result of the sample data to be labeled for each preset labeling category is determined based on the semantic embedding vector of the sample data to be labeled, the determined classification weight matrix information and the classification bias vector information. Based on the probability distribution prediction results of the sample data to be labeled, the information entropy result of the sample data to be labeled is determined, and based on the semantic embedding vector of the sample data to be labeled and the determined first nearest neighbor sample information, the semantic density result of the sample data to be labeled is determined. Based on the information entropy and semantic density results of the sample data to be labeled, the comprehensive sample value score of the sample data to be labeled is determined. Based on the comprehensive sample value score of each of the unlabeled sample data, the first sample data that meets the preset high value screening conditions is determined from all the unlabeled sample data; Based on all the first sample data, a list of candidate pseudo-labels is determined.

[0009] As an optional implementation, in a first aspect of the present invention, the candidate pseudo-label list includes at least one first sample data that meets a preset high-value screening condition from all the sample data to be labeled. And, the step of performing a pseudo-label determination operation based on the candidate pseudo-label list to obtain a pseudo-label set includes: Based on the determined accuracy of the validation set annotation in the current iteration round, determine the adaptive confidence threshold; Based on the probability distribution prediction results of each first sample data for each preset labeled category, determine the maximum predicted probability value of each first sample data. Based on the maximum predicted probability value of each first sample data and the adaptive confidence threshold, second sample data that meets the preset pseudo-label conditions are determined from all the first sample data. Based on each second sample data and its maximum predicted probability value, generate pseudo-label information for each second sample data. Based on each of the second sample data and its pseudo-label information, a set of pseudo-labels is determined; Furthermore, the method further includes: Based on the pre-constructed labeled dataset, the supervision loss information is determined, and based on the pseudo-label set, the pseudo-label loss information is determined. Based on the determined pseudo-label loss weight information, the supervision loss information, and the pseudo-label loss information for the current training phase, the joint loss function information is determined. Based on the joint loss function information, a parameter update operation is performed to obtain the parameter update results for the feature extraction model.

[0010] As an optional implementation, in the first aspect of the present invention, the pseudo-label set includes at least one second sample data and its pseudo-label information that satisfy the preset pseudo-label conditions from all the sample data to be labeled. And, the step of performing multi-view processing and special sample processing operations based on the pseudo-label set to obtain a valid sample data set includes: Based on the second sample data in the pseudo-label set, determine one or more target second sample data; For each target second sample data, at least two enhanced view sample results corresponding to the target second sample data are determined according to the data format type of the target second sample data; Based on all the enhanced view sample results corresponding to the second sample data of the target, determine the model prediction distribution result for each enhanced view sample result; Based on all the model prediction distribution results corresponding to the second sample data of the target, determine the prediction difference information corresponding to the second sample data of the target; Based on the prediction difference information of each of the target second sample data, third sample data that does not meet the preset prediction conflict condition is determined from all the target second sample data and is taken as valid sample data; and / or, Based on the prediction difference information of each target second sample data, a fourth sample data that meets the preset prediction conflict condition is determined from all the target second sample data and used as conflict sample data; based on the conflict sample data, a conflict sample data removal operation is performed on the pseudo-label set to obtain valid sample data; Based on all the aforementioned valid sample data, determine the set of valid sample data.

[0011] As an optional implementation, in the first aspect of the present invention, the effective sample data set includes at least one effective sample data, and the effective sample data includes a third sample data that does not meet the preset prediction conflict condition among all the sample data to be labeled. And, the step of performing a multi-level annotation quality assessment operation based on the valid sample data set to obtain the quality assessment result includes: Based on the probability distribution prediction results of each of the valid sample data, determine the first confidence level result and the first uncertainty result of each of the valid sample data; Based on the first confidence result of each of the valid sample data, sample data that meets the preset low confidence condition is determined from all the valid sample data and is used as low-quality sample data; and / or, based on the first uncertainty result of each of the valid sample data, sample data that meets the preset high uncertainty condition is determined from all the valid sample data and is used as low-quality sample data. Based on all the aforementioned low-quality sample data, a review candidate set is determined; Based on the first confidence result of all the valid sample data, determine the confidence variance result for each preset labeling category, and determine the quality warning labeling category based on the confidence variance result and sample size of each preset labeling category; Based on all the valid sample data and their first confidence level results and first uncertainty results, global quality indicator information is determined; The quality assessment result is determined based on the review candidate set, the quality warning label category, and the global quality indicator information.

[0012] A second aspect of the present invention discloses a data annotation apparatus, the apparatus comprising: The basic information determination module is used to determine the set of data to be labeled and its semantic embedding vector results that need to be labeled, wherein the set of data to be labeled includes at least one sample data to be labeled. The candidate pseudo-label determination module is used to perform sample value analysis and sample screening operations based on the dataset to be labeled and the semantic embedding vector results to obtain a list of candidate pseudo-labels. The pseudo-label determination module is used to perform a pseudo-label determination operation based on the candidate pseudo-label list to obtain a pseudo-label set; The effective sample determination module is used to perform multi-view processing and special sample processing operations based on the pseudo-label set to obtain an effective sample data set; The quality assessment module is used to perform multi-level annotation quality assessment operations based on the effective sample data set to obtain quality assessment results; The final annotation result determination module is used to determine the final data annotation result based on the quality assessment result.

[0013] As an optional implementation, in a second aspect of the invention, the quality assessment result includes a review candidate set, which includes at least one low-quality sample data. Furthermore, the method by which the final annotation result determination module determines the final data annotation result based on the quality assessment result specifically includes: Based on the low-quality sample data, determine the second nearest neighbor sample information, and based on the second nearest neighbor sample information, determine the highest frequency labeling category that meets the preset highest occurrence frequency condition; Based on the annotation information of the low-quality sample data and the most frequent annotation category, determine whether the low-quality sample data meets the preset automatic correction category conditions; When it is determined that the low-quality sample data meets the automatic correction category conditions, the automatic correction category operation is performed on the low-quality sample data according to the highest frequency labeling category to obtain the automatically corrected sample data, which is used as the first verification sample data. When it is determined that the low-quality sample data does not meet the automatic correction category conditions, the low-quality sample data is identified as the first review sample data; Based on the first review sample data, the final data annotation result is determined.

[0014] As an optional implementation, in the second aspect of the present invention, the method by which the final annotation result determination module determines the final data annotation result based on the first review sample data specifically includes: Determine the second confidence level result and the second uncertainty result of the first review sample data, and based on the second confidence level result and the second uncertainty result, determine whether the first review sample data meets the preset manual review conditions; When it is determined that the first review sample data does not meet the manual review conditions, an update operation is performed on the pseudo-label set based on the first review sample data to obtain the updated pseudo-label set; When it is determined that the first review sample data meets the manual review conditions, a manual review and annotation operation is performed on the first review sample data to obtain the second review sample data; the second review sample data is added to the pre-constructed labeled data set to obtain the updated labeled data set. Based on the updated set of pseudo-labels, determine whether the preset label convergence condition is met. When it is determined that the current annotation convergence condition is met, the final data annotation result is determined based on the updated pseudo-label set and the updated labeled data set.

[0015] As an optional implementation, in a second aspect of the invention, the semantic embedding vector result includes the semantic embedding vector of each of the labeled sample data. Furthermore, the candidate pseudo-label determination module performs sample value analysis and sample screening operations based on the dataset to be labeled and the semantic embedding vector results to obtain the candidate pseudo-label list in the following specific ways: For each of the sample data to be labeled, the probability distribution prediction result of the sample data to be labeled for each preset labeling category is determined based on the semantic embedding vector of the sample data to be labeled, the determined classification weight matrix information and the classification bias vector information. Based on the probability distribution prediction results of the sample data to be labeled, the information entropy result of the sample data to be labeled is determined, and based on the semantic embedding vector of the sample data to be labeled and the determined first nearest neighbor sample information, the semantic density result of the sample data to be labeled is determined. Based on the information entropy and semantic density results of the sample data to be labeled, the comprehensive sample value score of the sample data to be labeled is determined. Based on the comprehensive sample value score of each of the unlabeled sample data, the first sample data that meets the preset high value screening conditions is determined from all the unlabeled sample data; Based on all the first sample data, a list of candidate pseudo-labels is determined.

[0016] As an optional implementation, in a second aspect of the invention, the candidate pseudo-label list includes at least one first sample data that meets a preset high-value screening condition from all the sample data to be labeled. Furthermore, the pseudo-label determination module performs a pseudo-label determination operation based on the candidate pseudo-label list to obtain a pseudo-label set in the following specific ways: Based on the determined accuracy of the validation set annotation in the current iteration round, determine the adaptive confidence threshold; Based on the probability distribution prediction results of each first sample data for each preset labeled category, determine the maximum predicted probability value of each first sample data. Based on the maximum predicted probability value of each first sample data and the adaptive confidence threshold, second sample data that meets the preset pseudo-label conditions are determined from all the first sample data. Based on each second sample data and its maximum predicted probability value, generate pseudo-label information for each second sample data. Based on each of the second sample data and its pseudo-label information, a set of pseudo-labels is determined; The device also includes: The model parameter optimization module is used to determine supervised loss information based on the pre-constructed labeled dataset and pseudo-label loss information based on the pseudo-label dataset; determine joint loss function information based on the determined pseudo-label loss weight information, supervised loss information, and pseudo-label loss information in the current training stage; and perform parameter update operation based on the joint loss function information to obtain parameter update results for the feature extraction model.

[0017] As an optional implementation, in a second aspect of the present invention, the pseudo-label set includes at least one second sample data and its pseudo-label information that satisfy a preset pseudo-label condition from all the sample data to be labeled. Furthermore, the effective sample determination module performs multi-view processing and special sample processing operations based on the pseudo-label set to obtain the effective sample data set in the following specific ways: Based on the second sample data in the pseudo-label set, determine one or more target second sample data; For each target second sample data, at least two enhanced view sample results corresponding to the target second sample data are determined according to the data format type of the target second sample data; Based on all the enhanced view sample results corresponding to the second sample data of the target, determine the model prediction distribution result for each enhanced view sample result; Based on all the model prediction distribution results corresponding to the second sample data of the target, determine the prediction difference information corresponding to the second sample data of the target; Based on the prediction difference information of each of the target second sample data, third sample data that does not meet the preset prediction conflict condition is determined from all the target second sample data and is taken as valid sample data; and / or, Based on the prediction difference information of each target second sample data, a fourth sample data that meets the preset prediction conflict condition is determined from all the target second sample data and used as conflict sample data; based on the conflict sample data, a conflict sample data removal operation is performed on the pseudo-label set to obtain valid sample data; Based on all the aforementioned valid sample data, determine the set of valid sample data.

[0018] As an optional implementation, in a second aspect of the present invention, the effective sample data set includes at least one effective sample data, and the effective sample data includes a third sample data that does not meet the preset prediction conflict condition among all the sample data to be labeled. Furthermore, the quality assessment module performs multi-level annotation quality assessment operations based on the valid sample data set to obtain the quality assessment results in the following specific ways: Based on the probability distribution prediction results of each of the valid sample data, determine the first confidence level result and the first uncertainty result of each of the valid sample data; Based on the first confidence result of each of the valid sample data, sample data that meets the preset low confidence condition is determined from all the valid sample data and is used as low-quality sample data; and / or, based on the first uncertainty result of each of the valid sample data, sample data that meets the preset high uncertainty condition is determined from all the valid sample data and is used as low-quality sample data. Based on all the aforementioned low-quality sample data, a review candidate set is determined; Based on the first confidence result of all the valid sample data, determine the confidence variance result for each preset labeling category, and determine the quality warning labeling category based on the confidence variance result and sample size of each preset labeling category; Based on all the valid sample data and their first confidence level results and first uncertainty results, global quality indicator information is determined; The quality assessment result is determined based on the review candidate set, the quality warning label category, and the global quality indicator information.

[0019] A third aspect of the present invention discloses another data annotation apparatus, the apparatus comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute a data annotation method disclosed in the first aspect of the present invention.

[0020] The fourth aspect of the present invention discloses a computer storage medium storing computer instructions, which, when invoked, are used to execute a data annotation method disclosed in the first aspect of the present invention.

[0021] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: In this embodiment of the invention, a set of data to be labeled and its semantic embedding vector results are determined, the set of data to be labeled including at least one sample data to be labeled; based on the set of data to be labeled and the semantic embedding vector results, sample value analysis and sample screening operations are performed to obtain a candidate pseudo-label list; based on the candidate pseudo-label list, a pseudo-label determination operation is performed to obtain a pseudo-label set; based on the pseudo-label set, multi-view processing and special sample processing operations are performed to obtain a valid sample data set; based on the valid sample data set, a multi-level labeling quality assessment operation is performed to obtain a quality assessment result; based on the quality assessment result, the final data labeling result is determined. It can be seen that this invention can achieve intelligent data labeling through sample value analysis and sample screening operations, pseudo-label determination operations, multi-view processing and special sample processing operations, multi-level labeling quality assessment operations, and final data labeling result determination operations. This is beneficial to improving the comprehensiveness of data labeling processing methods, thereby improving the accuracy and reliability of data labeling, improving the efficiency and convenience of data labeling, and ultimately improving the quality of data labeling and achieving a synergistic improvement in labeling efficiency and quality under weak supervision. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart illustrating a data annotation method disclosed in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of a data annotation device disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of another data annotation device disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of another data annotation device disclosed in an embodiment of the present invention. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or end that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or ends.

[0026] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0027] This invention discloses a data annotation method, apparatus, and storage medium. It enables intelligent data annotation through sample value analysis and screening, pseudo-label identification, multi-view processing and special sample processing, multi-level annotation quality assessment, and final data annotation result determination. This improves the comprehensiveness of data annotation processing, thereby enhancing the accuracy and reliability of annotation, as well as the efficiency and convenience of annotation. Ultimately, it improves data annotation quality and achieves a synergistic improvement in annotation efficiency and quality under weakly supervised conditions. Detailed descriptions follow.

[0028] Example 1 Please see Figure 1 , Figure 1 This is a flowchart illustrating a data annotation method disclosed in an embodiment of the present invention. Figure 1 The described method can be applied to a data annotation device, which may include a server, including a local server or a cloud server; this embodiment of the invention is not limited to this. Figure 1 As shown, the annotation method for this data includes the following operations: 101. Determine the set of data to be labeled and its semantic embedding vector results that need to be labeled. The set of data to be labeled includes at least one sample data to be labeled.

[0029] Optionally, the semantic embedding vector results of the sample data annotations are determined in the following way: Based on the data format type of each sample data to be labeled, determine the feature extraction network for each sample data to be labeled; Based on the feature extraction network of each sample data to be labeled, the feature extraction operation is performed to obtain the semantic feature vector (i.e. the original feature vector) of each sample data to be labeled. The semantic feature vector of each unlabeled sample data is input into the shared projection layer for analysis to obtain the semantic embedding vector of each unlabeled sample data (i.e., normalized unified embedding). The semantic embedding vector result is determined based on the semantic embedding vector of each sample data to be labeled.

[0030] Optionally, the feature extraction network can correspond to the backbone network, and may include, but is not limited to, the image feature extraction network ResNet-50, the text feature extraction network BERT (Bidirectional Encoder Representations from Transformers), the speech feature extraction network Wav2Vec, etc., and the embodiments of the present invention are not limited thereto.

[0031] Optionally, the data format type may include, but is not limited to, one or more of image types, text types, voice types, and other data types, and this embodiment of the invention does not impose any limitations.

[0032] Optionally, the semantic embedding vector of the sample data to be labeled is illustrated in the following example: the semantic embedding vector of the sample data to be labeled is z. i =W2·ReLU(W1·h i +b1)+b2, where hi is the semantic feature vector of the sample data to be labeled, W1 and W2 are the projection weight information, b1 and b2 are the projection bias information, and ReLU is the activation function. This embodiment of the invention is not limited.

[0033] Optionally, regarding the semantic feature vector h of the sample data to be labeled i For example, when the sample data to be labeled is image data, each image data is scaled to 224×224 pixels, and h is obtained by forward propagation through ResNet-50 and taking the output of the global average pooling layer. i ∈R 2048 That is, h i h is a 2048-dimensional real vector. i It has 2048 real numbers; when the sample data to be labeled is text data, each text data is truncated or padded to a maximum length of 128 tags, and h is obtained by taking the hidden layer output corresponding to the [CLS] tag through BERT-base-chinese forward propagation. i ∈R 768 That is, h i h is a 768-dimensional real vector. iIt has 768 real numbers, but the embodiments of the present invention are not limited thereto.

[0034] Optionally, regarding the semantic embedding vector of the sample data to be labeled, an example is given below: the feature vector of the sample data to be labeled is input into the hidden layer of the semantic embedding projection layer to obtain the intermediate feature vector h output by the hidden layer. hidden Specifically, linear transformation and nonlinear activation can be performed using the following formula: h hidden = ReLU(W1·h i +b1), where W1 is the weight matrix of the hidden layer, b1 is the bias vector of the hidden layer, and ReLU is the modified linear unit activation function. Specifically, the dimension of the weight matrix W1 of the hidden layer is R. d in ×512 d in R represents the input feature dimension, 512 represents the output dimension of the hidden layer, and the dimension of the bias vector b1 is R. 512 Furthermore, the sample data to be labeled may come from different modalities or different feature extraction networks, and the dimensions of their original feature vectors may also be different. Therefore, the weight matrix W1 of the hidden layer can be configured to adapt to different input dimensions d. in For example, when the sample data to be labeled is image data, and the feature vector obtained by the image feature extraction network has a dimension of 2048, then the corresponding hidden layer weight matrix is ​​represented as W. 1image ∈R 2048×512 For example, when the sample data to be labeled is text data, the feature vector obtained by the text feature extraction network has a dimension of 768, and the corresponding hidden layer weight matrix is ​​represented as W. 1text ∈R 768×512 Therefore, the semantic embedding projection layer can uniformly map input features of different modalities and dimensions to the same 512-dimensional hidden space, thereby achieving cross-modal feature alignment. Furthermore, the intermediate feature vector output from the hidden layer is further input into the output layer of the semantic embedding projection layer to obtain the unified embedding vector z output by the semantic embedding projection layer. i Specifically, a linear transformation can be performed using the following formula: z i =W2·h hidden +b2, where W2 is the weight matrix of the output layer, and b2 is the bias vector of the output layer. Specifically, the dimension of the weight matrix W2 of the output layer is R. 512×256 The dimension of the bias vector b2 of the output layer is R. 256 Therefore, after each sample data to be labeled is processed by the semantic embedding projection layer, a 256-dimensional unified embedding vector z is finally obtained. i Furthermore, for the unified embedding vector z i L2 normalization is performed, specifically through the formula z.i =z i / ||z i ||2, After normalization, the unified embedding vector satisfies ||z i ||2=1, meaning the unified embedding vector is constrained to the unit hypersphere. Therefore, the semantic similarity between different unlabeled sample data can be directly measured by the inner product or cosine similarity. Furthermore, the bias vectors b1 and b2 of the hidden layer and the output layer are initialized to zero. The weight matrices W1 and W2, along with the bias vectors b1 and b2, are jointly optimized during the backpropagation of the overall loss function as part of the end-to-end training. The overall loss function is: L total =L sup +λ t ·L pseudo +η·L cons , where L sup For the purpose of monitoring losses, L pseudo For pseudo-label loss, L cons For consistency loss, λ t η and η represent the hyperparameters used to balance the weights of each loss term. During backpropagation, the Adam optimizer is used to jointly optimize these parameters. Specifically, the learning rate of the Adam optimizer can be set to lr = 1 × 10⁻⁶. -4 Furthermore, the optimizer type, learning rate, and other hyperparameters can be adjusted according to specific task requirements, and this embodiment of the invention does not impose any limitations.

[0035] Optionally, the L2 normalization operation described above can eliminate the difference in magnitude of the outputs of different modal backbone networks. Because different modal backbone networks have different structures, the magnitude distribution of their output feature vectors varies significantly. For example, the magnitude of the feature vector output by an image modal backbone network (such as ResNet-50) may be several times greater than that of the feature vector output by a text modal backbone network (such as BERT). If cross-modal semantic comparison is performed directly, the difference in magnitude will interfere with the semantic distance metric. After L2 normalization, the unified embedding vectors of each modality are constrained to a unit hypersphere, retaining only directional information, and cross-modal semantic comparison is no longer affected by magnitude. Furthermore, the L2 normalization operation facilitates semantic similarity calculation. After normalization, any two unified embedding vectors z... i With z j The cosine similarity between them can be simplified to a dot product, i.e., cos(z) i ,z j )=z i ·z j When using the FAISS nearest neighbor retrieval tool, cosine distance retrieval can be transformed into inner product retrieval, significantly improving retrieval speed. Furthermore, L2 normalization ensures consistency in nearest neighbor operations, resulting in a higher semantic density D(x). iBoth the near neighbor consistency correction (e.g., based on K=20 nearest neighbors) and the near neighbor consistency correction (e.g., based on F=15 nearest neighbors) are based on the semantic distance in the normalized space. The normalized space provides a unified scale basis for all subsequent semantic judgments, avoiding the problem of inconsistent nearest neighbor judgments caused by differences in modulus length. This is beneficial to improving the stability and annotation accuracy of the semantic judgment of the sample data to be labeled. This embodiment of the invention does not limit the scope of the problem.

[0036] Optionally, the input dataset may include, but is not limited to, the dataset to be labeled, the dataset already labeled, and the label class number; more specifically, the dataset to be labeled is denoted as D. U ={x1, x2, ..., x N}, where x i Let N represent the i-th unlabeled data point, and N represent the number of unlabeled sample data points. Unlabeled sample data can be understood as unlabeled data, and the number of unlabeled data points can be set to N=50000, etc. The set of labeled data is denoted as D. L ={(x1, y1), ..., (x M , y M )}, where x i Let M represent the i-th labeled sample data, and M represent the number of labeled sample data. Labeled sample data can be understood as manually precisely labeled data. The number of manually precisely labeled data can be set to M=7500, accounting for approximately 15% of the total data, etc. In addition, the total number of labeled categories is denoted as C, which represents the total number of categories corresponding to the sample data in the input dataset. For example, the total number of labeled categories can be set to C=50. This embodiment of the invention does not impose any limitations.

[0037] Optionally, the semantic embedding vector result can also be determined in the following ways: (1) Use cross-modal contrastive learning (such as CLIP architecture, etc.) to directly train the multimodal joint embedding space, so that sample data of different modalities can be compared and learned in a unified space without the need for a separate modal backbone network; (2) Use a knowledge distillation scheme, use a large-scale pre-trained multimodal model (such as CLIP, ALIGN, etc.) as the teacher model, and obtain a lightweight student model through distillation training. The student model completes the extraction of semantic embedding vectors. This embodiment of the invention does not limit the scope of the invention.

[0038] 102. Based on the dataset to be labeled and the semantic embedding vector results, perform sample value analysis and sample screening operations to obtain a list of candidate pseudo-labels.

[0039] 103. Based on the candidate pseudo-label list, perform the pseudo-label determination operation to obtain the pseudo-label set.

[0040] 104. Based on the set of pseudo-labels, perform multi-view processing and special sample processing operations to obtain a set of valid sample data.

[0041] 105. Based on the valid sample data set, perform multi-level annotation quality assessment operations to obtain the quality assessment results.

[0042] 106. Based on the quality assessment results, determine the final data labeling results.

[0043] As can be seen, the data annotation method described in this embodiment of the invention can achieve intelligent data annotation through sample value analysis and sample screening operations, pseudo-label determination operations, multi-view processing and special sample processing operations, multi-level annotation quality evaluation operations, and final data annotation result determination operations. This is beneficial to improving the comprehensiveness of data annotation processing, thereby improving the accuracy and reliability of data annotation, improving the efficiency and convenience of data annotation, and thus improving the quality of data annotation and achieving a synergistic improvement in annotation efficiency and quality under weak supervision conditions.

[0044] In an optional embodiment, the quality assessment result may include a review candidate set, which includes at least one low-quality sample data; further optionally, determining the final data annotation result based on the quality assessment result may include: Based on low-quality sample data, determine the information of the second nearest neighbor sample, and based on the information of the second nearest neighbor sample, determine the highest frequency labeling category that meets the preset highest occurrence frequency condition; Based on the annotation information and the most frequent annotation category of low-quality sample data, determine whether the low-quality sample data meets the preset automatic correction category conditions; When it is determined that the low-quality sample data meets the automatic correction category conditions, the automatic correction category operation is performed on the low-quality sample data according to the most frequent labeled category to obtain the automatically corrected sample data, which is used as the first verification sample data. When it is determined that low-quality sample data does not meet the automatic correction category conditions, the low-quality sample data is identified as the first review sample data. Based on the first review sample data, the final data annotation results are determined.

[0045] Optionally, regarding the second nearest neighbor sample information, an example is given below: In the semantic embedding space, for low-quality sample data, F nearest neighbor confirmed labeled samples are retrieved as the second nearest neighbor samples. These nearest neighbor confirmed labeled samples can come from the labeled dataset D. L Or a set of pseudo-labels with high confidence D P F can be set according to actual needs (e.g., F=15, etc.). F can be the same as the number K corresponding to the first nearest neighbor sample information mentioned above, or it can be set independently of K. This embodiment of the invention does not limit this.

[0046] Further optionally, the determination of the highest frequency label category that meets the preset highest occurrence frequency condition based on the second nearest neighbor sample information may include: Based on the second nearest neighbor sample information, determine the nearest neighbor label distribution results; Based on the distribution results of nearest neighbor labels, determine the highest frequency label category that meets the preset highest occurrence frequency condition.

[0047] Optionally, the nearest neighbor label distribution results can be calculated using the following formula: ; Where, count c The distribution results of nearest neighbor labels; NN F (x i The corresponding second nearest neighbor sample information, that is, F second nearest neighbor samples NN F (x i ); C represents the total number of labeled categories.

[0048] Optionally, the most frequently labeled category can be calculated using the following formula: ; in, c majority The most frequently labeled category, C To indicate the total number of categories, count c Let be the number of samples of class c among all second nearest neighbors, and let arg max be the value that makes count... c The largest category is c.

[0049] Further optionally, the above-mentioned determination of whether low-quality sample data meets the preset automatic correction category conditions based on the annotation information and the most frequent annotation category of low-quality sample data may include: Based on the annotation information of low-quality sample data, determine the current annotation category and the nearest neighbor consistency ratio of the most frequent annotation category; Determine whether the current annotation category matches the most frequent annotation category, and determine whether the nearest neighbor consistency ratio is greater than the preset nearest neighbor consistency ratio threshold; When it is determined that the current labeled category does not match the most frequent labeled category and the nearest neighbor consistency ratio is greater than the nearest neighbor consistency ratio threshold, the low-quality sample data is determined to meet the preset automatic correction category conditions. When it is determined that the current labeled category matches the most frequent labeled category and / or the nearest neighbor consistency ratio is less than or equal to the nearest neighbor consistency ratio threshold, it is determined that the low-quality sample data does not meet the preset automatic correction category conditions.

[0050] Further, optionally, the above-mentioned automatic category correction operation on low-quality sample data based on the most frequently labeled category, to obtain automatically corrected sample data, may include: The labeling categories of low-quality sample data are corrected to the most frequent labeling categories to obtain automatically corrected sample data.

[0051] Further optionally, after performing automatic category correction on the low-quality sample data based on the most frequently labeled category to obtain automatically corrected sample data, the method may further include the following operations: Correction information is recorded in the correction log, which includes one or more of the following: sample ID information, original label information, new label information, nearest neighbor consistency ratio, etc. of the low-quality sample data.

[0052] As can be seen, this optional embodiment can match the corresponding first verification sample data determination operation for low-quality sample data that meets the automatic correction category conditions and for low-quality sample data that does not meet the automatic correction category conditions, and then determine the final data annotation result based on the first verification sample data. This is beneficial to improving the comprehensiveness of the automatic correction annotation category method, improving the execution accuracy and timeliness of the automatic correction annotation category operation, and thus improving the accuracy, reliability, and timeliness of sample data correction. This, in turn, is beneficial to improving the accuracy and reliability of the determined first verification sample data, further improving the accuracy and reliability of the determined final data annotation result. In addition, it is beneficial to improving the efficiency and convenience of sample data correction.

[0053] In another optional embodiment, determining the final data annotation result based on the first review sample data may include: Determine the second confidence level and the second uncertainty level of the first review sample data, and based on the second confidence level and the second uncertainty level, determine whether the first review sample data meets the preset manual review conditions; When it is determined that the first review sample data does not meet the conditions for manual review, an update operation is performed on the pseudo-label set based on the first review sample data to obtain the updated pseudo-label set; When it is determined that the first review sample data meets the conditions for manual review, the manual review and annotation operation is performed on the first review sample data to obtain the second review sample data; the second review sample data is added to the pre-constructed labeled data set to obtain the updated labeled data set. Based on the updated set of pseudo-labels, determine whether the preset label convergence condition is met. When it is determined that the current annotation convergence condition is met, the final data annotation result is determined based on the updated pseudo-label set and the updated labeled data set.

[0054] Optionally, the calculation method for the second confidence result may refer to, but is not limited to, the calculation method for the first confidence result described below, and the determination method for the second uncertainty result may refer to, but is not limited to, the calculation method for the first uncertainty result described below. This embodiment of the invention does not impose any limitations.

[0055] Further optionally, the determination of whether the first review sample data meets the preset manual review conditions based on the second confidence level result and the second uncertainty result may include: Determine whether the second confidence level result is less than a preset second confidence threshold, and determine whether the second uncertainty result is greater than a preset second uncertainty threshold; When it is determined that the second confidence level result is less than the second confidence threshold and / or the second uncertainty result is greater than the second uncertainty threshold, the first review sample data is determined to meet the preset manual review conditions. When it is determined that the second confidence level result is greater than or equal to the second confidence threshold and the second uncertainty result is less than or equal to the second uncertainty threshold, it is determined that the first review sample data does not meet the preset manual review conditions.

[0056] Optionally, the above-mentioned manual review and annotation operation is performed on the first review sample data to obtain the second review sample data. An example is given below: the first review sample data is added to the manual review queue Q. human The system prompts users to perform manual review and precise annotation on the review sample data in the manual review queue. The review sample data obtained by the user after completing the manual review and precise annotation is used as the second review sample data. This embodiment of the invention does not limit the scope of the invention.

[0057] Alternatively, the method may further include the following steps: Move the automatically corrected samples and the manually reviewed samples (i.e., the first and second reviewed sample data) back to the labeled dataset D. L Or the pseudo-label set D P This clears the list of conflicting samples, that is, clears the low-quality sample data in the review candidate set.

[0058] Further, optionally, the above determination of whether the preset annotation convergence condition is met based on the updated pseudo-label set may include: Based on the updated pseudo-label set, determine the total number of samples corresponding to the updated pseudo-label set, and based on the updated pseudo-label set, determine the number of dedicated samples corresponding to the data samples whose labels have changed but are still in the updated pseudo-label set; Calculate the annotation change rate based on the total number of samples and the number of specific samples; Determine whether the annotation change rate is less than the preset annotation change rate threshold; When it is determined that the annotation change rate is less than the annotation change rate threshold, it is determined that the current annotation convergence condition is met. When it is determined that the annotation change rate is greater than or equal to the annotation change rate threshold, it is determined that the preset annotation convergence condition is not met.

[0059] Optionally, after determining the final data labeling results based on the updated pseudo-label set and the updated labeled data set, a complete quality assessment report is generated. This quality assessment report may include, but is not limited to, confidence scores per sample and quality indicators per category. This embodiment of the invention does not impose any limitations on these aspects.

[0060] Optionally, the rate of change of labeling can be calculated using the following formula: change rate =|Target sample size| / |D P |; Among them, change rate To indicate the rate of change; |D P | represents the total number of samples corresponding to the updated pseudo-label set; the target number of samples corresponds to the number of data samples whose labels have changed and are still in the updated pseudo-label set.

[0061] Alternatively, the method may further include the following operations: When it is determined that the current labeling convergence condition is not met, the above steps of performing sample value analysis and sample screening operations based on the data set to be labeled and the semantic embedding vector results are executed again to obtain a list of candidate pseudo-labels, and then the next round of iteration is entered.

[0062] Optionally, the determination of whether the current annotation convergence condition is met can also be achieved in the following ways: (1) Based on the stability of the number of pseudo-labels, if the change in the number of newly generated pseudo-labels in three consecutive rounds is less than 5%, then the annotation convergence condition is met; (2) Based on the stability of the validation set accuracy, if the fluctuation of the validation accuracy in three consecutive rounds is less than 0.5%, then the annotation convergence condition is met; (3) Based on the convergence of the global quality index Q, if |Q t -Q t-1 If |<0.005, then the annotation convergence condition is met. This embodiment of the invention does not impose any limitations.

[0063] As can be seen, this optional embodiment can perform manual review and annotation operations on the first review sample data when the manual review conditions are met. It matches corresponding information set update methods for the first review sample data depending on whether the manual review conditions are met or not. When the annotation convergence condition is determined based on the updated information set, the final data annotation result is determined based on the updated pseudo-label set. This improves the accuracy, reliability, and timeliness of the manual review and annotation operation, enhances the diversity and flexibility of data annotation and review methods, and further improves the accuracy and reliability of the second review sample data obtained from the manual review and annotation. This, in turn, improves the accuracy and reliability of the determined final data annotation result. Furthermore, it effectively simplifies the sample data requiring manual review, improving the efficiency of manual review and reducing annotation costs to some extent. Moreover, directly determining the final data annotation result when the annotation convergence condition is met improves the timeliness, efficiency, and accuracy of the final data annotation result output.

[0064] In another optional embodiment, the semantic embedding vector result may include the semantic embedding vector of each sample data to be labeled; further optionally, the above-mentioned sample value analysis and sample screening operations based on the set of data to be labeled and the semantic embedding vector result to obtain a candidate pseudo-label list may include: For each sample data to be labeled, based on the semantic embedding vector of the sample data to be labeled, the determined classification weight matrix information and classification bias vector information, the probability distribution prediction result of the sample data to be labeled for each preset labeling category is determined; Based on the probability distribution prediction results of the sample data to be labeled, the information entropy result of the sample data to be labeled is determined, and based on the semantic embedding vector of the sample data to be labeled and the determined first nearest neighbor sample information, the semantic density result of the sample data to be labeled is determined. Based on the information entropy and semantic density results of the sample data to be labeled, the comprehensive sample value score of the sample data to be labeled is determined. Based on the comprehensive sample value score of each unlabeled sample data, the first sample data that meets the preset high-value screening criteria is determined from all unlabeled sample data. Based on all the first sample data, a list of candidate pseudo-labels is determined.

[0065] Optional, probability distribution prediction result p i It can be calculated using the following formula: p i =Softmax(W c ·z i +b c ); Where Softmax is the normalization function, W c For classification weight matrix information, z i b is a semantic embedding vector. c The classification bias vector information is not limited in the embodiments of this invention.

[0066] Optionally, the probability distribution prediction results are illustrated as follows: Based on the semantic embedding vector, classification weight matrix information, and classification bias vector information of the sample data to be labeled, the original scores of each category to which the sample data to be labeled belongs are determined. Specifically, the semantic embedding vector and the classification weight matrix are linearly transformed and superimposed with the classification bias vector to obtain the original scores of the sample data to be labeled in the number of labeled categories C. The original scores are normalized using the Softmax function to obtain the category probability distribution of the number of labeled categories C, and the category probability distribution is used as the probability distribution prediction result of the sample data to be labeled. Each dimension component in the probability distribution prediction result represents the probability that the sample data to be labeled belongs to the corresponding labeled category, and the sum of each dimension component is 1. This embodiment of the invention does not limit the scope of the prediction.

[0067] Optionally, the information entropy result can be calculated using the following formula: ; Wherein, U(x) i ) represents the information entropy result, p i The embodiments of this invention do not limit the probability distribution prediction results.

[0068] Optionally, the semantic density result can be calculated using the following formula: ; Where D(x) i ) represents the semantic density result of the sample data to be labeled, indicating the distribution density of the sample data to be labeled in the semantic embedding space, D(x) i The first nearest neighbor sample information is determined based on the first nearest neighbor sample information, which includes the number of nearest neighbors. K and its corresponding nearest neighbor set NN K (x i ), where NN K (x i ) represents the sample data to be labeled x i The Euclidean distance is minimized in the semantic embedding space. K The set consisting of the nearest neighbor samples, K The nearest neighbor samples come from the unlabeled dataset D. U With the labeled dataset D L union of D U ∪D LIn other words, the retrieval scope of the first nearest neighbor sample is the entire sample set. Further examples are given below, such as the number of nearest neighbors. K It can be set to K =20, meaning that the semantic embedding space is used to retrieve the sample data x to be labeled. i 20 nearest neighbor samples NN 20 (x i z is used as the first nearest neighbor sample; i z j For semantic embedding vectors; further, the semantic density result D(x) i Based on the sample data to be labeled x i semantic embedding vector z i The semantic embedding vector z of its first nearest neighbor sample j The distance between them is determined, specifically, if the semantic density result D(x) i The smaller the value of D(x), the more densely packed the sample data to be labeled is in, meaning there are many similar samples around it; if the semantic density result D(x) is smaller... i The larger the value, the more likely the sample data to be labeled is in a sparse region or a boundary region, meaning that there are fewer similar samples around it. This embodiment of the invention does not impose any limitations.

[0069] Optionally, the overall sample value score can be calculated using the following formula: ; Where V(x) i This represents the comprehensive sample value score result; α , β For weighting coefficients, α=0.7 and β=0.3, but this embodiment of the invention is not limited.

[0070] Optionally, based on the comprehensive sample value score of each unlabeled sample data, the first sample data that meets the preset high-value screening conditions is determined from all unlabeled sample data. For example, the comprehensive sample value score results of all unlabeled sample data are sorted in descending order to obtain a comprehensive sample value score sequence. The unlabeled sample data in the first V (e.g., V=256) of the comprehensive sample value score sequence are taken as the first sample data that meets the preset high-value screening conditions. This embodiment of the invention does not limit this.

[0071] Optionally, the above method determines the candidate pseudo-label list based on all the first sample data. For example, candidate pseudo-label list B is constructed from all the first sample data. U The embodiments of the present invention are not limited.

[0072] Furthermore, the method may also include the following operations: Based on the comprehensive sample value score of each unlabeled sample data, sample data that does not meet the preset high value screening criteria are identified from all unlabeled sample data and are designated as low value sample data. All low-value sample data will be retained for evaluation in the next iteration round.

[0073] Optionally, nearest neighbor voting correction can also be determined in the following ways: (1) Label propagation method: The label propagation algorithm is used to perform global label diffusion on the sample similarity map in the semantic embedding space, and all low-confidence samples are processed at once. Specifically, it involves: ① Graph construction, using a high-confidence pseudo-label set D P All samples in the graph are nodes. A similarity matrix between samples is calculated, retaining only the top-50 nearest neighbors of each node and setting the rest to zero, resulting in a sparse similarity graph. The specific formula for the similarity matrix between samples is as follows: ; Among them, z i and z j Let represent the semantic embedding vectors of the i-th sample and the j-th sample, respectively; Indicate z i and z j The square of the Euclidean distance between them is used to determine the similarity; the smaller the distance, the more similar they are. σ is the Gaussian kernel scale parameter, used to control how quickly similarity decays with distance; exp is the Gaussian kernel function, which converts distance into a similarity between 0 and 1; ② Initialize labels, assigning high-confidence samples (e.g., confidence level c) to the labels. i One-hot pseudo-labels with a probability greater than 0.85 are set as deterministic labels. Low-confidence samples are initialized to a uniform distribution. Specifically, before label propagation begins, the initial probability of low-confidence samples is made equal in all categories. ③ Iterative propagation, performed according to the following formula: F (t+1) =α·S·F (t) +(1-α)·F (0) , of which F (t+1) F is the label distribution matrix of all samples after the (t+1)th iteration; (t) Let F be the label distribution matrix of all samples in the t-th iteration; (0) Let S be the initial label distribution matrix, where high-confidence samples are one-hot labeled and low-confidence samples are uniformly distributed; α is the propagation coefficient, which can be taken as α=0.9; S is the normalized similarity matrix, specifically, S=D -1 / 2 WD -1 / 2 Where W is the original similarity matrix and D is the degree matrix; further iterations until convergence can be performed, and the convergence condition can be set as ||F(t+1) -F (t) ||<1×10 -6 ; ④ Correction output: For low-confidence samples, take the largest class (argmax) in the convergence result as the correction label; (2) Graph Convolutional Network Approach: Graph convolutional networks are used to explicitly model the semantic relationships between samples. Correction and confidence reassessment are performed simultaneously using graph node classification. Specifically, a graph is first constructed, using a set of high-confidence pseudo-labels D. P All samples in the graph are nodes. The top-50 nearest neighbors of each node are retained, and the adjacency matrix is ​​symmetricized, i.e., G←G+GT, to obtain a symmetric adjacency matrix G. Next, forward propagation is performed, using the semantic embedding matrix as input to the graph convolutional network, i.e., H. (0) =[z i ], which has 256 dimensions, is further calculated layer by layer according to the following formula: H (l+1) =ReLU(D -1 / 2 GD -1 / 2 H (l) W (l) ), where H (l) For the first l The node feature matrix of the layer, W (l) ) is the first l The learnable weight matrix of the layer, D -1 / 2 GD -1 / 2 To achieve a symmetric normalized adjacency matrix, the GCN (Graph Convolutional Network) can be configured with two layers. After processing by the two layers of graph convolutional networks, the linear classification head outputs C-dimensional logits. Finally, training supervision is performed, using pseudo-labels of high-confidence samples (e.g., ci>0.85) as supervision signals. Cross-entropy loss is used to train the GCN parameters. After training, inference is performed on the entire graph, and the corrected labels and re-evaluation confidence of each sample are output. (3) Model ensemble method: Abandoning nearest neighbor correction, we use model ensemble correction instead. We use multiple models with different initializations to predict low-confidence samples, and use voting or weighted average results as correction labels. Specifically, we first train the model and initialize the same architecture model with different random seeds (e.g., seed=1~5) on the same labeled dataset D. L With the set of high-confidence pseudo-labels D P First, perform independent training, for example, training for 100 rounds to obtain multiple (e.g., 5) independent models; then perform ensemble inference, using the aforementioned multiple models to predict each low-confidence sample, for example, by using a voting method to determine the ensemble result. That is, to count the number of times each model predicts a category; or, for example, to use a soft integration approach pensemble (c)=(1 / 5)·Σp i (m) (c) This involves averaging the class probabilities output by each model to obtain the ensemble probability distribution; finally, a correction judgment is made. If the majority of classes in the voting results meet the set conditions (e.g., receiving 4 or more votes, i.e., vote), then the ensemble is determined. (cmajority) ≥4), and the majority category is inconsistent with the current pseudo-label (i.e., c). majority ≠Current i If the low-confidence sample is labeled as such, then the labeling of that sample is corrected to the majority class; simultaneously, based on the entropy value p of the ensemble probability distribution... ensemble The confidence level of the low-confidence sample is re-evaluated, but this embodiment of the invention does not limit the scope of the invention.

[0074] As can be seen, this optional embodiment can determine the first sample data that meets the preset high-value screening conditions by determining the probability distribution prediction results, information entropy results, semantic density results, and comprehensive sample value score results of the sample data to be labeled, and determine the candidate pseudo-label list based on all the first sample data. This is beneficial to improving the comprehensiveness of the method for determining the first sample data, the comprehensiveness of the parameters used to determine the first sample data, and thus effectively identifying high-value sample data. It is also beneficial to improving the accuracy and reliability of the determined first sample data (i.e., high-value sample data), thereby improving the accuracy and reliability of the determined candidate pseudo-label list.

[0075] In another optional embodiment, the candidate pseudo-label list may include at least one first sample data that meets the preset high-value screening criteria from all the sample data to be labeled; further optionally, the above-mentioned pseudo-label determination operation based on the candidate pseudo-label list to obtain a pseudo-label set may include: Based on the determined accuracy of the validation set annotation in the current iteration round, determine the adaptive confidence threshold; Based on the probability distribution prediction results of each first sample data for each preset labeled category, determine the maximum predicted probability value of each first sample data. Based on the maximum predicted probability value and adaptive confidence threshold of each first sample data, second sample data that meet the preset pseudo-label conditions are determined from all first sample data. Based on each second sample data and its maximum predicted probability value, generate pseudo-label information for each second sample data. Based on each second sample data and its pseudo-label information, determine the pseudo-label set.

[0076] Optionally, the adaptive confidence threshold can be calculated using the following formula: ; Where, δ t Let A be the adaptive confidence threshold for round t. t Let δ be the accuracy of the validation set annotation in round t, δ0 be the initial confidence threshold, and γ be the adaptive step size coefficient; further, δ0 = 0.85 and γ = 0.1, which are not limited in the embodiments of the present invention.

[0077] Optional, regarding the validation set annotation accuracy A t For example, a validation set independent of the training process is set up to evaluate the model's annotation accuracy and monitor the training process. Specifically, 1000 validation samples independent of manual annotation are reserved at once. These validation samples are independent of the training data and do not enter the already labeled dataset D. L It also does not enter the high-confidence pseudo-label set D. P After each training round, forward inference is performed on the validation set to obtain the predicted labels and predicted probability distributions of the validation set samples. The predicted labels of the validation set samples can be obtained using the following formula: ;in, Indicates the first i Predicted labels for each validation set sample. p val ( c The validation set samples belong to the category. c The predicted probability, c Indicates the label category; Furthermore, the validation set annotation accuracy A t It can be obtained through the following formula: ; in, N val Indicates the number of samples in the validation set, such as N val =1000; Indicates the first i The true labels of the validation set samples; Indicates an indicator function; Furthermore, the accuracy A of the validation set annotation will be... t The results of the t+1 round of screening are used to adjust the confidence threshold or screening strategy; simultaneously, the validation set annotation accuracy A is recorded. t The logs are used to monitor the training process. For example, if the accuracy of the validation set labeling decreases instead of increasing for three consecutive rounds, an alarm is triggered to indicate training abnormality or model degradation. This embodiment of the invention does not limit this.

[0078] Further optionally, the process of determining the second sample data that meets the preset pseudo-label conditions from all the first sample data based on the maximum predicted probability value and adaptive confidence threshold of each first sample data may include: For each first sample data, when the maximum predicted probability value of the first sample data is greater than the adaptive confidence threshold, the first sample data is determined to be the second sample data that meets the preset pseudo-label conditions.

[0079] Optional, regarding the pseudo-label information for the second sample data, an example is given below: Assume the second sample data x i The maximum predicted probability is max(p) i If max(p) i )>δ t Then the second sample data x is determined. i The pseudo-label information is The embodiments of the present invention are not limited.

[0080] Further optionally, determining the pseudo-label set based on each second sample data and its pseudo-label information may include: combining the second sample data and its pseudo-label information... Add pseudo-tag set D P The embodiments of the present invention are not limited.

[0081] Alternatively, the method may further include the following operations: Based on the maximum predicted probability value and adaptive confidence threshold of each first sample data, sample data that do not meet the preset pseudo-label conditions are identified from all the first sample data, and these sample data that do not meet the preset pseudo-label conditions are retained in the unlabeled data set D. U In the meantime, it awaits evaluation in the next iteration round.

[0082] Optionally, the adaptive confidence threshold can also be determined in the following ways: (1) Based on the Top-R percentage strategy, after each training round, the samples are sorted from high to low according to the prediction confidence, and the top R% samples with the highest prediction confidence are selected to generate pseudo-labels. The value of R can be dynamically adjusted with the training rounds, for example, from the initial 20% to 50% linearly. The initial value, termination value and growth method of R can be set according to actual needs; (2) Based on the course learning strategy, the samples are sorted from easy to difficult according to the prediction difficulty. The prediction difficulty can be defined by information entropy. The smaller the information entropy, the easier the sample is. The larger the information entropy, the more difficult the sample is. Each training round gradually expands the pseudo-label acceptance range with a fixed step size, so that the model first learns easy samples and then gradually accepts more difficult samples, thereby realizing course-style training from easy to difficult; (3) Abandon the single global threshold and use the category adaptive threshold instead. The threshold is calculated independently for each category. The confidence threshold of the cth category can be expressed as: δt (c)=δ0+γ×(1-A t (c)), where δ t (c) represents the confidence threshold for class c in the t-th training round, δ0 represents the initial threshold, γ represents the adjustment coefficient, and A t (c) represents the individual accuracy of class c in the t-th training round; further, if the accuracy of a certain class is low, its confidence threshold is increased accordingly, thereby tightening the acceptance conditions for pseudo-labels of that class; if the accuracy of a certain class is high, its confidence threshold is decreased accordingly, thereby relaxing the acceptance conditions for pseudo-labels of that class. This embodiment of the invention does not limit this.

[0083] As can be seen, this optional embodiment can determine the second sample data that meets the preset pseudo-label conditions by determining the adaptive confidence threshold and the maximum predicted probability value of the first sample data, and determine the pseudo-label set based on the second sample data and its pseudo-label information. This is beneficial to improving the comprehensiveness of the second sample data determination method and the comprehensiveness of the determination parameters used to determine the second sample data, thereby effectively identifying pseudo-label samples and improving the accuracy and reliability of the determined second sample data (i.e., pseudo-label samples). In addition, each second sample data generates matching pseudo-label information, which is beneficial to improving the targeting and accuracy of the pseudo-label information, thereby improving the accuracy and reliability of the determined pseudo-label set.

[0084] In yet another optional embodiment, the method may further include the following operations: Based on the pre-constructed labeled dataset, the supervision loss information is determined, and based on the pseudo-label set, the pseudo-label loss information is determined. Based on the determined pseudo-label loss weight information, supervision loss information, and pseudo-label loss information for the current training phase, determine the joint loss function information; Based on the joint loss function information, a parameter update operation is performed to obtain the parameter update results for the feature extraction model.

[0085] Optionally, the supervision loss information can be calculated using the following formula: ; Among them, L sup To monitor loss information, p i For the probability distribution prediction result, y i This is pseudo-label information; S 1 can be understood as starting from the labeled dataset D L A set consisting of a predetermined number of precisely labeled samples randomly sampled from the middle, | S 1| represents a set S The sample size in 1, precisely labeled with the number of samples collected.S 1| can be set to 32, or adjusted according to the training batch size, computing resources, and task requirements; additionally, if y i For the labeled dataset D L The true labeled labels of the mid-samples then provide the supervised loss information L. sup The method used is supervised cross-entropy loss, and the embodiments of the present invention are not limited thereto.

[0086] Optionally, the pseudo-label loss information can be calculated using the following formula: ; Among them, L pseudo For pseudo-label loss information, p i For the probability distribution prediction result, y i This is pseudo-label information; S 2 can be understood as starting from the set of pseudo-labels D P A set consisting of a predetermined number of randomly sampled pseudo-labeled samples, | S 2| represents a set S The number of samples in 2, the number of pseudo-label samples | S 1| can be set to 128, or adjusted according to the training batch size, computing resources, and task requirements; p max The maximum predicted probability of the i-th sample is the confidence level of that sample. δ t The adaptive confidence threshold for the t-th round of training is not limited in this embodiment of the invention.

[0087] Optionally, the joint loss function information can be calculated using the following formula: L total = L sup +λ t ·L pseudo +η·L cons ; Among them, L total For joint loss function information, L pseudo For pseudo-label loss information, L sup To monitor loss information, λ t η represents the pseudo-label loss weight information for the current training phase, and L... cons For consistency loss information; further, λ t Specifically, the dynamic weight λ t =1.0×(t / 100), where t is the current iteration round (e.g., 1~100); η is specifically the consistency loss weight, which can take the value η=0.1, but is not limited in this embodiment of the invention.

[0088] Optionally, based on the labeled dataset D Land the set of pseudo-tags D P A joint training batch set B is constructed, and then loss calculation operations (i.e., corresponding supervised loss information, pseudo-label loss information, and joint loss function information) and parameter update operations are performed. Furthermore, a joint training batch set can be constructed using some or all of the labeled data from the labeled dataset and some or all of the pseudo-label samples (i.e., second sample data) from the pseudo-label set. For example: from the labeled dataset D... L Randomly sample 32 (or other numbers) precisely labeled samples from the pseudo-label set D. P 128 (or other numbers) pseudo-label samples are randomly sampled from the dataset to construct a joint training batch set B. This embodiment of the invention does not limit the scope of the sample.

[0089] Optionally, based on the joint loss function information, the above-mentioned parameter update operation is performed to obtain the parameter update results for the feature extraction model. For example, the model parameters are updated using the Adam optimizer, specifically according to the learning rate lr = 1 × 10⁻⁶. -4 Weight decay 1×10 -5 The configuration is configured to iteratively update the model parameters through gradient backpropagation; furthermore, the updated model parameters can be used for subsequent multi-view processing operations, which is not limited in this embodiment of the invention.

[0090] As can be seen, this optional embodiment can provide a parameter update method for feature extraction models applied to sample data annotation. By performing parameter update operations based on the supervised loss information of the labeled dataset and the pseudo-label loss information of the pseudo-label dataset, it is beneficial to improve the comprehensiveness and applicability of the model update method, the comprehensiveness of the parameters determined by the joint loss function information, and thus the accuracy and reliability of parameter updates. This, in turn, helps to improve the accuracy and reliability of subsequent data annotation applications based on the updated parameters.

[0091] In another optional embodiment, the pseudo-label set may include at least one second sample data and its pseudo-label information that satisfy the preset pseudo-label conditions from all the sample data to be labeled; further optionally, the above-mentioned multi-view processing and special sample processing operations based on the pseudo-label set to obtain the effective sample data set may include: Based on the second sample data in the pseudo-label set, determine one or more target second sample data; For each target second sample data, at least two enhanced view sample results corresponding to the target second sample data are determined according to the data format type of the target second sample data; Based on all augmented view sample results corresponding to the second sample data of the target, determine the model prediction distribution of each augmented view sample result; Based on the distribution results of all model predictions corresponding to the second sample data of the target, determine the prediction difference information corresponding to the second sample data of the target; Based on the prediction difference information of the second sample data for each target, third sample data that does not meet the preset prediction conflict conditions are determined from all the second sample data for all targets, and these are taken as valid sample data; and / or, Based on the prediction difference information of the second sample data for each target, the fourth sample data that meets the preset prediction conflict conditions is determined from all the second sample data of the targets and is used as the conflict sample data; based on the conflict sample data, the conflict sample data removal operation is performed on the pseudo-label set to obtain the valid sample data; Based on all valid sample data, determine the set of valid sample data.

[0092] Optionally, the target second sample data can be obtained by performing multi-view processing and special sample processing operations on all second sample data in the pseudo-label set; or it can be obtained by performing multi-view processing and special sample processing operations on a portion of the second sample data randomly determined in the pseudo-label set; furthermore, the target second sample data can correspond to the second sample data in the joint training batch set determined by the pseudo-label set, and this embodiment of the invention does not limit it.

[0093] Optionally, the enhanced viewpoint sample results for each target's second sample data can be generated using the aforementioned determined joint training batch set and parameter update results. This embodiment of the invention does not impose any limitations on this.

[0094] Optionally, the data format type of the target second sample data may include, but is not limited to, one or more of image type, text type, voice type and other data format types, and the embodiments of the present invention do not limit it.

[0095] Further optionally, the determination of at least two enhanced viewpoint sample results corresponding to the second sample data of the target based on the data format type of the second sample data of the target may include: Based on the data format type of the second sample data of the target, determine at least two pre-defined target modal enhancement schemes for that data format type; Based on each target modal enhancement scheme, the second sample data of the target are processed by enhancement viewpoint processing to obtain the enhancement viewpoint sample results corresponding to each target modal enhancement scheme. The enhanced viewpoint sample results corresponding to all target modal enhancement schemes are determined as at least two enhanced viewpoint sample results corresponding to the second sample data of the target.

[0096] Optionally, the target second sample data, the target modality enhancement scheme, and the enhanced viewpoint sample result can be illustrated as follows: If the target second sample data is in image format, the target modality enhancement scheme can be a combination of viewpoint processing actions: "random horizontal flip + random cropping + color jittering," along with specific parameter values ​​based on this combination. For example, "random horizontal flipping with a probability of 0.5; random cropping with an area ratio of 0.8-1.0; and color jittering with brightness 0.2, contrast 0.2, saturation 0.2, and hue 0.1." Furthermore, the enhanced viewpoint sample result can be understood as the enhanced sample obtained by performing the viewpoint enhancement operation "random horizontal flipping with a probability of 0.5; random cropping with an area ratio of 0.8-1.0; and color jittering with brightness 0.2, contrast 0.2, saturation 0.2, and hue 0.1" on the image-type target second sample data. The target modality enhancement scheme for image types can also... Other perspective processing action type combinations and / or other specific parameter values ​​based on these perspective processing action type combinations can be set, which will not be listed in detail here. If the data format of the target second sample data is text, the target modality enhancement scheme can be a perspective processing action type combination of "random synonym replacement and random word deletion" and specific parameter values ​​based on this perspective processing action type combination. For example, random synonym replacement with a replacement ratio of 15% and random word deletion with a deletion ratio of 10%. Furthermore, the enhanced perspective sample result can be understood as: the enhanced sample obtained after performing the perspective enhancement operation of "random synonym replacement with a replacement ratio of 15% and random word deletion with a deletion ratio of 10%" on the target second sample data of text type. The target modality enhancement scheme corresponding to text type can also set other perspective processing action type combinations and / or other specific parameter values ​​based on these perspective processing action type combinations, which will not be listed in detail here. The embodiments of the present invention do not limit this.

[0097] Optional, for example: for the target second sample data x i According to its data mode, the corresponding enhancement operator combination is applied to generate M=2 enhancement perspectives; each sample data only performs enhancement of its mode (such as image samples only perform image enhancement, text samples only perform text enhancement, etc.), and the two enhancement perspectives are obtained by two independent random samplings of the same enhancement operator combination: (1) when the target second sample data x i When preparing image samples: Random horizontal flipping (e.g., probability 0.5), random cropping (e.g., area ratio 0.8-1.0), and color dithering (e.g., brightness 0.2, contrast 0.2, saturation 0.2, hue 0.1) are applied sequentially. This random combination process is executed independently twice to obtain the first enhanced viewpoint sample result. and second enhanced perspective sample results (2) When the target second sample data x i When processing text samples: random synonym replacement (e.g., replacement ratio 15%) and random word deletion (e.g., deletion ratio 10%) are applied. This random combination process is executed independently twice to obtain the first enhanced perspective sample result. and second enhanced perspective sample results The embodiments of the present invention are not limited.

[0098] Optional, for example: Suppose the target is the second sample data x i All its enhanced view sample results include the first enhanced view sample results. and second enhanced perspective sample results The results of the first enhanced perspective sample and second enhanced perspective sample results The relevant models were input separately for analysis to obtain the first-view augmentation sample results. The corresponding first model predicted the distribution result Second enhanced perspective sample results The corresponding second model predicted the distribution results. Furthermore, the prediction difference information is determined by combining the distribution prediction results of the first model and the distribution prediction results of the second model (i.e., the distribution prediction results of all models for the second sample data of the target). Based on this, the prediction difference information of the second sample data of the target can be calculated using the following formula: ; Where, d i To predict discrepancy information, Predict the distribution results for the first model. The distribution result is predicted by the second model; in addition, the method for determining the prediction difference information when the target second sample data corresponds to other numbers of enhanced view sample results can be obtained in the same way, and the embodiments of the present invention are not limited.

[0099] Alternatively, the method may further include the following operations: Based on the results of all augmented viewpoint samples corresponding to the second sample data of the target and the total batch size corresponding to the determined joint training batch set, the consistency loss information is determined.

[0100] Further optional, consistency loss information L cons It can be calculated using the following formula: ; Among them, S total S represents the total batch size corresponding to the joint training batch set. total The joint training batch set can be referenced above, including the data from the labeled dataset D.L The set consisting of a predetermined number of precisely labeled samples randomly sampled from the set of pseudo-labels D. P The set consisting of a predetermined number of randomly sampled pseudo-labeled samples, i.e., S. total =|S1|+|S2|;S total Other values ​​can also be determined according to the actual situation, and the embodiments of the present invention are not limited thereto.

[0101] Further optionally, the process of determining the third sample data that does not meet the preset prediction conflict condition from all the target second sample data based on the prediction difference information of each target second sample data may include: Based on the prediction difference information of each target second sample data, target second sample data with prediction difference values ​​less than or equal to a preset consistency threshold are determined from all target second sample data and are used as third sample data that do not meet the preset prediction conflict conditions.

[0102] Further optionally, the process of determining the fourth sample data that meets the preset prediction conflict condition from all the target second sample data based on the prediction difference information of each target second sample data may include: Based on the prediction difference information of each target second sample data, the target second sample data with a prediction difference value greater than the preset consistency threshold is determined from all target second sample data and used as the fourth sample data that meets the preset prediction conflict condition.

[0103] Further, optionally, the above-mentioned process of removing conflicting sample data from the pseudo-label set based on conflicting sample data to obtain valid sample data may include: Remove conflicting sample data and their pseudo-label information from the pseudo-label set to obtain a pseudo-label set after removal; determine all target second sample data or all second sample data in the pseudo-label set after removal as valid sample data.

[0104] Further, optionally, the information about removing conflict sample data from the pseudo-label set based on the conflict sample data is recorded in the conflict log, and then awaits manual review or re-evaluation in the next iteration. This embodiment of the invention does not limit this.

[0105] Optionally, after determining the third sample data that does not meet the preset prediction conflict condition from all the second sample data of all targets based on the prediction difference information of the second sample data of each target, and using it as valid sample data, the valid sample data participates normally in subsequent joint training, and the consistency loss information L is determined based on the valid sample data participating in joint training. cons And the consistency loss information L cons The embodiments of the present invention do not limit the total loss.

[0106] Optionally, the predicted variance information can also be determined in the following ways: (1) Based on the symmetric KL divergence method, specifically, the predicted difference information can be determined using symmetric KL divergence, that is, the KL divergence between the first-view probability distribution and the second-view probability distribution is calculated respectively, and the sum of the two is used as the predicted difference information. Specifically, it can be calculated using the following formula: ; Wherein, KL(·||·) represents KL divergence. Symmetric KL divergence can simultaneously measure the two-way difference between the predicted distributions from two perspectives (i.e., the model's predicted distribution results). (2) Based on the Jensen-Shannon divergence method, which is a symmetric and bounded measure, it is more conducive to threshold setting than KL divergence and can avoid the instability of difference measurement due to the overly extreme predicted distribution from a certain perspective. (3) Based on the hard label consistency method, the argmax prediction results of the same sample under multiple perspectives are required to be completely consistent; if the argmax prediction results of multiple perspectives are inconsistent, it is determined that the sample has a prediction conflict; if the argmax prediction results of multiple perspectives are consistent, it is determined that the sample meets the consistency requirements. The hard label consistency method is suitable for scenarios with extremely high labeling accuracy requirements. This embodiment of the invention does not limit this.

[0107] As can be seen, this optional embodiment can obtain corresponding augmentation perspective results for different sample data format types, thereby determining the model prediction distribution results. It determines valid sample data based on the prediction difference information of all model prediction distribution results, which is beneficial to improving the comprehensiveness of the method for determining valid sample data, the comprehensiveness of the parameters used to determine valid sample data, and the relevance and diversity of the data augmentation perspective results. This, in turn, improves the accuracy and reliability of the determined prediction difference information, thereby improving the accuracy and reliability of the valid sample data determined based on the prediction difference information. Furthermore, it enhances the diversity and flexibility of the method for determining valid sample data.

[0108] In another optional embodiment, the effective sample data set may include at least one effective sample data set, which may include a third sample data set from all unannotated sample data sets that does not meet the preset prediction conflict condition; further optionally, the above-mentioned multi-level annotation quality assessment operation based on the effective sample data set to obtain the quality assessment result may include: Based on the probability distribution prediction results of each valid sample data, determine the first confidence level result and the first uncertainty result of each valid sample data; Based on the first confidence result of each valid sample data, sample data that meet the preset low confidence condition are determined from all valid sample data and are used as low-quality sample data; and / or, based on the first uncertainty result of each valid sample data, sample data that meet the preset high uncertainty condition are determined from all valid sample data and are used as low-quality sample data. Based on all low-quality sample data, a candidate set for review was determined; Based on the first confidence level results of all valid sample data, determine the confidence level variance for each preset labeling category, and determine the quality warning labeling category based on the confidence level variance for each preset labeling category and the sample size. Based on all valid sample data and their first confidence level and first uncertainty results, determine the global quality index information; The quality assessment results are determined based on the candidate set for review, the quality warning label categories, and the global quality indicator information.

[0109] Optionally, the first confidence level result can be calculated using the following formula: e i =max(p i ); where e i For the first confidence level result, p i This represents the prediction result for the probability distribution.

[0110] Optionally, the first uncertainty result can be calculated using the following formula: ; Among them, u i This represents the first uncertainty result; C is the number of labeled categories; c represents the category index, c=1,2,…,C; p i This represents the prediction result for the probability distribution.

[0111] Further optionally, the process of determining sample data that meets the preset low confidence condition from all valid sample data based on the first confidence result of each valid sample data may include: Based on the first confidence level result of each valid sample data, valid sample data with a first confidence level less than a preset first confidence level threshold are determined from all valid sample data and are used as sample data that meet the preset low confidence level condition.

[0112] Further optionally, the process of determining the sample data that meets the preset high uncertainty condition from all valid sample data based on the first uncertainty result of each valid sample data may include: Based on the first uncertainty result of each valid sample data, valid sample data whose first uncertainty result is greater than a preset first uncertainty threshold are determined from all valid sample data and used as sample data that meet the preset high uncertainty condition.

[0113] Optionally, sample data that meets the preset low confidence condition is marked as low confidence sample, and sample data that meets the preset high uncertainty condition is marked as high uncertainty sample. Further, low quality sample data includes low confidence sample and / or high uncertainty sample, which is not limited in this embodiment of the invention.

[0114] Further, optionally, determining the quality warning label category based on the confidence variance result and sample size for each preset label category may include: For each preset labeling category, when the confidence variance of the preset labeling category is greater than the preset confidence variance threshold and the number of samples of the preset labeling category is greater than the preset number of samples threshold, the preset labeling type is determined to be a quality warning labeling category.

[0115] Alternatively, the method may further include the following operations: After determining the quality warning label category, a focused review operation is performed for that quality warning label category.

[0116] Optionally, the confidence variance result can be calculated using the following formula: ; in, The confidence level variance result; variance is the variance function; D c This embodiment of the invention does not limit the scope of all pseudo-label samples, i.e., valid sample data, for the preset labeling category c.

[0117] Optionally, global quality metric information can be calculated using the following formula: ; Where Q represents global quality metric information, |D P | represents the set of pseudo-labels D p The corresponding number of samples, e i For the first confidence level result, u i This is the first uncertainty outcome.

[0118] Optionally, the first uncertainty result can also be determined in the following ways: (1) Based on the Bayesian neural network approach, the prediction variance is calculated by using a Bayesian neural network and multiple Monte Carlo Dropout (MC Dropout) samplings. This prediction variance is then used as the first uncertainty result. Furthermore, the prediction variance can be calculated using the following formula: U(x i )=Var(p i (1) , p i (2) , ..., p i (L) ), where U(x) i p represents the first uncertainty result for the i-th sample to be labeled. i (l) For the first l The probability distribution prediction result obtained from the sampling, where L is the number of samplings and Var() represents the variance operation; (2) Using the deep ensemble method, train multiple models with different initializations, and use the standard deviation of the predicted distribution of each model as the uncertainty measure. Specifically, use multiple models to predict the same sample to obtain multiple probability distribution prediction results, and calculate their standard deviation as the first uncertainty result of the sample. (3) Using Evidential Deep Learning, the parameters of the Dirichlet distribution (hyperparameters of the Dirichlet distribution) are directly output by Evidential Deep Learning, and the amount of evidence of the distribution is used as the uncertainty measure. This scheme can obtain the prediction category and uncertainty at the same time without multiple sampling. This embodiment of the invention does not limit this.

[0119] As can be seen, this optional embodiment can determine the quality assessment result by identifying the review candidate set, the quality warning label category, and the global quality indicator information. It identifies low-quality sample data through the first confidence result and the first uncertainty result of the valid sample data. This is beneficial to improving the diversity and specificity of the basis for determining low-quality sample data, and the diversity and specificity of the methods for determining low-quality sample data, thereby improving the comprehensiveness and accuracy of the identified low-quality sample data. Furthermore, it is beneficial to improving the comprehensiveness of the methods for determining the quality warning label category, thereby improving the accuracy and reliability of the identified quality warning label category. Additionally, it is beneficial to improving the comprehensiveness of the methods for determining the global quality indicator information, thereby improving the accuracy and reliability of the identified global quality indicator information. Finally, it is beneficial to improving the comprehensiveness of the basis for determining the quality assessment result, thereby improving the accuracy and reliability of the determined quality assessment result.

[0120] Example 2 Please see Figure 2 , Figure 2 This is a schematic diagram of the structure of a data annotation device disclosed in an embodiment of the present invention. Figure 2 The described apparatus may include a server, wherein the server includes a local server or a cloud server, and the embodiments of the present invention are not limited thereto. Figure 2 As shown, the data annotation device may include: The basic information determination module 301 is used to determine the set of data to be labeled and its semantic embedding vector results that need to be labeled. The set of data to be labeled includes at least one sample data to be labeled.

[0121] The candidate pseudo-label determination module 302 is used to perform sample value analysis and sample screening operations based on the dataset to be labeled and the semantic embedding vector results, and obtain a list of candidate pseudo-labels.

[0122] The pseudo-label determination module 303 is used to perform a pseudo-label determination operation based on the candidate pseudo-label list to obtain a pseudo-label set.

[0123] The effective sample determination module 304 is used to perform multi-view processing and special sample processing operations based on the pseudo-label set to obtain the effective sample data set.

[0124] The quality assessment module 305 is used to perform multi-level annotation quality assessment operations based on the valid sample data set to obtain the quality assessment results.

[0125] The final annotation result determination module 306 is used to determine the final data annotation result based on the quality assessment result.

[0126] It is evident that implementation Figure 2 The described data annotation device can achieve intelligent data annotation through sample value analysis and sample screening, pseudo-label identification, multi-view processing and special sample processing, multi-level annotation quality assessment, and final data annotation result determination. This is beneficial to improving the comprehensiveness of data annotation processing methods, thereby improving the accuracy and reliability of data annotation, improving the efficiency and convenience of data annotation, and thus improving the quality of data annotation and achieving a synergistic improvement in annotation efficiency and quality under weak supervision.

[0127] In an optional embodiment, the quality assessment result includes a review candidate set, which includes at least one low-quality sample data; and the final annotation result determination module 306 determines the final data annotation result based on the quality assessment result in the following specific ways: Based on low-quality sample data, determine the information of the second nearest neighbor sample, and based on the information of the second nearest neighbor sample, determine the highest frequency labeling category that meets the preset highest occurrence frequency condition; Based on the annotation information and the most frequent annotation category of low-quality sample data, determine whether the low-quality sample data meets the preset automatic correction category conditions; When it is determined that the low-quality sample data meets the automatic correction category conditions, the automatic correction category operation is performed on the low-quality sample data according to the most frequent labeled category to obtain the automatically corrected sample data, which is used as the first verification sample data. When it is determined that low-quality sample data does not meet the automatic correction category conditions, the low-quality sample data is identified as the first review sample data. Based on the first review sample data, the final data annotation results are determined.

[0128] It is evident that implementation Figure 3 The described apparatus can match corresponding first verification sample data determination operations for low-quality sample data that meets the automatic correction category conditions and for low-quality sample data that does not meet the automatic correction category conditions. Then, it determines the final data annotation result based on the first verification sample data. This is beneficial to improving the comprehensiveness of the automatic correction annotation category method, improving the execution accuracy and timeliness of the automatic correction annotation category operation, and thus improving the accuracy, reliability, and timeliness of sample data correction. This, in turn, improves the accuracy and reliability of the determined first verification sample data, further improves the accuracy and reliability of the determined final data annotation result, and also improves the efficiency and convenience of sample data correction.

[0129] In another optional embodiment, the final annotation result determination module 306 determines the final data annotation result based on the first review sample data in the following specific ways: Determine the second confidence level and the second uncertainty level of the first review sample data, and based on the second confidence level and the second uncertainty level, determine whether the first review sample data meets the preset manual review conditions; When it is determined that the first review sample data does not meet the conditions for manual review, an update operation is performed on the pseudo-label set based on the first review sample data to obtain the updated pseudo-label set; When it is determined that the first review sample data meets the conditions for manual review, the manual review and annotation operation is performed on the first review sample data to obtain the second review sample data; the second review sample data is added to the pre-constructed labeled data set to obtain the updated labeled data set. Based on the updated set of pseudo-labels, determine whether the preset label convergence condition is met. When it is determined that the current annotation convergence condition is met, the final data annotation result is determined based on the updated pseudo-label set and the updated labeled data set.

[0130] It is evident that implementation Figure 3 The described apparatus can also perform manual verification and annotation operations on the first verification sample data when the manual verification conditions are met. It matches corresponding information set update methods for the first verification sample data depending on whether the manual verification conditions are met or not. When the current annotation convergence condition is determined based on the updated information set, the final data annotation result is determined based on the updated pseudo-label set. This improves the accuracy, reliability, and timeliness of the manual verification and annotation operation, enhances the diversity and flexibility of data annotation verification methods, and consequently improves the accuracy and reliability of the second verification sample data obtained through manual verification and annotation. This, in turn, improves the accuracy and reliability of the determined final data annotation result. Furthermore, it effectively reduces the sample data requiring manual verification, improving the efficiency and annotation cost of manual verification to some extent. Moreover, directly determining the final data annotation result when the annotation convergence condition is met improves the timeliness, efficiency, and accuracy of the final data annotation result output.

[0131] In another optional embodiment, the semantic embedding vector result includes the semantic embedding vector of each sample data to be labeled; and the candidate pseudo-label determination module 302 performs sample value analysis and sample screening operations based on the set of data to be labeled and the semantic embedding vector result to obtain the candidate pseudo-label list in the following specific ways: For each sample data to be labeled, based on the semantic embedding vector of the sample data to be labeled, the determined classification weight matrix information and classification bias vector information, the probability distribution prediction result of the sample data to be labeled for each preset labeling category is determined; Based on the probability distribution prediction results of the sample data to be labeled, the information entropy result of the sample data to be labeled is determined, and based on the semantic embedding vector of the sample data to be labeled and the determined first nearest neighbor sample information, the semantic density result of the sample data to be labeled is determined. Based on the information entropy and semantic density results of the sample data to be labeled, the comprehensive sample value score of the sample data to be labeled is determined. Based on the comprehensive sample value score of each unlabeled sample data, the first sample data that meets the preset high-value screening criteria is determined from all unlabeled sample data. Based on all the first sample data, a list of candidate pseudo-labels is determined.

[0132] It is evident that implementation Figure 3The described apparatus can also determine the first sample data that meets the preset high-value screening conditions by determining the probability distribution prediction results, information entropy results, semantic density results, and comprehensive sample value score results of the sample data to be labeled, and determine the candidate pseudo-label list based on all the first sample data. This is beneficial to improving the comprehensiveness of the method for determining the first sample data, the comprehensiveness of the parameters used to determine the first sample data, and thus effectively identifying high-value sample data. It is also beneficial to improving the accuracy and reliability of the determined first sample data (i.e., high-value sample data), thereby improving the accuracy and reliability of the determined candidate pseudo-label list.

[0133] In another optional embodiment, the candidate pseudo-label list includes at least one first sample data that meets the preset high-value screening criteria from all the sample data to be labeled; and the pseudo-label determination module 303 performs a pseudo-label determination operation based on the candidate pseudo-label list to obtain the pseudo-label set in the following specific ways: Based on the determined accuracy of the validation set annotation in the current iteration round, determine the adaptive confidence threshold; Based on the probability distribution prediction results of each first sample data for each preset labeled category, determine the maximum predicted probability value of each first sample data. Based on the maximum predicted probability value and adaptive confidence threshold of each first sample data, second sample data that meet the preset pseudo-label conditions are determined from all first sample data. Based on each second sample data and its maximum predicted probability value, generate pseudo-label information for each second sample data. Based on each second sample data and its pseudo-label information, determine the pseudo-label set.

[0134] It is evident that implementation Figure 3 The described apparatus can also determine second sample data that meets preset pseudo-label conditions by determining an adaptive confidence threshold and the maximum predicted probability value of the first sample data, and determine a set of pseudo-labels based on the second sample data and its pseudo-label information. This is beneficial to improving the comprehensiveness of the second sample data determination method and the comprehensiveness of the parameters used to determine the second sample data, thereby effectively identifying pseudo-label samples and improving the accuracy and reliability of the determined second sample data (i.e., pseudo-label samples). In addition, each second sample data generates matching pseudo-label information, which is beneficial to improving the targeting and accuracy of the pseudo-label information, thereby improving the accuracy and reliability of the determined set of pseudo-labels.

[0135] In yet another alternative embodiment, such as Figure 3 As shown, the device may further include: The model parameter optimization module 307 is used to determine the supervised loss information based on the pre-constructed labeled dataset and the pseudo-label loss information based on the pseudo-label dataset; determine the joint loss function information based on the determined pseudo-label loss weight information, supervised loss information and pseudo-label loss information in the current training stage; and perform parameter update operation based on the joint loss function information to obtain the parameter update results for the feature extraction model.

[0136] It is evident that implementation Figure 3 The described apparatus can also provide a parameter update method for feature extraction models applied to sample data annotation. Based on the supervised loss information of the labeled dataset and the pseudo-label loss information of the pseudo-label dataset, the parameter update operation is performed, which helps to improve the comprehensiveness and applicability of the model update method, improves the comprehensiveness of the parameters determined by the joint loss function information, and thus helps to improve the accuracy and reliability of parameter updates. This, in turn, helps to improve the accuracy and reliability of subsequent data annotation applications based on the model after parameter updates.

[0137] In another optional embodiment, the pseudo-label set includes at least one second sample data and its pseudo-label information that meet the preset pseudo-label conditions from all the sample data to be labeled; and the effective sample determination module 304 obtains the effective sample data set by performing multi-view processing and special sample processing operations based on the pseudo-label set in the following specific ways: Based on the second sample data in the pseudo-label set, determine one or more target second sample data; For each target second sample data, at least two enhanced view sample results corresponding to the target second sample data are determined according to the data format type of the target second sample data; Based on all augmented view sample results corresponding to the second sample data of the target, determine the model prediction distribution of each augmented view sample result; Based on the distribution results of all model predictions corresponding to the second sample data of the target, determine the prediction difference information corresponding to the second sample data of the target; Based on the prediction difference information of the second sample data for each target, third sample data that does not meet the preset prediction conflict conditions are determined from all the second sample data for all targets, and these are taken as valid sample data; and / or, Based on the prediction difference information of the second sample data for each target, the fourth sample data that meets the preset prediction conflict conditions is determined from all the second sample data of the targets and is used as the conflict sample data; based on the conflict sample data, the conflict sample data removal operation is performed on the pseudo-label set to obtain the valid sample data; Based on all valid sample data, determine the set of valid sample data.

[0138] It is evident that implementation Figure 3 The described apparatus can also obtain corresponding augmentation perspective results for different sample data format types, thereby determining the model prediction distribution results. Based on the prediction difference information of all model prediction distribution results, it determines the effective sample data, which helps to improve the comprehensiveness of the method for determining effective sample data, the comprehensiveness of the parameters used to determine effective sample data, and the relevance and diversity of the data augmentation perspective results. This, in turn, helps to improve the accuracy and reliability of the determined prediction difference information, thereby improving the accuracy and reliability of the effective sample data determined based on the prediction difference information. Furthermore, it helps to improve the diversity and flexibility of the method for determining effective sample data.

[0139] In another optional embodiment, the effective sample data set includes at least one effective sample data set, which includes a third sample data set among all unannotated sample data sets that does not meet the preset prediction conflict condition; and the quality assessment module 305 performs a multi-level annotation quality assessment operation based on the effective sample data set to obtain the quality assessment result in the following specific ways: Based on the probability distribution prediction results of each valid sample data, determine the first confidence level result and the first uncertainty result of each valid sample data; Based on the first confidence result of each valid sample data, sample data that meet the preset low confidence condition are determined from all valid sample data and are used as low-quality sample data; and / or, based on the first uncertainty result of each valid sample data, sample data that meet the preset high uncertainty condition are determined from all valid sample data and are used as low-quality sample data. Based on all low-quality sample data, a candidate set for review was determined; Based on the first confidence level results of all valid sample data, determine the confidence level variance for each preset labeling category, and determine the quality warning labeling category based on the confidence level variance for each preset labeling category and the sample size. Based on all valid sample data and their first confidence level and first uncertainty results, determine the global quality index information; The quality assessment results are determined based on the candidate set for review, the quality warning label categories, and the global quality indicator information.

[0140] It is evident that implementation Figure 3The described apparatus can also determine the quality assessment result by identifying the review candidate set, the quality warning label category, and the global quality indicator information. It identifies low-quality sample data based on the first confidence level and the first uncertainty level of the valid sample data. This improves the specificity of the basis for identifying low-quality sample data, enhances the diversity and specificity of the methods for identifying low-quality sample data, and consequently improves the comprehensiveness and accuracy of the identified low-quality sample data. Furthermore, it improves the comprehensiveness of the methods for determining the quality warning label category, thereby improving the accuracy and reliability of the identified quality warning label category. Additionally, it improves the comprehensiveness of the methods for determining the global quality indicator information, thereby improving the accuracy and reliability of the identified global quality indicator information. Finally, it improves the comprehensiveness of the basis for determining the quality assessment result, thereby improving the accuracy and reliability of the determined quality assessment result.

[0141] Example 3 Please see Figure 4 , Figure 4 This is a schematic diagram of the structure of another data annotation device disclosed in an embodiment of the present invention. Figure 4 The described apparatus may include a server, wherein the server includes a local server or a cloud server, and the embodiments of the present invention are not limited thereto. Figure 4 As shown, the device may include: Memory 401 storing executable program code; Processor 402 coupled to memory 401; Furthermore, it may also include an input interface 403 coupled to the processor 402 and an output interface 404; The processor 402 calls the executable program code stored in the memory 401 to execute the steps in the data annotation method described in Embodiment 1.

[0142] Example 4 This invention discloses a computer storage medium storing computer instructions, wherein when the computer instructions are invoked, they are used to execute the steps in a data annotation method described in Embodiment 1.

[0143] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in a data annotation method described in Embodiment 1.

[0144] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0145] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), one-time programmable read-only memory (OTPROM), electrically-Erasable Programmable Read-Only Memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0146] Finally, it should be noted that the data annotation method, apparatus, and storage medium disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data annotation method, characterized in that, The method includes: Determine the set of data to be labeled and its semantic embedding vector results that need to be labeled, wherein the set of data to be labeled includes at least one sample data to be labeled; Based on the dataset to be labeled and the semantic embedding vector results, perform sample value analysis and sample screening operations to obtain a candidate pseudo-label list; Based on the candidate pseudo-label list, perform a pseudo-label determination operation to obtain a pseudo-label set; Based on the pseudo-label set, perform multi-view processing and special sample processing operations to obtain a valid sample data set; Based on the set of valid sample data, perform a multi-level annotation quality assessment operation to obtain the quality assessment results; Based on the quality assessment results, the final data labeling results are determined.

2. The data annotation method according to claim 1, characterized in that, The quality assessment results include a review candidate set, which includes at least one low-quality sample data. And, determining the final data annotation result based on the quality assessment result includes: Based on the low-quality sample data, determine the second nearest neighbor sample information, and based on the second nearest neighbor sample information, determine the highest frequency labeling category that meets the preset highest occurrence frequency condition; Based on the annotation information of the low-quality sample data and the most frequent annotation category, determine whether the low-quality sample data meets the preset automatic correction category conditions; When it is determined that the low-quality sample data meets the automatic correction category conditions, the automatic correction category operation is performed on the low-quality sample data according to the highest frequency labeling category to obtain the automatically corrected sample data, which is used as the first verification sample data. When it is determined that the low-quality sample data does not meet the automatic correction category conditions, the low-quality sample data is identified as the first review sample data; Based on the first review sample data, the final data annotation result is determined.

3. The data annotation method according to claim 2, characterized in that, The step of determining the final data annotation result based on the first review sample data includes: Determine the second confidence level result and the second uncertainty result of the first review sample data, and based on the second confidence level result and the second uncertainty result, determine whether the first review sample data meets the preset manual review conditions; When it is determined that the first review sample data does not meet the manual review conditions, an update operation is performed on the pseudo-label set based on the first review sample data to obtain the updated pseudo-label set; When it is determined that the first review sample data meets the manual review conditions, a manual review and annotation operation is performed on the first review sample data to obtain the second review sample data; the second review sample data is added to the pre-constructed labeled data set to obtain the updated labeled data set. Based on the updated set of pseudo-labels, determine whether the preset label convergence condition is met. When it is determined that the current annotation convergence condition is met, the final data annotation result is determined based on the updated pseudo-label set and the updated labeled data set.

4. A data annotation method according to any one of claims 1-3, characterized in that, The semantic embedding vector result includes the semantic embedding vector of each of the labeled sample data; And, based on the dataset to be labeled and the semantic embedding vector results, the step of performing sample value analysis and sample screening operations to obtain a candidate pseudo-label list includes: For each of the sample data to be labeled, the probability distribution prediction result of the sample data to be labeled for each preset labeling category is determined based on the semantic embedding vector of the sample data to be labeled, the determined classification weight matrix information and the classification bias vector information. Based on the probability distribution prediction results of the sample data to be labeled, the information entropy result of the sample data to be labeled is determined, and based on the semantic embedding vector of the sample data to be labeled and the determined first nearest neighbor sample information, the semantic density result of the sample data to be labeled is determined. Based on the information entropy and semantic density results of the sample data to be labeled, the comprehensive sample value score of the sample data to be labeled is determined. Based on the comprehensive sample value score of each of the unlabeled sample data, the first sample data that meets the preset high value screening conditions is determined from all the unlabeled sample data; Based on all the first sample data, a list of candidate pseudo-labels is determined.

5. The data annotation method according to claim 4, characterized in that, The candidate pseudo-label list includes at least one first sample data that meets the preset high-value screening criteria from all the sample data to be labeled; And, the step of performing a pseudo-label determination operation based on the candidate pseudo-label list to obtain a pseudo-label set includes: Based on the determined accuracy of the validation set annotation in the current iteration round, determine the adaptive confidence threshold; Based on the probability distribution prediction results of each first sample data for each preset labeled category, determine the maximum predicted probability value of each first sample data. Based on the maximum predicted probability value of each first sample data and the adaptive confidence threshold, second sample data that meets the preset pseudo-label conditions are determined from all the first sample data. Based on each second sample data and its maximum predicted probability value, generate pseudo-label information for each second sample data. Based on each of the second sample data and its pseudo-label information, a set of pseudo-labels is determined; Furthermore, the method further includes: Based on the pre-constructed labeled dataset, the supervision loss information is determined, and based on the pseudo-label set, the pseudo-label loss information is determined. Based on the determined pseudo-label loss weight information, the supervision loss information, and the pseudo-label loss information for the current training phase, the joint loss function information is determined. Based on the joint loss function information, a parameter update operation is performed to obtain the parameter update results for the feature extraction model.

6. The data annotation method according to claim 5, characterized in that, The pseudo-label set includes at least one second sample data and its pseudo-label information from all the sample data to be labeled that meet the preset pseudo-label conditions; And, the step of performing multi-view processing and special sample processing operations based on the pseudo-label set to obtain a valid sample data set includes: Based on the second sample data in the pseudo-label set, determine one or more target second sample data; For each target second sample data, at least two enhanced view sample results corresponding to the target second sample data are determined according to the data format type of the target second sample data; Based on all the enhanced view sample results corresponding to the second sample data of the target, determine the model prediction distribution result for each enhanced view sample result; Based on all the model prediction distribution results corresponding to the second sample data of the target, determine the prediction difference information corresponding to the second sample data of the target; Based on the prediction difference information of each of the target second sample data, third sample data that does not meet the preset prediction conflict condition is determined from all the target second sample data and is taken as valid sample data; and / or, Based on the prediction difference information of each target second sample data, a fourth sample data that meets the preset prediction conflict condition is determined from all the target second sample data and used as conflict sample data; based on the conflict sample data, a conflict sample data removal operation is performed on the pseudo-label set to obtain valid sample data; Based on all the aforementioned valid sample data, determine the set of valid sample data.

7. A data annotation method according to claim 6, characterized in that, The effective sample data set includes at least one effective sample data, and the effective sample data includes a third sample data that does not meet the preset prediction conflict condition among all the unlabeled sample data; And, the step of performing a multi-level annotation quality assessment operation based on the valid sample data set to obtain the quality assessment result includes: Based on the probability distribution prediction results of each of the valid sample data, determine the first confidence level result and the first uncertainty result of each of the valid sample data; Based on the first confidence result of each of the valid sample data, sample data that meets the preset low confidence condition is determined from all the valid sample data and is used as low-quality sample data; and / or, based on the first uncertainty result of each of the valid sample data, sample data that meets the preset high uncertainty condition is determined from all the valid sample data and is used as low-quality sample data. Based on all the aforementioned low-quality sample data, a review candidate set is determined; Based on the first confidence result of all the valid sample data, determine the confidence variance result for each preset labeling category, and determine the quality warning labeling category based on the confidence variance result and sample size of each preset labeling category; Based on all the valid sample data and their first confidence level results and first uncertainty results, global quality index information is determined; The quality assessment result is determined based on the review candidate set, the quality warning label category, and the global quality indicator information.

8. A data annotation device, characterized in that, The device includes: The basic information determination module is used to determine the set of data to be labeled and its semantic embedding vector results that need to be labeled, wherein the set of data to be labeled includes at least one sample data to be labeled. The candidate pseudo-label determination module is used to perform sample value analysis and sample screening operations based on the dataset to be labeled and the semantic embedding vector results to obtain a list of candidate pseudo-labels. The pseudo-label determination module is used to perform a pseudo-label determination operation based on the candidate pseudo-label list to obtain a pseudo-label set; The effective sample determination module is used to perform multi-view processing and special sample processing operations based on the pseudo-label set to obtain the effective sample data set; The quality assessment module is used to perform multi-level annotation quality assessment operations based on the effective sample data set to obtain quality assessment results; The final annotation result determination module is used to determine the final data annotation result based on the quality assessment result.

9. A data annotation device, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute a data annotation method as described in any one of claims 1-7.

10. A computer storage medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked, are used to execute a data annotation method as described in any one of claims 1-7.

Images