A method and system for labeling data sets of AI model service platform

Abstract

The present invention discloses a data set annotation method and system for an AI model service platform, which relates to the technical field of data set annotation. The data set to be annotated is split according to the data type, and it is identified whether there is an existing annotation model in the AI ​​model service platform that can process the currently split data set to be annotated respectively; the annotation performance of the annotation model is evaluated based on the evaluation index of the annotation model, and then the annotation model is updated based on data enhancement; the split data set to be annotated is annotated respectively using the updated annotation model to generate a first annotation result; the first annotation result is annotated again based on multi-model joint annotation to generate a second annotation result; unqualified annotated data sets in the second annotation result are identified through confidence interval screening, and manual annotation is triggered. An efficient, low-cost, high-quality, flexible and scalable data set annotation process is realized, thereby significantly improving the annotation capability and service quality of the AI ​​model service platform.

Description

Technical Field

[0001] The present invention relates to the technical field of data set annotation, and specifically to a data set annotation method and system for an AI model service platform. Background Art

[0002] Artificial intelligence models, especially supervised learning models, rely on a large amount of labeled data during the training process. The quality of these data sets directly affects the learning effect and application accuracy of the models. Therefore, how to efficiently and accurately label data sets is an important issue in the field of AI research and application. In addition, different types of AI models (such as image recognition, natural language processing, speech recognition, etc.) require different types of data labeling, making it a challenge to build a platform and system that can cope with various data labeling needs.

[0003] Many existing annotation methods use a single model to process all types of data, often ignoring the differences in data types. Different types of data (such as text, images, voice, etc.) have different characteristics and requirements. A model may perform better on some types of data and worse on other types. Traditional annotation methods do not effectively split and process data, which can easily lead to reduced annotation accuracy; and traditional annotation methods rely on fixed annotation models without an effective mechanism to update and optimize these models. As the amount of data increases and the distribution of data changes, the original model may no longer be applicable, resulting in a gradual decline in the annotation effect; and although automated annotation can reduce labor costs, existing methods often do not have an effective quality control mechanism, resulting in the use of manual comprehensive modifications when the annotation results do not meet actual needs. This manual intervention is not only inefficient but also costly.

[0004] Therefore, in response to the above problems, there is an urgent need for an AI model service platform data set annotation method and system. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention provides a method and system for labeling data sets for an AI model service platform, which solves the problems of low efficiency, high cost, uneven quality and difficulty in adapting to diverse data types in the data set labeling process.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a method for labeling a data set of an AI model service platform, comprising the following steps: receiving a data set to be labeled, splitting the data set to be labeled according to data types, and identifying whether there is an existing labeling model in the AI ​​model service platform that can process the currently split data sets to be labeled respectively; evaluating the labeling performance of the labeling model based on the evaluation indicators of the labeling model, and then updating the labeling model based on data enhancement; using the updated labeling model to label the split data sets to be labeled respectively, and generating a first labeling result; re-labeling the first labeling result based on multi-model joint labeling to generate a second labeling result; identifying unqualified labeling data sets in the second labeling result through confidence interval screening, and then triggering manual labeling.

[0007] Furthermore, the specific analysis of identifying whether there is an existing annotation model in the AI ​​model service platform that can process the currently split data set to be annotated is as follows: obtaining the data set to be annotated, and then preprocessing the data set to be annotated, the preprocessing including denoising, missing value processing and data standardization; splitting the annotated data set into multiple types of data sets according to the data type, the data type including text, voice, picture and video, and the multiple types of data sets including text data subset, voice data subset, picture data subset and video data subset; extracting the features of the multiple types of data sets respectively, the features of the multiple types of data sets including data format and data distribution characteristics, the specific data formats including JSON, CSV, MP4, JPG, and the specific data distribution characteristics including text length distribution, voice duration, picture resolution and video duration; and then generating feature vectors of different types of data sets according to the features of different types of data sets; retrieving the list of existing annotation models in the AI ​​model service platform, and then identifying the appropriateness of each annotation model. The scope of application includes the annotation data types supported by the annotation model and the data format requirements for the annotation model input; the feature vector of each annotation model is generated according to the applicable scope of each annotation model; the feature similarity between different types of data sets and each annotation model is obtained according to the feature vector of the data set to be annotated and the feature vectors of different types of data sets; the feature similarity between different types of data sets and each annotation model is compared with the similarity threshold respectively. If the feature similarity with each annotation model is lower than the similarity threshold, it is marked that there is no existing annotation model in the AI ​​model service platform that can process this type of data set, and then this type of data set is randomly divided into a training set and an automated annotation data set, and the manual annotation demand is triggered to obtain the results of the manually annotated training set, and the annotation model of the automated annotation data set is trained by the results of the manually annotated training set for automated annotation; if there is a feature similarity with the annotation model that exceeds the similarity threshold, the annotation model with the largest feature similarity is marked as the annotation model for this type of data set.

[0008] Furthermore, the specific analysis of evaluating the labeling performance of the labeling model based on the evaluation indicators of the labeling model and then updating the labeling model based on data enhancement is as follows: obtaining the evaluation indicators of the labeling model, and then obtaining the labeling performance score of the labeling model according to the evaluation indicators of the labeling model; comparing the labeling performance score of the labeling model with the labeling performance threshold, and when the labeling performance score of the labeling model is lower than the labeling performance threshold, triggering the labeling model update mechanism; obtaining the historical labeling record of the labeling model, wherein the historical labeling record is specifically the labeling accuracy of various types of data historically labeled by the labeling model, comparing the labeling accuracy of various types of data historically labeled by the labeling model with the labeling accuracy threshold, and marking the data types whose labeling accuracy is lower than the labeling accuracy threshold The method comprises the following steps: identifying the rare annotation categories of the annotation model; performing data category retrieval on the training data set of the annotation model, and screening out the training data of the rare annotation categories of the annotation model in the training data set; performing data enhancement on the training data of the rare annotation categories, and the specific data enhancement includes: for text data, expanding and enhancing through synonym replacement, text generation, and back translation; for image data and video data, generating new training images through selection, scaling, cropping, flipping, and color adjustment; for speech data, expanding and enhancing through audio editing, noise addition, and speed change; obtaining an enhanced data set based on data enhancement, adding the enhanced data set to the training data set of the annotation model to retrain the annotation model, and updating the annotation model.

[0009] Furthermore, evaluation indicators of the labeling model are obtained, and then the labeling performance score of the labeling model is obtained based on the evaluation indicators of the labeling model. Specifically, the evaluation indicators of the labeling model include labeling accuracy and labeling efficiency, and the labeling performance score of the labeling model is calculated based on the labeling accuracy and labeling efficiency.

[0010] Furthermore, the updated annotation models are used to annotate the split data sets to be annotated respectively, and the updated annotation models in the first annotation result generated include the annotation model of the text data subset, the annotation model of the voice data subset, the annotation model of the picture data subset and the annotation model of the video data subset. The first annotation result includes the result of annotating the text data subset based on the annotation model of the text data subset, the result of annotating the voice data subset based on the annotation model of the voice data subset, the result of annotating the picture data subset based on the annotation model of the picture data subset, and the result of annotating the video data subset based on the annotation model of the video data subset.

[0011] Furthermore, the first annotation result is annotated again based on multi-model joint annotation, and the specific analysis of generating the second annotation result is: the first annotation result is input as an additional feature into the updated annotation model to further obtain an enhanced annotation model, and the enhanced annotation model includes an enhanced annotation model for a text data subset, an enhanced annotation model for a voice data subset, an enhanced annotation model for a picture data subset, and an enhanced annotation model for a video data subset; based on the enhanced annotation model, the text data subset, the voice data subset, the picture data subset, and the video data subset are annotated respectively to obtain enhanced text data subset annotation results, enhanced voice data subset annotation results, enhanced picture data subset annotation results, and enhanced video data subset annotation results; the average of the enhanced text data subset annotation results, enhanced voice data subset annotation results, enhanced picture data subset annotation results, and enhanced video data subset annotation results and the corresponding first annotation results are taken as the second annotation result for output.

[0012] Furthermore, the unqualified labeled data sets in the second labeling result are identified through confidence interval screening, and the specific analysis of triggering manual labeling is as follows: obtaining the labeling integrity confidence interval based on the historical data set labeling records of the AI ​​model service platform; obtaining the labeling integrity of each labeled data set in the second labeling result, and comparing the labeling integrity of each labeled data set with the labeling integrity confidence interval; when the labeling integrity exceeds the labeling integrity confidence interval, marking the second labeling result of the labeled data set as unqualified, triggering the manual labeling requirement; when the labeling integrity is within the labeling integrity confidence interval, marking the second labeling result of the labeled data set as qualified, and outputting and recording the labeling result of the labeled data set.

[0013] Furthermore, the specific analysis of obtaining the annotation completeness confidence interval based on the annotation records of the historical data set of the AI ​​model service platform is as follows: obtaining the annotation completeness of each annotation based on the annotation records of the historical data set of the AI ​​model service platform, and integrating the annotation completeness of each annotation to obtain the annotation completeness set; obtaining the mean and standard deviation of the annotation completeness set, and obtaining the confidence level based on the annotation requirements; combining the mean, standard deviation and confidence level of the annotation completeness set to output the upper and lower limits of the annotation completeness confidence interval.

[0014] An AI model service platform data set annotation system applies the above-mentioned AI model service platform data set annotation method, including: an annotation model identification module, used to receive a data set to be annotated, split the data set to be annotated according to data type, and identify whether there is an existing annotation model in the AI ​​model service platform that can process the currently split data set to be annotated respectively; an annotation model update module, used to evaluate the annotation performance of the annotation model based on the evaluation index of the annotation model, and then update the annotation model based on data enhancement; a first annotation result output module, used to use the updated annotation model to annotate the split data set to be annotated respectively, and generate a first annotation result; a second annotation result output module, used to perform secondary annotation on the first annotation result based on multi-model joint annotation, and generate a second annotation result; a manual annotation module, used to identify unqualified annotation data sets in the second annotation result through confidence interval screening, and then trigger manual annotation.

[0015] The present invention has the following beneficial effects:

[0016] The AI ​​model service platform dataset annotation method and system can ensure that each data subset is annotated by the most appropriate model by splitting the dataset to be annotated and applying the existing annotation model to process it, thereby avoiding the deviation that may be caused when a single model processes all data and improving the accuracy and quality of the overall annotation. The annotation model is evaluated using evaluation indicators and updated through data enhancement, so that the annotation model can self-optimize according to actual performance. The application of data enhancement can help the model adapt to more diverse annotation tasks in different scenarios, improve its ability to process complex data, and avoid overfitting of the model on specific datasets. Through multi-model joint annotation, the potential erroneous annotation of a single model can be reduced. For example, different models may have different interpretations of the same data sample, and by integrating the annotations of multiple models As a result, the credibility of the final labeling results can be effectively improved, and the labeled data set can be re-labeled, thereby increasing the robustness of the results; unqualified labeled data sets are identified through confidence interval screening and manual labeling is triggered. This process realizes the combination of manual and automation. The intervention of manual labeling is mainly for the parts that the model cannot accurately label, while the majority of easy-to-label parts are handled by the automated model, thereby reducing the workload of manual labeling and improving efficiency; the data set labeling has a high degree of automation, and can complete the labeling task through the existing labeling model in most cases. Manual intervention is limited to situations where the model cannot judge or lacks confidence, making the overall labeling process more efficient and reducing the cost and time of manual labeling. It not only improves the intelligence level of the AI ​​model service platform, but also effectively optimizes the data set labeling process.

[0017] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a method for labeling a data set of an AI model service platform according to the present invention.

[0019] Figure 2 This is a structural diagram of a data set annotation system for an AI model service platform of the present invention. DETAILED DESCRIPTION

[0020] The embodiment of the present application realizes an efficient, low-cost, high-quality, and flexible and scalable dataset annotation process through a method and system for an AI model service platform dataset annotation, thereby significantly improving the annotation capability and service quality of the AI ​​model service platform.

[0021] The overall idea of ​​the embodiments of this application is:

[0022] The quality and efficiency of dataset annotation are improved through automated and multi-stage annotation. At the same time, artificial intelligence models and manual review are combined to work together to ensure the accuracy of annotation results and adapt to complex and diverse dataset requirements.

[0023] See also Figure 1 , an embodiment of the present invention provides a technical solution: a method for labeling a data set of an AI model service platform, comprising the following steps: receiving a data set to be labeled, splitting the data set to be labeled according to data types, and identifying whether there is an existing labeling model in the AI ​​model service platform that can process the currently split data sets to be labeled respectively; evaluating the labeling performance of the labeling model based on the evaluation index of the labeling model, and then updating the labeling model based on data enhancement; using the updated labeling model to label the split data sets to be labeled respectively, and generating a first labeling result; re-labeling the first labeling result based on multi-model joint labeling to generate a second labeling result; identifying unqualified labeling data sets in the second labeling result through confidence interval screening, and then triggering manual labeling.

[0024] Specifically, the specific analysis for identifying whether there is an existing annotation model in the AI ​​model service platform that can process the currently split data set to be annotated is as follows: obtain the data set to be annotated, and then preprocess the data set to be annotated, the preprocessing includes denoising, missing value processing and data standardization; split the annotated data set into multiple types of data sets according to the data type, the data type includes text, voice, picture and video, and the multiple types of data sets include text data subsets, voice data subsets, picture data subsets and video data subsets; extract the features of the multiple types of data sets respectively, the features of the multiple types of data sets include data format and data distribution characteristics, the specific data formats include JSON, CSV, MP4, JPG, and the specific data distribution characteristics include text length distribution, voice duration, picture resolution and video duration; and then generate feature vectors of different types of data sets based on the features of different types of data sets; retrieve the list of existing annotation models in the AI ​​model service platform, and then identify the scope of application of each annotation model. , the scope of application includes the annotation data types supported by the annotation model and the data format requirements for the annotation model input; the feature vector of each annotation model is generated according to the applicable scope of each annotation model; the feature similarity between different types of datasets and each annotation model is obtained according to the feature vector of the dataset to be annotated and the feature vector of different types of datasets; the feature similarity between different types of datasets and each annotation model is compared with the similarity threshold respectively. If the feature similarity with each annotation model is lower than the similarity threshold, it is marked that there is no existing annotation model in the AI ​​model service platform that can process this type of dataset, and then this type of dataset is randomly divided into a training set and an automatic annotation dataset, and the manual annotation demand is triggered, and the manual annotation training set results are obtained. The annotation model of the automatic annotation dataset is trained by the manual annotation training set results for automatic annotation; if there is a feature similarity with the annotation model that exceeds the similarity threshold, the annotation model with the largest feature similarity is marked as the annotation model for this type of dataset.

[0025] In this implementation scheme, the feature vector is a multidimensional numerical representation generated by a feature extraction algorithm and is used to describe the key features of the data. The specific generation method is: various features (such as text length, resolution, duration) are converted into feature vectors using standard feature extraction methods, such as TF-IDF (text), MFCC (speech), HOG (picture), etc.

[0026] Calculate the similarity between the feature vector of the dataset to be labeled and the feature vector of each labeled model. Commonly used methods include cosine similarity and Euclidean distance. The specific cosine similarity calculation example is as follows: , where Represents the cosine similarity between the feature vector of the dataset to be labeled and the feature vector of the labeled model, represents the feature vector of the dataset to be labeled, represents the feature vector of the annotation model, Represents the modulus length of the feature vector of the dataset to be labeled and the feature vector of the labeled model, Indicates the similarity between the feature vector of the dataset to be labeled and the feature vector of the labeled model.

[0027] The data set to be annotated is obtained through data collection or user upload; the list of annotation models is obtained from the AI ​​model service platform through API query or regular synchronization; the similarity threshold is used to determine whether two feature vectors are similar enough to decide whether to select the annotation model for annotation, which is obtained by the data science team through experiments or an empirical value is set in the system; the annotation model of the automatically annotated data set refers to the annotation model trained by the results of the manually annotated training set, which is used for subsequent automatic annotation.

[0028] The specific steps for generating the annotation model of the automatically annotated dataset are as follows: annotate the dataset by manual annotators and upload it to generate a manually annotated training set; use the manually annotated dataset to train the machine learning model, which can be a deep learning model (such as CNN, RNN, etc.) or a traditional machine learning model (such as SVM, decision tree, etc.); the trained annotation model is used to automatically annotate the dataset to generate automatic annotation results.

[0029] Intelligently evaluate the characteristics of the data set and guide human participation when necessary to ensure the quality of annotation while improving efficiency. Automated annotation can handle most data, while manual annotation is suitable for situations that AI models cannot handle or the similarity is insufficient. Through feature vector and similarity calculation, the most suitable annotation model can be selected for different types of data sets, avoiding the wrong model selection and inefficient use of resources. It supports the annotation of multiple data types such as text, voice, picture, video, etc., and has strong versatility. The selection of the annotation model is based on the feature vector of the data set, so that the selected annotation model can match the data more accurately.

[0030] Specifically, the specific analysis of evaluating the labeling performance of the labeling model based on the evaluation indicators of the labeling model and then updating the labeling model based on data enhancement is as follows: obtaining the evaluation indicators of the labeling model, and then obtaining the labeling performance score of the labeling model based on the evaluation indicators of the labeling model; comparing the labeling performance score of the labeling model with the labeling performance threshold, and when the labeling performance score of the labeling model is lower than the labeling performance threshold, triggering the labeling model update mechanism; obtaining the historical labeling records of the labeling model, the historical labeling records are specifically the labeling accuracy of various types of data historically labeled by the labeling model, comparing the labeling accuracy of various types of data historically labeled by the labeling model with the labeling accuracy threshold, and marking the data categories whose labeling accuracy is lower than the labeling accuracy threshold. as the rare annotation category of the annotation model; perform data category retrieval on the training data set of the annotation model, and screen out the training data of the rare annotation category of the annotation model in the training data set; perform data enhancement on the training data of the rare annotation category, and the specific data enhancement includes: for text data, expansion and enhancement through synonym replacement, text generation, and back translation; for image data and video data, generate new training images through selection, scaling, cropping, flipping and color adjustment; for voice data, expansion and enhancement through audio editing, noise addition, and speed change; based on data enhancement, obtain an enhanced data set, add the enhanced data set to the training data set of the annotation model to retrain the annotation model, and update the annotation model.

[0031] In this implementation, evaluation indicators of the labeling model are obtained, and then the labeling performance score of the labeling model is obtained based on the evaluation indicators of the labeling model. Specifically, the evaluation indicators of the labeling model include labeling accuracy and labeling efficiency, and the labeling performance score of the labeling model is calculated based on the labeling accuracy and labeling efficiency.

[0032] The labeling performance score is a score calculated by calculating the labeling accuracy and labeling efficiency of the model, which reflects the overall performance of the model under a specific task. The labeling accuracy refers to the proportion of correct labeling by the model. The labeling efficiency refers to the speed at which the model labels or the time required for each labeling, which is usually related to factors such as system processing power and resource optimization. The labeling performance threshold is a preset standard used to determine whether the model needs to be updated. When the labeling performance score of the model is lower than the threshold, the update mechanism is triggered. It can be set according to historical data, industry standards or experiments. The labeling accuracy threshold refers to the minimum requirement for the labeling accuracy of the model. Data categories below this threshold will be marked as rare categories for special processing or data enhancement. Rare labeling categories refer to data categories with labeling accuracy lower than the labeling accuracy threshold in historical labeling records. The model performance of these categories is poor, which may be due to insufficient data or the model’s failure to effectively learn the characteristics of these categories.

[0033] The accuracy of the labeling model evaluation index is obtained by calculating the proportion of correctly labeled samples, and the efficiency is measured by the completion time or processing speed of each task; the specific expression for obtaining the labeling performance score is as follows: , where represents the annotation performance score, represents the labeling accuracy, It represents the labeling efficiency. The labeling performance threshold is set through historical data analysis or experiments. It is the performance standard that the labeling model should achieve under normal circumstances. It can also be obtained through cross-validation or testing on different data sets. The labeling accuracy threshold is set through historical data analysis or industry practice. It defines which categories have an accuracy rate lower than a certain standard. It is usually set according to the actual performance of the model. For example, a labeling accuracy rate lower than 90% can be regarded as unsatisfactory performance.

[0034] Specific data enhancement examples are:

[0035] Data enhancement for text data: Synonym replacement: replace certain words in the text with words with similar meanings through a synonym library to increase the diversity of the text, for example, the original text "He walked into the room" can be changed to "He entered the room"; Text generation: Use a pre-trained generative model (such as GPT) to generate new sentences or paragraphs based on the original text to expand the training data, for example, generate more relevant variants from "I like to eat apples", such as "I love to eat apples" and "I often eat apples"; Translation and back translation: translate the text into another language, and then translate it back to the original language to obtain changes in sentence structure and vocabulary, for example, the original text "Today's weather is nice" is first translated into English "Today's weather is nice", and then translated back into Chinese as "Today's weather is good".

[0036] Perform data enhancement on image data: Rotation: rotate the image by a certain angle to increase the diversity of the image; Scaling: scale the image to make it larger or smaller; Crop: randomly crop part of the image, retaining the central area or a random area; Flip: flip the image horizontally or vertically; Color adjustment: change the brightness, contrast or saturation of the image to make the image color different.

[0037] Perform data enhancement on speech data: Audio clipping: Cut different segments from the original audio to generate multiple different audio samples; Noise addition: Add background noise (such as white noise) to the audio signal so that the model can be trained in a noisy environment; Speed ​​change: Generate new audio samples by changing the audio playback speed.

[0038] By evaluating the performance of the annotation model, the update mechanism is automatically triggered to ensure that the model can adjust itself according to changes in annotation accuracy and efficiency, thereby improving long-term performance; the data enhancement process specifically solves the model's problems in labeling rare categories, especially categories with poor annotation performance, thereby avoiding overfitting or annotation bias; through data enhancement and model updates, especially in rare categories, the model can learn more variants and improve overall accuracy; through automated evaluation, data enhancement and training updates, the need for manual intervention is reduced, thereby reducing costs and improving efficiency.

[0039] Specifically, the updated annotation models are used to annotate the split data sets to be annotated respectively, and the updated annotation models in the first annotation result generated include the annotation model of the text data subset, the annotation model of the voice data subset, the annotation model of the picture data subset and the annotation model of the video data subset. The first annotation result includes the result of annotating the text data subset based on the annotation model of the text data subset, the result of annotating the voice data subset based on the annotation model of the voice data subset, the result of annotating the picture data subset based on the annotation model of the picture data subset, and the result of annotating the video data subset based on the annotation model of the video data subset.

[0040] In this implementation scheme, different types of data (text, voice, picture, video) have significant differences in features and processing methods. If all data are uniformly submitted to a general model for labeling, the characteristics of each data type may not be fully utilized, resulting in unsatisfactory labeling results. By designing a special labeling model for each data type, targeted optimization can be performed according to the particularity of each data, thereby improving the labeling accuracy. For text data, it may involve natural language processing (NLP) tasks, such as entity recognition, sentiment analysis, etc., voice data may involve voice recognition and sentiment classification, picture data may involve image recognition or target detection, and video data may need to process image and voice labeling tasks at the same time. Each data type has its own specific labeling model, which can It can make more accurate predictions and annotations based on different features; different types of data may require different processing procedures or model architectures when annotating. By splitting data subsets and adopting specialized models, the processing speed and efficiency can be improved. For example, a speech annotation model may use audio features for rapid recognition, while a text annotation model may use deep learning technology to process language data. The advantages of the models can be fully utilized in their respective areas of expertise to avoid unnecessary waste of computing resources. At the same time, as the data type and annotation requirements change, the annotation model of a certain data subset can be easily expanded or adjusted. For example, if there is new progress in the processing technology of speech data, the speech annotation model can be updated without affecting the annotation of other types of data, which helps to flexibly respond to new technical challenges and requirements.

[0041] Specifically, the first annotation result is annotated again based on multi-model joint annotation, and the specific analysis of generating the second annotation result is: the first annotation result is input as an additional feature into the updated annotation model to further obtain an enhanced annotation model, and the enhanced annotation model includes an enhanced annotation model for a text data subset, an enhanced annotation model for a voice data subset, an enhanced annotation model for a picture data subset, and an enhanced annotation model for a video data subset; based on the enhanced annotation model, the text data subset, the voice data subset, the picture data subset, and the video data subset are annotated respectively to obtain enhanced text data subset annotation results, enhanced voice data subset annotation results, enhanced picture data subset annotation results, and enhanced video data subset annotation results; the average of the enhanced text data subset annotation results, enhanced voice data subset annotation results, enhanced picture data subset annotation results, and enhanced video data subset annotation results and the corresponding first annotation results are taken as the second annotation result for output.

[0042] In this implementation scheme, the enhanced annotation model is a model enhanced by the first annotation result, and the first annotation result is used as an additional feature input and combined with multimodal data for annotation training; the enhanced annotation result represents the annotation result independently trained for each subset (text, voice, picture, video), and the enhanced annotation output is obtained by annotating the corresponding data subset through the enhanced annotation model; the second annotation result is the final annotation result generated based on the average of the enhanced annotation result and the first annotation result.

[0043] In addition to inputting the first annotation result as an additional feature into the updated annotation model, you can also perform secondary annotation in the following ways:

[0044] Model self-supervised learning: Through self-supervised learning methods, the model automatically corrects and enhances the first annotation result. For example, it uses unlabeled data for pre-training, and then fine-tunes the model based on the first annotation result. Ensemble learning: Through multiple annotation models, the output results of different models are integrated (for example, using weighted average, voting and other strategies) as the second annotation result output. The annotation result can be further optimized by introducing different types of models (such as deep learning models, traditional machine learning models, etc.); Error correction model: Design a special error correction module to repair potential errors in the first annotation result. For example, rules or other training models can be used to identify and correct errors based on the first annotation result. Annotation adjustment based on contrastive learning: Through contrastive learning, the first annotation result is compared with the true label (such as manual annotation) to optimize the model's understanding of the annotation task and generate a more accurate second annotation result.

[0045] By inputting the first annotation result as an additional feature into the updated annotation model, the training data of the model is enhanced and the annotation accuracy is improved. Through multi-model joint annotation, the annotation results of different subsets can complement each other and reduce the annotation error of a single model; by training models separately with different data subsets (text, voice, picture, video) to form independent enhanced annotation models, the characteristics of different data types can be effectively coped with and the robustness and adaptability of the overall annotation system can be improved; by combining multimodal data of text, voice, picture and video, different types of data can be comprehensively annotated, so that different data sources can complement each other and jointly improve the accuracy of the annotation results; by annotating multiple data subsets separately and calculating the mean of the enhanced annotation result of each subset and the first annotation result, the annotation results of different data types can be made consistent, thereby ensuring the consistency and reliability of the annotation.

[0046] Specifically, the unqualified labeled data sets in the second labeling result are identified through confidence interval screening, and the specific analysis of triggering manual labeling is as follows: obtaining the labeling integrity confidence interval based on the historical data set labeling records of the AI ​​model service platform; obtaining the labeling integrity of each labeled data set in the second labeling result, and comparing the labeling integrity of each labeled data set with the labeling integrity confidence interval; when the labeling integrity exceeds the labeling integrity confidence interval, marking the second labeling result of the labeled data set as unqualified, triggering the manual labeling requirement; when the labeling integrity is within the labeling integrity confidence interval, marking the second labeling result of the labeled data set as qualified, and outputting and recording the labeling result of the labeled data set.

[0047] In this implementation scheme, the specific analysis of obtaining the annotation completeness confidence interval based on the annotation records of the historical data set of the AI ​​model service platform is as follows: obtaining the annotation completeness of each annotation based on the annotation records of the historical data set of the AI ​​model service platform, and integrating the annotation completeness of each annotation to obtain the annotation completeness set; obtaining the mean and standard deviation of the annotation completeness set, and obtaining the confidence level based on the annotation requirements; combining the mean, standard deviation and confidence level of the annotation completeness set to output the upper and lower limits of the annotation completeness confidence interval.

[0048] Label completeness indicates the completeness level of each labeled data set, usually measured by a certain ratio or score; the label completeness confidence interval is based on the label completeness of the historical data set, indicating the credible range of label completeness, usually composed of a mean and a standard deviation, and is used to determine whether the label completeness of the second labeled data set is qualified; the mean of the label completeness set indicates the center position of the label completeness distribution of the historical data set; the standard deviation of the label completeness set is used to measure the degree of dispersion of the label completeness of the historical data set. The larger the standard deviation, the greater the difference in label completeness, and vice versa; the confidence level indicates the reliability of the specified confidence interval, usually with 95%, 99%, etc. as the standard. The higher the confidence level, the larger the interval range; the upper and lower limits of label completeness indicate the upper and lower limits of the label completeness confidence interval calculated based on the confidence level, which are used to compare with the label completeness of the second labeled data set.

[0049] According to the mean, standard deviation and confidence level, the upper and lower limits of the confidence interval are calculated using the normal distribution. The upper and lower limits of the integrity confidence interval are specifically marked and obtained by the expression:

[0050] , ; In the formula represents the lower limit of the confidence interval for annotation completeness, represents the upper limit of the confidence interval for annotation completeness, represents the mean of the annotation completeness set, Represents the Z value of the standard normal distribution, which is determined by the confidence level. For example, when the confidence level is 95%, , Indicates the number of samples, specifically the number of annotations in the historical dataset annotation records of the AI ​​model service platform. Indicates the standard deviation of the annotation completeness set.

[0051] By setting the confidence interval, unqualified annotated data sets can be automatically identified, avoiding manual participation in tedious screening work; for annotated data sets that meet the standards, no manual intervention is required and the results can be directly output, saving time and resources; by triggering manual annotation, when the annotation completeness exceeds the confidence interval, it can ensure that unqualified annotations are rechecked and corrected, thereby improving the overall data quality; adapt to different data sets and annotation tasks, and flexibly respond to different annotation needs by adjusting the calculation method of the confidence level and confidence interval.

[0052] See also Figure 2, an AI model service platform data set annotation system, applying the above-mentioned AI model service platform data set annotation method, including: an annotation model identification module, used to receive a data set to be annotated, split the data set to be annotated according to data type, and identify whether there is an existing annotation model in the AI ​​model service platform that can process the currently split data set to be annotated respectively; an annotation model update module, used to evaluate the annotation performance of the annotation model based on the evaluation index of the annotation model, and then update the annotation model based on data enhancement; a first annotation result output module, used to use the updated annotation model to annotate the split data set to be annotated respectively, and generate a first annotation result; a second annotation result output module, used to perform secondary annotation on the first annotation result based on multi-model joint annotation, and generate a second annotation result; a manual annotation module, used to identify unqualified annotation data sets in the second annotation result through confidence interval screening, and then trigger manual annotation.

[0053] In summary, this application has at least the following effects:

[0054] By automatically identifying and processing data sets that can be processed by existing annotation models, the need for manual intervention is reduced, thereby improving the efficiency of annotation. Using data enhancement technology to update the annotation model can further improve the annotation ability of the model and ensure the accuracy of the annotation results. Secondary processing of the preliminary annotation results through multi-model joint annotation can further reduce annotation errors and improve the reliability of the annotation results. Through automated annotation and multi-model joint annotation, the amount of data that requires manual annotation is significantly reduced, thereby reducing the annotation cost. The automated processing flow and multi-model collaboration enable the annotation work to be completed more efficiently, further reducing the cost. Through standardized annotation processes and quality control mechanisms, the consistency of annotation results can be ensured, avoiding the problem of inconsistent annotation caused by differences in understanding between different annotators. By identifying unqualified annotation data sets through confidence interval screening, annotation errors can be discovered and corrected in a timely manner, further improving the consistency of annotation quality. The dataset to be annotated is split according to the data type, and the applicable annotation model is automatically identified. Therefore, it has high flexibility and can adapt to different types of datasets, reducing the opportunity for manual direct contact with sensitive data, thereby reducing the risk of data leakage.

[0055] It should be understood by those skilled in the art that the embodiments of the present invention may be provided as methods and systems. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0056] The present invention is described with reference to the flowcharts and structure diagrams of the methods and systems according to the embodiments of the present invention. It should be understood that each process and combination of modules in the flowcharts and structure diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowcharts. Figure 1 process or multiple processes and structures Figure 1 A device that specifies functionality in a module or modules.

[0057] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 process or multiple processes and structures Figure 1 A function specified in a module or multiple modules.

[0058] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 process or multiple processes and structures Figure 1 Steps to specify functionality in a module or multiple modules.

[0059] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0060] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for labeling a data set of an AI model service platform, characterized in that: The following steps are involved: Receive the data set to be labeled, split the data set to be labeled according to data type, and identify whether there is an existing labeling model in the AI ​​model service platform that can process the currently split data set to be labeled respectively; Evaluate the annotation performance of the annotation model based on the evaluation index of the annotation model, and then update the annotation model based on data enhancement; Using the updated annotation model to respectively annotate the split data sets to be annotated, and generate a first annotation result; Performing secondary annotation on the first annotation result based on multi-model joint annotation to generate a second annotation result; Identify unqualified annotated data sets in the second annotation result through confidence interval screening, thereby triggering manual annotation; The specific analysis of evaluating the annotation performance of the annotation model based on the evaluation indicators of the annotation model and then updating the annotation model based on data enhancement is as follows: Obtaining evaluation indicators of the annotation model, and then obtaining an annotation performance score of the annotation model based on the evaluation indicators of the annotation model; Compare the annotation performance score of the annotation model with the annotation performance threshold. When the annotation performance score of the annotation model is lower than the annotation performance threshold, trigger the annotation model update mechanism. Obtaining historical annotation records of the annotation model, wherein the historical annotation records are specifically the annotation accuracy of various types of data historically annotated by the annotation model, comparing the annotation accuracy of various types of data historically annotated by the annotation model with an annotation accuracy threshold, and marking data categories with an annotation accuracy lower than the annotation accuracy threshold as rare annotation categories of the annotation model; Perform data category retrieval on the training data set of the annotation model, and filter out the training data of the rare annotation categories of the annotation model in the training data set; Data enhancement is performed on training data of rare annotated categories. Specific data enhancement includes: for text data, expansion and enhancement through synonym replacement, text generation, and translation back translation; Generate new training images for image and video data by selecting, scaling, cropping, flipping, and color adjustment; For voice data, it is expanded and enhanced through audio editing, noise addition, and speed change; An enhanced data set is obtained based on data enhancement, and the enhanced data set is added to the training data set of the annotation model to retrain the annotation model and update the annotation model; Obtaining the evaluation index of the labeling model, and then obtaining the labeling performance score of the labeling model according to the evaluation index of the labeling model. Specifically, the evaluation index of the labeling model includes the labeling accuracy and the labeling efficiency, and the labeling performance score of the labeling model is calculated according to the labeling accuracy and the labeling efficiency; The first annotation result is annotated again based on the multi-model joint annotation, and the specific analysis of generating the second annotation result is as follows: Inputting the first annotation result as an additional feature into the updated annotation model, further obtaining an enhanced annotation model, wherein the enhanced annotation model includes an enhanced annotation model for a text data subset, an enhanced annotation model for a voice data subset, an enhanced annotation model for a picture data subset, and an enhanced annotation model for a video data subset; Based on the enhanced annotation model, the text data subset, the voice data subset, the image data subset and the video data subset are respectively annotated to obtain an enhanced text data subset annotation result, an enhanced voice data subset annotation result, an enhanced image data subset annotation result and an enhanced video data subset annotation result; The enhanced text data subset annotation results, the enhanced voice data subset annotation results, the enhanced picture data subset annotation results, and the enhanced video data subset annotation results are respectively taken as the average of the corresponding first annotation results for output.

2. According to claim 1, a method for labeling a data set of an AI model service platform is characterized in that: The specific analysis of identifying whether there is an existing annotation model in the AI ​​model service platform that can process the currently split data set to be annotated is as follows: Obtain the data set to be labeled, and then preprocess the data set to be labeled. The preprocessing includes denoising, missing value processing and data standardization. The labeled data set is split into multiple data sets of different types according to the data type, the data type includes text, voice, picture and video, and the multiple data sets of different types include a text data subset, a voice data subset, a picture data subset and a video data subset; Extracting features of multiple types of data sets respectively, wherein the features of the multiple types of data sets include data formats and data distribution characteristics, specific data formats include JSON, CSV, MP4, and JPG, and specific data distribution characteristics include text length distribution, voice duration, image resolution, and video duration; Then, feature vectors of different types of data sets are generated according to the features of different types of data sets; Retrieve the list of existing annotation models in the AI ​​model service platform, and then identify the applicable scope of each annotation model, including the annotation data types supported by the annotation model and the data format requirements for the annotation model input; Generate a feature vector for each annotation model according to the applicable scope of each annotation model; According to the feature vector of the dataset to be labeled and the feature vectors of different types of datasets, the feature similarity between different types of datasets and each annotation model is obtained respectively; The feature similarity of different types of data sets with each annotation model is compared with the similarity threshold. If the feature similarity with each annotation model is lower than the similarity threshold, it is marked that there is no existing annotation model in the AI ​​model service platform that can process this type of data set. Then, this type of data set is randomly divided into a training set and an automatic annotation data set, and the manual annotation demand is triggered to obtain the results of the manual annotation training set. The annotation model of the automatic annotation data set is obtained by training the results of the manual annotation training set for automatic annotation. If there is a labeling model whose feature similarity exceeds the similarity threshold, the labeling model with the largest feature similarity is marked as the labeling model for this type of data set.

3. The AI ​​model service platform data set annotation method according to claim 1, characterized in that: The updated annotation models are used to respectively annotate the split data sets to be annotated, and the updated annotation models in the generated first annotation result include the annotation model of the text data subset, the annotation model of the voice data subset, the annotation model of the picture data subset and the annotation model of the video data subset. The first annotation result includes the result of annotating the text data subset based on the annotation model of the text data subset, the result of annotating the voice data subset based on the annotation model of the voice data subset, the result of annotating the picture data subset based on the annotation model of the picture data subset, and the result of annotating the video data subset based on the annotation model of the video data subset.

4. The AI ​​model service platform data set annotation method according to claim 1, characterized in that: The specific analysis of identifying unqualified labeled data sets in the second labeling result through confidence interval screening and triggering manual labeling is as follows: Obtain annotation completeness confidence intervals based on the annotation records of the historical data set of the AI ​​model service platform; Obtaining the annotation completeness of each annotated data set in the second annotation result, comparing the annotation completeness of each annotated data set with the annotation completeness confidence interval, and when the annotation completeness exceeds the annotation completeness confidence interval, marking the second annotation result of the annotated data set as unqualified, triggering a manual annotation requirement; When the annotation completeness is within the annotation completeness confidence interval, the second annotation result of the annotation data set is marked as qualified, and the annotation result of the annotation data set is output and recorded.

5. The AI ​​model service platform data set annotation method according to claim 4, characterized in that: The specific analysis of obtaining the annotation completeness confidence interval based on the annotation records of the historical data set of the AI ​​model service platform is as follows: Based on the annotation records of the historical data set of the AI ​​model service platform, the annotation completeness of each annotation is obtained, and the annotation completeness set is obtained by integrating the annotation completeness of each annotation; Get the mean and standard deviation of the annotation completeness set, and get the confidence level based on the annotation requirements; Combine the mean, standard deviation, and confidence level of the annotation completeness set to output the upper and lower limits of the annotation completeness confidence interval.

6. An AI model service platform dataset annotation system, applying an AI model service platform dataset annotation method according to any one of claims 1 to 5, characterized in that: include: The annotation model identification module is used to receive the data set to be annotated, split the data set to be annotated according to the data type, and identify whether there is an existing annotation model in the AI ​​model service platform that can process the currently split data set to be annotated respectively; The annotation model update module is used to evaluate the annotation performance of the annotation model based on the evaluation indicators of the annotation model, and then update the annotation model based on data enhancement; A first annotation result output module is used to use the updated annotation model to respectively annotate the split to-be-annotated data sets to generate a first annotation result; A second annotation result output module, used to perform secondary annotation on the first annotation result based on multi-model joint annotation to generate a second annotation result; The manual labeling module is used to identify unqualified labeling data sets in the second labeling result through confidence interval screening, thereby triggering manual labeling.

Images