Data labeling method, electronic equipment, storage medium and program product

Abstract

The invention provides a data annotation method, electronic equipment, a storage medium and a program product. The data annotation method disclosed by the invention comprises the following steps: acquiring to-be-annotated data; performing multi-angle labeling on the to-be-labeled data based on the plurality of large language models to obtain a plurality of model labeling results; and performing optimization processing on the plurality of model labeling results to obtain a target labeling result of the to-be-labeled data.

Description

Technical Field

[0001] The present disclosure relates to technical fields such as data processing, and in particular to a data labeling method, electronic device, storage medium, and program product. Background Art

[0002] In today's era of digital information explosion, data has become the core driving force for innovation and development in many fields, especially in cutting-edge technology fields such as artificial intelligence (AI), machine learning (ML) and big data analysis, where high-quality data labeling plays a vital role.

[0003] With the rapid development of deep learning algorithms, the demand for massive and accurately annotated data in deep neural networks has shown an exponential growth trend. For example, in the field of image recognition, in order to train a model that can accurately identify various objects, scenes and people, it is necessary to annotate millions or even more images in detail, including information such as the category, location, shape and other information of the objects in the image; in natural language processing, whether it is text classification, sentiment analysis or machine translation tasks, it is inseparable from complex operations such as part-of-speech tagging, semantic tagging and discourse tagging of large-scale text data.

[0004] Traditional data labeling methods mainly rely on manual operations. In this mode, labelers need to analyze and label the data one by one.

[0005] However, manual data labeling consumes a lot of manpower and time costs, and the data labeling efficiency is low; and manual data labeling is easily affected by subjective factors, resulting in high inconsistency and error rate in the labeling results. Summary of the invention

[0006] The present disclosure provides a data labeling method, an electronic device, a storage medium, and a program product.

[0007] According to one aspect of the present disclosure, a data annotation method is provided, comprising: Obtain the data to be labeled; Annotating the data to be annotated from multiple angles based on multiple large language models to obtain multiple model annotation results; The multiple model labeling results are optimized to obtain target labeling results for the data to be labeled.

[0008] According to the data annotation method of at least one embodiment of the present disclosure, the step of optimizing the multiple model annotation results includes: Determining a target user of the data to be labeled from a candidate user group; receiving a processing instruction from the target user; The multiple model annotation results are optimized according to the processing instructions.

[0009] According to the data labeling method of at least one embodiment of the present disclosure, the step of determining a target user of the data to be labeled from a candidate user group includes: Randomly determine the target user of the data to be labeled from the candidate user group; or, The target user of the data to be labeled is determined from the candidate user group according to a matching dimension, wherein the matching dimension includes one or more of user characteristics, data characteristics of the data to be labeled, and task characteristics of a labeling task for the data to be labeled.

[0010] According to at least one embodiment of the data labeling method of the present disclosure, after obtaining the target labeling result of the data to be labeled, the method further includes: The multiple large language models are fine-tuned according to the target labeling results.

[0011] According to the data labeling method of at least one embodiment of the present disclosure, the step of obtaining the data to be labeled includes: Get the data to be labeled from the dataset; or, Obtain the data to be labeled from the business system.

[0012] According to at least one embodiment of the data labeling method of the present disclosure, the step of obtaining the data to be labeled from the business system includes: Acquire the data to be annotated from the business system based on manual instructions; or The data to be annotated is obtained from the business system based on one or more triggering factors including conditions, events and time.

[0013] According to at least one embodiment of the data labeling method of the present disclosure, the step of collecting the data to be labeled from the business system includes: Obtain business data from business systems; Acquiring characteristic information of the business data; Acquire a processing method of the business data according to the characteristic information; Processing the business data using the processing method to obtain data to be labeled; Optionally, the characteristic information includes: one or more of data type, data format, data quality, data range and system environment; and / or the processing method includes one or more of missing value filling, outlier correction, deduplication, standardization, conversion, data merging and splitting; Optionally, the obtaining of the processing mode of the business data according to the feature information comprises: inputting the feature information into a large language model to obtain the processing mode of the business data; or obtaining the processing mode of the business data according to the feature information based on a processing rule; Optionally, the multi-angle labeling of the data to be labeled based on multiple large language models includes: Get candidate label set; Based on multiple large language models, the data to be labeled is labeled from multiple angles according to the candidate label set; Optionally, the multi-angle annotation includes: Labeling based on at least two dimensions of data category dimension, data content dimension, data attribute dimension, situation dimension, domain dimension and anomaly dimension.

[0014] According to another aspect of the present disclosure, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, so that the processor executes the data labeling method of any embodiment of the present disclosure.

[0015] According to another aspect of the present disclosure, a readable storage medium is provided, wherein the readable storage medium stores execution instructions, and when the execution instructions are executed by a processor, the data labeling method of any embodiment of the present disclosure is implemented.

[0016] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the data labeling method of any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The accompanying drawings illustrate exemplary embodiments of the present disclosure and together with the description serve to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are incorporated in and constitute a part of this specification.

[0018] Figure 1 It is a schematic diagram of an application scenario of the data labeling method according to an embodiment of the present disclosure.

[0019] Figure 2 This is a process of a data annotation method provided by an embodiment of the present disclosure. Figure 1 .

[0020] Figure 3 yes Figure 2 The process of the annotation optimization method in the data annotation method shown Figure 1 .

[0021] Figure 4 yes Figure 2 The process of the annotation optimization method in the data annotation method shown Figure 2 .

[0022] Figure 5 yes Figure 2 The process of the annotation optimization method in the data annotation method shown Figure 3 .

[0023] Figure 6 This is a process of a data annotation method provided by an embodiment of the present disclosure. Figure 2 .

[0024] Figure 7 This is a process of a data annotation method provided by an embodiment of the present disclosure. Figure 3 .

[0025] Figure 8 This is a process of a data annotation method provided by an embodiment of the present disclosure. Figure 4 .

[0026] Fig. 9 This is a process of a data annotation method provided by an embodiment of the present disclosure. Figure 5 .

[0027] Fig.10 This is a process of a data annotation method provided by an embodiment of the present disclosure. Figure 6 .

[0028] Fig.11 yes Figure 8 The process of the data acquisition method in the data annotation method shown Figure 1 .

[0029] Fig.12 yes Figure 8 The process of the data acquisition method in the data annotation method shown Figure 2 .

[0030] Fig.13 yes Figure 8 The process of the data acquisition method in the data annotation method shown Figure 3 .

[0031] Fig.14 yes Figure 1 Flowchart of the model annotation method in the data annotation method shown.

[0032] Fig.15 It is a schematic flowchart of a data labeling method according to an embodiment of the present disclosure.

[0033] Fig.16 It is a schematic block diagram of the structure of a data labeling device according to another embodiment of the present invention.

[0034] Fig.17 It is a schematic block diagram of the structure of an electronic device according to another embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The present disclosure is further described in detail below in conjunction with the accompanying drawings and examples. It is understood that the specific examples described herein are only used to explain the relevant content, rather than to limit the present disclosure. It should also be noted that, for ease of description, only the parts related to the present disclosure are shown in the accompanying drawings.

[0036] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present disclosure can be combined with each other. The technical solution of the present disclosure will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0037] Take an e-commerce platform as an example. The e-commerce platform needs to annotate a large number of user product reviews to analyze emotions in order to optimize the recommendation system and product improvement strategy. The annotation of product reviews can include positive, negative and neutral. The annotation process can include: collecting product review texts from e-commerce platforms (such as "The battery life of this phone is too long, I am very satisfied", etc.); defining annotation rules, such as annotating comments that express emotions such as satisfaction, affirmation, and recommendation as positive, annotating comments that express emotions such as dissatisfaction, complaints, and criticism as negative, and annotating comments that do not clearly express emotional tendencies or only narrate facts as neutral; the annotator reads the content of the product review text, selects the corresponding label according to the emotional tendency of the comment, and submits it for system recording after the annotation is completed. Since some product review texts may have ambiguous emotions (such as "The appearance is good, but the price is too expensive"), different annotators may give different labels; and manual annotation takes a long time, especially long comments that need to be read and judged repeatedly, resulting in low data annotation efficiency.

[0038] To this end, the present disclosure proposes a data labeling method, electronic device, storage medium and program product. The present disclosure can be implemented by data labeling software set on electronic devices such as mobile phones, tablet computers, computers, servers, etc.

[0039] Figure 1 The application scenario of the data annotation method of an embodiment of the present disclosure is shown. In this application scenario, a business system 100 and a data annotation device 200 may be included, and the business system 100 communicates with the data annotation device 200 through a network. The business system 100 is used to generate data to be annotated and transmit it to the data annotation device 200, and the data annotation device 200 is used to annotate the data to be annotated.

[0040] For the convenience of description and to make the technical solutions of the specific implementation methods of the present disclosure easier to understand, before describing the data annotation method implemented in the present disclosure, the technical terms involved in the specific implementation methods of the present disclosure are explained as follows: Multi-angle annotation refers to annotating the same data object from multiple different dimensions, perspectives or aspects.

[0041] A business system is a set of interrelated components, processes, rules, and technologies built to achieve specific business functions.

[0042] Figure 2 FIG. 1 is a schematic diagram showing the overall process of a data annotation method M100 according to an embodiment of the present disclosure. Figure 2 The data labeling method shown includes steps S110 to S130. The method can be executed by electronic devices such as mobile phones and computers.

[0043] Specifically, Figure 2 The data annotation methods shown include: Step S110, obtaining data to be labeled.

[0044] In some embodiments of the present disclosure, step S110 may obtain the data to be labeled from the internal database of the data labeling software, step S110 may also obtain the data to be labeled from the business system, step S110 may also obtain the data to be labeled from a public data source, a third-party platform, or from the Internet through crawler technology.

[0045] The data to be annotated obtained through step S110 may be text data (such as articles, comments, emails, forum posts, social media content, product descriptions, etc.), image data (such as photos, scans, charts, etc.), audio data (such as voice, ambient sound, music, etc.), video data, geospatial data, medical data, etc.

[0046] It is understandable that the data involved in the technical solution of the present disclosure (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions, such as the General Data Protection Regulation (GDPR), the Personal Information Protection Law, the Cybersecurity Law, the Data Security Law, etc.

[0047] Step S120 , annotating the data to be annotated from multiple angles based on multiple large language models to obtain multiple model annotation results.

[0048] In some embodiments of the present disclosure, step S120 may first clarify the specific contents of multiple angles of data annotation. Data annotation at different angles is usually a different task type. Different large language models may be selected for multi-angle annotation according to the requirements of the annotation task.

[0049] When selecting a large language model, you can pre-test and evaluate the effect of each large language model on annotations of different angles, and select the model with the best performance in a certain angle as the large language model for annotations of that angle; you can also first clarify the characteristics and requirements of each angle and understand the usage scenarios and performance capabilities of each large language model, and then select the most suitable large language model for each angle based on rule matching or scoring, etc. For example, for sentiment analysis annotations, you can use RoBERTa or GPT series large language models; for named entity recognition annotations, you can use BERT or SpaCy large language models; for topic extraction annotations, you can use GPT series or T5 large language models.

[0050] The multiple large language models in step S120 may be existing large language models, such as BERT, GPT series, RoBERTa, etc. In step S120, each large language model independently annotates the data to be annotated from different perspectives, and the annotation processes of different large language models can be executed in parallel to improve data annotation efficiency.

[0051] Step S130, optimizing the labeling results of multiple models to obtain target labeling results of the data to be labeled.

[0052] In some embodiments of the present disclosure, step S130 may integrate multiple model annotation results (such as using voting mechanism, weighted average, probability fusion, etc.) to obtain a target annotation result of the data to be annotated; step S130 may also use preset rules or logic to automatically correct multiple model annotation results; step S130 may also correct multiple model annotation results based on manual review.

[0053] The data annotation method provided by the present disclosure can more comprehensively reflect the diversity and complexity of data by annotating the data from multiple angles; by optimizing the annotation results of multiple models, the quality of the annotation results can be improved and errors can be reduced. The method performs data annotation based on a large language model, which can solve the problems in the prior art that manual data annotation consumes a lot of manpower and time costs, and the data annotation efficiency is low; and manual data annotation is easily affected by subjective factors, resulting in high inconsistency and error rate in the annotation results.

[0054] Regarding step S130, in some embodiments of the present disclosure, it may include the following steps: Figure 3 Steps S131 to S133 are shown.

[0055] Step S131, determining the target user of the data to be labeled from the candidate user group.

[0056] In some embodiments of the present disclosure, the target user determined by step S131 may be one or more. After the target user is determined by step S131, the data to be labeled can be assigned to the target user, so that the target user can query the data labeling task to be processed by the target user by means of task query or the like.

[0057] Step S132: receiving a processing instruction from the target user.

[0058] In some embodiments of the present disclosure, the processing instructions received through step S132 may be input by the target user after reviewing the multiple model annotation results; for example: when the target user believes that a certain annotation is incorrect, the processing instruction may be an instruction to correct the label; when the target user believes that the multiple model annotation results are incomplete, the processing instruction may be an instruction to add the annotation; when the target user believes that a certain annotation is redundant, the processing instruction may be an instruction to delete the annotation, etc.

[0059] Step S133: Optimize the labeling results of the multiple models according to the processing instructions.

[0060] In some embodiments of the present disclosure, step S133 can optimize the multiple model annotation results according to the specific content of the processing instruction received through step S132, such as modifying a certain annotation in the multiple model annotation results, deleting a certain annotation in the multiple model annotation results, adding an annotation, etc.

[0061] In some embodiments of the present disclosure, when there are multiple target users determined by step S131, step S133 can independently optimize multiple model annotation results based on the processing instructions of different target users, and after obtaining individual optimization results, determine whether the multiple individual optimization results are consistent; if they are consistent, use the individual optimization results as the target annotation results; if they are inconsistent, the target annotation results can be determined automatically or by an auditor.

[0062] The method of automatically determining the target labeling result can be to select the individual optimization result of a user with higher confidence from multiple individual optimization results as the target labeling result according to the user's confidence score; or to determine the target labeling result according to the user's historical labeling data. For example, if a user has a better labeling effect in a similar task, the individual optimization result of the user can be used as the target labeling result; or to select which user's individual optimization result to use as the target labeling result according to pre-defined rules (such as similarity, domain knowledge, labeling type, etc.).

[0063] When the auditor determines the target labeling result, in order to help the auditor make a reasonable decision, the auditor can provide the background information of the relevant users (such as confidence, etc.) while providing the individual labeling results. The auditor selects a target labeling result from multiple individual optimization results based on the relevant information, or re-optimizes multiple model labeling results based on his own experience. The process of the auditor re-optimizing multiple model labeling results based on his own experience is similar to that provided in step S132 and step S133.

[0064] Steps S131 to S133 optimize the labeling results of multiple models manually, and can flexibly determine the labelers according to the difficulty of the task, the urgency of the task, the labeler's skills, etc., and support multi-person collaboration, review, and task cross-validation, which can improve the quality and accuracy of the labeling results. In addition, by optimizing the labeling results with appropriate target users, it can reduce errors and rework time, reduce the cost of manual management, and improve the overall data labeling efficiency.

[0065] Regarding step S131, in some embodiments of the present disclosure, it can be replaced by Figure 4 The following steps are shown: Randomly determine the target user of the data to be labeled from the candidate user group.

[0066] In some embodiments of the present disclosure, the randomness used in this step may be simple randomness, weighted randomness, etc.

[0067] By randomly selecting target users, we can increase the diversity of annotations, reduce the bias of annotators, and avoid overwork of a single annotator.

[0068] Regarding step S131, in some embodiments of the present disclosure, it can be replaced by Figure 5 The following steps are shown: Determine the target users of the data to be labeled from the candidate user group based on the matching dimensions.

[0069] In some embodiments of the present disclosure, the matching dimensions in this step may include one or more of user features, data features of the data to be annotated, and task features of the annotation task for the data to be annotated. User features may include the region, language, past service records, and focus of the user who provides the data to be annotated; data features may include the content, type, complexity, language, and specific annotation requirements of the current data to be annotated; and task features may include the urgency, working time requirements, and quality requirements of the annotation task for the data to be annotated.

[0070] The process of determining the target users of the data to be labeled from the candidate user group through this step may include: first, determining the features included in the matching dimension according to the task requirements; second, building a feature database of the candidate user group, that is, organizing the information of the candidate user group into structured data, including skill information (such as what type of data they are good at processing, whether they have knowledge of a specific language or a specific field, etc.), past performance (such as the completion quality and efficiency of historical labeling tasks, average labeling accuracy and consistency, etc.), status information (such as current workload, whether they are online, available time periods and task preferences, etc.) and diversity information (such as whether excessive bias or poor consistency is introduced in the task); finally, when the matching dimension includes only a single feature, some users who meet the single feature can be directly screened out from the candidate user group according to the feature database, and the final target user can be determined from this part of users randomly or according to a certain rule (such as the current idle state, etc.); when the matching dimension includes multiple features, each feature can be given a weight according to the importance of the feature determined by the task requirements, and the target user can be determined according to the feature database based on conditional screening, user ratings or large language models.

[0071] Further, in some embodiments of the present disclosure, the data annotation method provided by the present disclosure may further include, after step S130, the following steps: Figure 6 Step S140 is shown.

[0072] Step S140: fine-tune multiple large language models according to the target labeling results.

[0073] In some embodiments of the present disclosure, when the data volume of the target labeling result is large and the computing resources are sufficient, the fine-tuning in step S140 can be specifically full-parameter fine-tuning, that is, all parameters of the large language model are updated; when it is necessary to be compatible with multi-task fine-tuning and reduce resource usage, the fine-tuning in step S140 can be specifically parameter efficient fine-tuning, that is, optimizing the large language model by adding a small number of new parameters.

[0074] In some embodiments of the present disclosure, the process of fine-tuning multiple large language models according to the target labeling results in step S140 may include: calculating the error (loss function) between multiple model labeling results output by the multiple large language models and the target labeling results, and updating the parameters of the large language model based on the error through back propagation.

[0075] Fine-tuning the large language model in step S140 can improve the annotation quality, domain adaptability, and efficiency of the large language model and reduce manual intervention.

[0076] Regarding step S110, in some embodiments of the present disclosure, it can be replaced by Figure 7 The following steps are shown: Get the data to be labeled from the dataset.

[0077] In some embodiments of the present disclosure, the data set in this step is usually an independent data file or structured data storage, which may come from historical data accumulation, public data sets, etc.; the data set may include data of types such as text, image, video, audio, etc. The data in the data set may be the original data of the public data set, etc.; in particular, in order to ensure the annotation quality of the data to be annotated and improve the annotation efficiency, the data in the data set may also be the data after pre-processing such as cleaning, normalization, and formatting of the original data.

[0078] This step can obtain high-quality and stable data to be labeled.

[0079] Regarding step S110, in some embodiments of the present disclosure, it can be replaced by Figure 8 The following steps are shown: Obtain the data to be labeled from the business system.

[0080] In some implementations of the present disclosure, the business system in this step may be an e-commerce platform, a customer management system, a log system, etc.

[0081] This step can obtain data to be annotated with strong business orientation, ensuring that the annotation task is highly consistent with the actual application scenario.

[0082] about Figure 8 In some embodiments of the present disclosure, step S110 shown in FIG. 1 may be replaced by Fig. 9 The following steps are shown: Obtain the data to be labeled from the business system based on manual instructions.

[0083] In some embodiments of the present disclosure, manual instructions are clear instructions provided by users according to actual needs to help filter out data to be annotated from the business system. The contents of the manual instructions may include data type, time range, specific conditions, data identification, etc.

[0084] This step of obtaining the data to be labeled based on manual instructions can ensure that the data to be labeled meets specific requirements and has strong flexibility.

[0085] about Figure 8 In some embodiments of the present disclosure, step S110 shown in FIG. 1 may be replaced by Fig.10 The following steps are shown: The data to be annotated is obtained from the business system based on one or more triggering factors including conditions, events and time.

[0086] In some embodiments of the present disclosure, the conditions in this step are screening conditions for screening the data to be labeled, such as a specific user group, transaction amount, product category, etc.; the events in this step are used to clarify which events will trigger the acquisition of the data to be labeled, such as a user placing an order, making a complaint, clicking on an advertisement, etc.; the time in this step is used to clarify the time range within which the data to be labeled is to be acquired, such as a certain day, week, month, etc.

[0087] When this step uses conditions as trigger factors, it can improve the accuracy and direction of acquiring data to be labeled; when this step uses events as trigger factors, it can improve the real-time and flexibility of acquiring data to be labeled; when this step uses time as trigger factors, it can improve the timeliness of acquiring data to be labeled; when there are multiple trigger factors in this step, it can cope with complex needs for acquiring data to be labeled. For example, using events and time as trigger factors can realize dynamic and quasi-real-time data acquisition, quickly acquire data to be labeled at critical moments, and thus avoid data lags.

[0088] about Figure 8 The step S110 shown obtains the data to be labeled from the business system. The obtained data to be labeled can be the original business data of the business system. In particular, in order to improve data quality, the obtained data to be labeled can also be data after processing the business data. The processing process can be performed using a predefined processing method.

[0089] In particular, in order to increase the flexibility of data processing, Figure 8 In some embodiments of the present disclosure, step S110 shown in FIG. 1 may include: Fig.11 Steps S111 to S114 are shown.

[0090] Step S111, obtaining business data from the business system.

[0091] In some embodiments of the present disclosure, step S111 may obtain business data from the business system through database query, API interface or business system log, etc. The business data obtained in step S111 may be transaction data, customer data, product data or marketing data, etc.

[0092] In order to obtain more diverse and comprehensive business data, step S111 can obtain business data from the business system through multiple channels, such as API interface, data import in standardized formats (such as data import in formats such as CSV, Excel, Json, etc.), message queues (such as Kafka, RabbitMQ, etc.), data synchronization tools (such as Talend, Apache Nifi, EventBus, etc.), HTTP and other channels, thereby realizing multi-channel data collection capabilities.

[0093] Step S112, acquiring characteristic information of the business data.

[0094] In some embodiments of the present disclosure, the feature information obtained through step S112 may include one or more of data type (such as numerical type, categorical type, time type, text type, etc.), data format (such as structured data, semi-structured data, unstructured data, etc.), data quality (such as missing value ratio, number of outliers, duplicate data, etc.), data range (such as numerical range or category distribution, etc.) and system environment (such as whether there are external factors or environment affecting the data, etc.).

[0095] Step S113, obtaining a processing method for the business data according to the characteristic information.

[0096] In some embodiments of the present disclosure, step S113 can automatically select the most appropriate processing method according to the feature information using methods such as rule engine, machine learning, dynamic optimization, quality assessment, etc. The processing method obtained by step S113 may include one or more of missing value filling, outlier correction, deduplication, standardization, conversion, data merging and splitting.

[0097] Step S114, using the processing method to process the business data to obtain data to be labeled.

[0098] Steps S111 to S114 process the business data by adaptively selecting a suitable processing method for the business data, thereby achieving differentiated processing for different data and improving the quality and processing efficiency of the data to be annotated. Adaptively selecting a suitable processing method for the business data can avoid the problem of reconstructing the processing flow due to changes in the data format and improve the data processing speed.

[0099] Regarding step S113, in some embodiments of the present disclosure, it can be replaced by Fig.12 The following steps are shown: Input feature information into the large language model to obtain the processing method of business data.

[0100] In some embodiments of the present disclosure, the large language model in this step may be an existing large language model, such as BERT, GPT series, RoBERTa and other models. In particular, in order to improve the performance of feature information analysis of the large language model, the large language model used in this step may also be a fine-tuned large language model. The fine-tuning method may be supervised fine-tuning, unsupervised fine-tuning, domain adaptation fine-tuning, reinforcement learning fine-tuning, etc.

[0101] Taking supervised fine-tuning as an example, the training data can be used as samples in advance to adjust the model parameters of the large language model in this step. Among them, the training data is a data pair consisting of training feature information and training processing methods; the process of fine-tuning the large language model is to input the training feature information into the large language model, and the large language model outputs the corresponding analysis results, and then calculates the loss between the analysis results and the training processing method, and optimizes the model parameters of the large language model with the purpose of reducing the loss; through multiple rounds of iterations, the analysis results output by the large language model can eventually be close to the expected training processing method, thereby improving the performance of the large language model.

[0102] This step uses a large language model to analyze the processing method, which can deeply understand the feature information and provide a reasonable processing method.

[0103] Regarding step S113, in some embodiments of the present disclosure, it can be replaced by Fig.13 The following steps are shown: The processing method of obtaining business data based on characteristic information based on processing rules.

[0104] In some embodiments of the present disclosure, the processing rules in this step may be a technical structure that can be dynamically selected, such as a decision tree structure, etc. Each node in the decision tree corresponds to a check rule to decide whether to adopt a certain processing method, and the leaf node gives the final processing method.

[0105] This step uses a decision tree approach, with a clear and hierarchical decision process, which helps to automatically select the most appropriate processing method under different data conditions.

[0106] Regarding step S120 of annotating the data to be annotated from multiple angles based on multiple large language models, the inputs of the multiple large language models may include the data to be annotated and subject prompts for data annotation (such as please annotate the data from angle X, etc.).

[0107] In particular, in order to improve the annotation efficiency and quality of the large language model, regarding step S120, in some embodiments of the present disclosure, it may include the following: Fig.14 Steps S121 to S122 are shown.

[0108] Step S121, obtaining a candidate tag set.

[0109] In some embodiments of the present disclosure, the candidate tag set obtained by step S121 is a set of predefined tags as possible annotation options, and the tags in the candidate tag set can be generated by expert definition, data drive, reference standard and / or large language model, etc. Different candidate tag sets can be used for different annotation tasks.

[0110] Step S122 , annotating the data to be annotated from multiple angles according to the candidate tag sets based on multiple large language models.

[0111] Steps S121 to S122 provide clear guidance in the data labeling process through the candidate label set, which can ensure the standardization of labeling and the reliability of results.

[0112] In some embodiments of the present disclosure, the multi-angle annotation in step S120 may include annotation based on at least two dimensions of data category dimension, data content dimension, data attribute dimension, context dimension, domain dimension and anomaly dimension.

[0113] The data labeling method provided in the present disclosure can use Figure 1 Steps S110 to S130 shown implement data labeling; in particular, in order to reduce the amount of manpower invested in the optimization process, after obtaining multiple model labeling results through step S120, the confidence of the multiple model labeling results can also be automatically detected; if the confidence is greater than a threshold, the multiple model labeling results are directly used as target labeling results; if the confidence is not greater than the threshold, the multiple model labeling results are optimized through step S130.

[0114] The data labeling method provided by the present invention can support multiple data sources and adapt to the needs of different business scenarios; the data labeling method can be applied to natural language processing, computer vision, speech processing, medical care, finance, retail, education, security, environment, social media and other fields to assist in artificial intelligence model training and data value mining.

[0115] Fig.15 An exemplary flowchart of the data annotation method implemented based on the present disclosure is shown.

[0116] Fig.15 In the flowchart shown, taking customer service chat record annotation as an example, the data annotation process may include: Step S210, obtaining a conversation log from the customer service system.

[0117] In some embodiments of the present disclosure, the conversation log in this step may include customer messages, customer service replies, conversation time, etc.

[0118] The conversation log obtained in step S210 is, for example: [2024-12-12 10:05:02]***System message: Session started*** [2024-12-12 10:05:05]Customer: Why hasn’t my order been shipped yet? ??? ??? I’ve been waiting for three days! !!! !!! I’m so disappointed!!! [2024-12-12 10:05:15] Customer Service: Sorry, your order is indeed delayed and is being processed. Please wait patiently. [2024-12-12 10:05:25]***System message: The client sent a file*** [2024-12-12 10:05:35]Customer: I really don’t understand! Why is it so slow? ? I also saw your website saying “shipping within 24 hours”. [2024-12-12 10:05:50] Customer Service: I understand your dissatisfaction, but we are speeding up the process, you can wait a moment... [2024-12-12 10:06:02]***System message: Session ended*** Step S220: Process the conversation log to obtain data to be annotated.

[0119] In some embodiments of the present disclosure, the process of processing the conversation log through step S220 may include data cleaning (such as unifying the case of text in the conversation log, removing unnecessary spaces and special symbols, etc.; for multilingual conversations, detecting and marking the language type, or uniformly translating into the target language), conversation role identification (distinguishing between the speech of customers and customer service), etc.

[0120] After processing the example of the conversation log in step S210 through this step, the data to be annotated is obtained as follows: Customer: Why hasn’t my order been shipped yet? I’ve been waiting for three days. I’m so disappointed! Customer Service: Sorry, your order is indeed delayed and is being processed. Please be patient. Customer: I don’t understand why it’s so slow? I saw on your website that it says “shipping within 24 hours”.

[0121] Customer Service: I understand your dissatisfaction, but we are speeding up the process and you can wait a moment.

[0122] Step S230 , annotating the data to be annotated with respect to data content dimensions and anomaly dimensions based on multiple large language models, and obtaining multiple model annotation results.

[0123] In some embodiments of the present disclosure, step S230 may use only the data to be labeled for abnormal dimension labeling. In order to identify abnormal situations from multiple angles, the data to be labeled may also be combined with user behavior data (such as operation frequency, device changes, access mode changes, etc.), time features (such as users performing a large number of activities at unconventional times (such as late at night, holidays, etc.)) for abnormal dimension labeling.

[0124] After multi-angle annotation of the data to be annotated based on multiple large language models in step S230, the data content dimension annotation "negative" and the abnormal dimension annotation "micro-business" can be obtained.

[0125] In some embodiments of the present disclosure, the multiple model annotation results obtained through step S230 may only include labels; in particular, in order to facilitate subsequent optimization processing, the multiple model annotation results obtained through step S230 may include, in addition to the label, the reason for generating the label.

[0126] Step S240, optimizing the labeling results of the multiple models to obtain a target labeling result of the data to be labeled.

[0127] In some embodiments of the present disclosure, step S240 can optimize the annotation results of multiple models based on manual instructions, such as correcting the annotation results corresponding to the data content dimension. For example, in the above example, although the customer expresses dissatisfaction, the emotion may not be completely negative. He may hope to get faster service. At this time, the annotation result corresponding to the data content dimension can be modified to "neutral".

[0128] In the process of optimizing the annotation results of multiple models through step S240, the annotation results corresponding to the abnormal dimensions obtained through step S230 can be used as part of the final target annotation results, or the annotation results corresponding to the abnormal dimensions can be used as relevant indicators to assist in optimizing the annotation results corresponding to the data content dimensions.

[0129] Step S250: fine-tune multiple large language models according to the target labeling results.

[0130] In some embodiments of the present disclosure, the large language model is fine-tuned through step S250, that is, the target labeling result is fed back to the large language model, so that the large language model can be continuously iterated and optimized, the accuracy of the large language model labeling is improved, and a positive cycle of data-model-labeling is formed.

[0131] The data annotation method provided by the present disclosure initially generates annotation results through a large language model, which can reduce the workload of manual annotation and improve data annotation efficiency.

[0132] The data annotation method provided by the present disclosure is an indispensable process for building a data set, and it has multiple usage scenarios: 1. Data annotation is one of the core steps in building a large model, mainly used to generate high-quality training data so that the large model can learn the instructions of a specific task; 2. Other complex scenarios, such as trend prediction, decision judgment, strategy optimization, statistical analysis, etc. This data annotation method is particularly suitable for scenarios that require rapid response and high-quality data support, such as market intelligence, autonomous driving, precision marketing, decision assistance, etc., and can effectively improve business decision-making efficiency and model performance.

[0133] Based on any of the above implementations, the present disclosure also provides a data labeling device.

[0134] Fig.16 It is a schematic block diagram of the structure of a data labeling device according to an embodiment of the present invention.

[0135] like Fig.16 As shown, the data labeling device includes: The data acquisition module 110 is used to acquire data to be labeled.

[0136] The model annotation module 120 is used to perform multi-angle annotation on the data to be annotated based on multiple large language models to obtain multiple model annotation results.

[0137] The labeling optimization module 130 is used to optimize the labeling results of multiple models to obtain target labeling results of the data to be labeled.

[0138] The data labeling device may be in the form of computer software, and each module of the data labeling device may be implemented by a computer software module.

[0139] The implementation process of the functions and effects of each module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, which will not be repeated here.

[0140] The executor of the data annotation method in the specific implementation manner of the present disclosure may be a mobile phone, a tablet computer, a computer, a server, or other electronic device.

[0141] Therefore, based on any one of the above embodiments, the present disclosure further provides an electronic device, which can execute the data labeling method of any one of the embodiments described above in the present disclosure.

[0142] Fig.17 1 is a schematic block diagram of the structure of an electronic device 1000 according to an embodiment of the present disclosure.

[0143] The hardware structure of the electronic device 1000 can be implemented using a bus architecture. The bus architecture can include any number of interconnected buses and bridges, depending on the specific application and overall design constraints of the hardware. The bus 1100 connects various circuits including one or more processors 1200, memory 1300 and / or hardware modules together. The bus 1100 can also connect various other circuits 1400 such as peripherals, voltage regulators, power management circuits, external antennas, etc.

[0144] The bus 1100 may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the figure only uses one connecting line, but does not mean that there is only one bus or one type of bus.

[0145] The present disclosure also provides a readable storage medium, in which a computer program is stored, and the computer program is used to implement the above method when executed by a processor. "Readable storage medium" can be any device that can contain, store, communicate, propagate or transmit a program for use in an instruction execution system, device or equipment or in combination with these instruction execution systems, devices or equipment. More specific examples of readable storage media include the following: an electrical connection portion (electronic device) with one or more wirings, a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and editable read-only memory (EPROM or flash memory), an optical fiber device, and a portable read-only memory (CDROM), etc.

[0146] The present disclosure also provides a computer program product. The method of the present disclosure can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instruction is loaded and executed, the process or function of the present disclosure is executed in whole or in part.

[0147] The computer program or instructions may be stored in a readable storage medium or transmitted from one readable storage medium to another readable storage medium, for example, the computer program or instructions may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired or wireless means. The readable storage medium may be any available medium that can be accessed or a data storage device such as a server, data center, etc. that integrates one or more available media. The available medium may be a magnetic medium, such as a floppy disk, a hard disk, or a magnetic tape; it may also be an optical medium, such as a digital video disk; it may also be a semiconductor medium, such as a solid state drive. The computer readable storage medium may be a volatile or non-volatile storage medium, or may include both volatile and non-volatile types of storage media.

[0148] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The present disclosure is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] In the description of this specification, the description with reference to the terms "one embodiment / method", "some embodiments / methods", "example", "specific example", or "some examples" etc. means that the specific features, structures, or characteristics described in conjunction with the embodiment / method or example are included in at least one embodiment / method or example of the present disclosure. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment / method or example. Moreover, the specific features, structures, or characteristics described may be combined in any one or more embodiments / methods or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments / methods or examples described in this specification and features of different embodiments / methods or examples, unless they are contradictory.

[0153] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, "plurality" means at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0154] Those skilled in the art should understand that the above embodiments are only for the purpose of clearly illustrating the present disclosure, and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications may be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A data labeling method, characterized in that: include: Obtain the data to be labeled; Annotating the data to be annotated from multiple angles based on multiple large language models to obtain multiple model annotation results; as well as The multiple model labeling results are optimized to obtain target labeling results for the data to be labeled.

2. The data labeling method according to claim 1, characterized in that: The optimizing the multiple model annotation results includes: Determining a target user of the data to be labeled from a candidate user group; receiving a processing instruction from the target user; and The multiple model annotation results are optimized according to the processing instructions.

3. The data labeling method according to claim 2, characterized in that: The determining of the target user of the data to be labeled from the candidate user group includes: Randomly determine the target user of the data to be labeled from the candidate user group; or, The target user of the data to be labeled is determined from the candidate user group according to a matching dimension, wherein the matching dimension includes one or more of user characteristics, data characteristics of the data to be labeled, and task characteristics of a labeling task for the data to be labeled.

4. The data labeling method according to any one of claims 1 to 3, characterized in that: After obtaining the target labeling result of the data to be labeled, the method further includes: The multiple large language models are fine-tuned according to the target labeling results.

5. The data labeling method according to any one of claims 1 to 3, characterized in that: The step of obtaining the data to be labeled includes: Get the data to be labeled from the dataset; or, Obtain the data to be labeled from the business system.

6. The data labeling method according to claim 5, characterized in that: The obtaining of the data to be labeled from the business system includes: Acquire the data to be annotated from the business system based on manual instructions; or The data to be annotated is obtained from the business system based on one or more triggering factors including conditions, events and time.

7. The data labeling method according to claim 5, characterized in that: The collecting of data to be labeled from the business system includes: Obtain business data from business systems; Acquiring characteristic information of the business data; Acquire a processing method of the business data according to the characteristic information; Processing the business data using the processing method to obtain data to be labeled; Optionally, the characteristic information includes: one or more of data type, data format, data quality, data range and system environment; and / or the processing method includes one or more of missing value filling, outlier correction, deduplication, standardization, conversion, data merging and splitting; Optionally, the obtaining of the processing mode of the business data according to the feature information comprises: inputting the feature information into a large language model to obtain the processing mode of the business data; or obtaining the processing mode of the business data according to the feature information based on a processing rule; Optionally, the multi-angle labeling of the data to be labeled based on multiple large language models includes: Get candidate label set; Based on multiple large language models, the data to be labeled is labeled from multiple angles according to the candidate label set; Optionally, the multi-angle annotation includes: Labeling based on at least two dimensions of data category dimension, data content dimension, data attribute dimension, situation dimension, domain dimension and anomaly dimension.

8. An electronic device, characterized in that: include: A memory storing execution instructions; as well as A processor, wherein the processor executes the execution instructions stored in the memory, so that the processor executes the data labeling method according to any one of claims 1 to 7.

9. A readable storage medium, characterized in that: The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the data labeling method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the data labeling method according to any one of claims 1 to 7 is implemented.

Images