Data processing method, system and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method, system and device, electronic equipment, a storage medium and a program product. In the scheme provided by the embodiment, the target data is one of a plurality of pieces of data to be labeled contained in a certain data labeling item. By means of the data annotation model, automatic annotation processing can be executed on the target data based on the called prompt word matched with the data annotation item, and a first annotation result of the target data is generated. The first annotation result is presented to an annotator, and the annotator executes manual annotation operation on the target data based on the first annotation result to generate a second annotation result of the target data. The second annotation result, the first annotation result and the target data are stored in an associated mode, and the cue word is optimized according to annotation difference information obtained through difference analysis by conducting difference analysis on the first annotation result and the second annotation result corresponding to the multiple pieces of stored target data in the subsequent steps.

Description

Technical Field

[0001] This specification relates to the field of data annotation technology, and in particular to a data processing method, system, device, electronic device, storage medium, and program product. Background Technology

[0002] With the widespread application of artificial intelligence technology, high-quality data annotation has become a core element in ensuring the training effect of machine learning models. Currently, the mainstream data annotation methods used in data annotation systems mainly rely on manual annotation or rule-based semi-automatic annotation, which cannot meet the needs of annotation processing massive amounts of data. Moreover, as the scale of data annotation expands and the complexity of data annotation tasks increases, data annotation systems need to have stronger adaptability to better meet different data annotation requirements.

[0003] Therefore, there is a need to provide a more efficient and adaptive data annotation system. Summary of the Invention

[0004] Several embodiments in this specification provide a data processing method, system, apparatus, electronic device, storage medium, and program product that can maintain superior annotation performance and flexibility in complex and dynamic data processing tasks. Among them, In a first embodiment, this specification provides a data processing method. The method includes: Obtain target data; the target data is one of multiple data to be labeled contained in a data labeling project; Invoke prompt words that are compatible with the data annotation project; the prompt words are used to guide the data annotation model to perform data annotation processing on the input target data; The target data and the prompt words are input into the data annotation model, and the first annotation result of the target data is output. In response to the manual annotation operation performed by the annotator on the target data based on the first annotation result, a second annotation result for the target data is obtained; The target data, the first annotation result, and the second annotation result are stored together in association; A difference analysis is performed on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; Based on the annotation difference information, optimize the prompt words.

[0005] In a second embodiment, this specification provides a data processing system. The system includes: The server is used to obtain target data; the target data is one of multiple unlabeled data included in a data annotation project; it calls a prompt word that is compatible with the data annotation project; the prompt word is used to guide the data annotation model to annotate the input target data; the target data and the prompt word are input into the data annotation model to obtain the first annotation result of the target data; The client is used to display the first annotation result of the target data; in response to the manual annotation operation performed by the annotator on the target data based on the first annotation result, obtain the second annotation result of the target data; and send the second annotation result to the server. The server is also used to associate and store the target data with the first annotation result and the second annotation result; perform difference analysis on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; and optimize the prompt words based on the annotation difference information.

[0006] In a third embodiment, this specification provides a data processing apparatus. The apparatus includes: The acquisition module is used to acquire target data; the target data is one of multiple data to be labeled contained in a data labeling project; The calling module is used to call prompt words that are compatible with the data annotation project; the prompt words are used to guide the data annotation model to annotate the input target data; The input module is used to input the target data and the prompt words into the data annotation model to obtain the first annotation result of the target data; The response module is used to respond to the manual annotation operation performed by the annotator on the target data based on the first annotation result, and obtain the second annotation result of the target data; An associated storage module is used to associate and store the target data, the first annotation result, and the second annotation result. The analysis module is used to perform difference analysis on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; An optimization module is used to optimize the prompt words based on the annotation difference information.

[0007] In a fourth embodiment, this specification provides an electronic device including a memory and a processor, wherein the memory stores executable program instructions, and when the processor executes the program instructions, it implements the method provided in the first embodiment.

[0008] In a fifth embodiment, this specification provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform the method provided in the first embodiment.

[0009] In a sixth embodiment, this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method provided in the first embodiment.

[0010] The solutions provided in the above embodiments of this specification involve target data that is one of multiple unlabeled data sets included in a data annotation project. Using a data annotation model, automatic annotation processing is performed on the target data based on prompts that are compatible with the data annotation project, thereby generating a first annotation result for the target data. The prompts are used to guide the data annotation model in annotating the input target data. Using a data annotation model to achieve automatic annotation processing of the target data reduces the workload of subsequent manual annotation. Further, the first annotation result is presented to the annotator, who then performs manual annotation operations on the target data based on the first annotation result, thereby obtaining a second annotation result for the target data. This second annotation result is stored in association with the first annotation result and the target data, enabling subsequent steps to optimize the prompts based on the annotation difference information obtained from the difference analysis by performing difference analysis on the first and second annotation results corresponding to each of the multiple target data sets. In summary, this solution enables the model to learn from past labeled data (including both automatic and manual annotations of the target data) and continuously adapt the prompts used to guide the data annotation model in the annotation process. This improves the quality of automatic data annotation, reduces manual workload, and enhances overall data annotation efficiency. Furthermore, the adaptive adjustment of prompts also enhances the responsiveness to changes in the data to be annotated and new data annotation requirements. Attached Figure Description

[0011] To more clearly illustrate the technical solutions of the various embodiments disclosed in this specification, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely examples of the various embodiments disclosed in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort. In the drawings: Figure 1 A schematic diagram of the technical architecture on which the implementation of each method in this specification is based, provided for exemplary embodiments; Figure 2 This is a schematic diagram of the structure of a data processing system provided for exemplary embodiments in this specification; Figure 3 A flowchart illustrating a data processing method provided as an exemplary embodiment of this specification; Figure 4 A schematic diagram of the structure of a data processing apparatus provided for exemplary embodiments in this specification; Figure 5 This is a schematic diagram of the structure of an electronic device provided as an exemplary embodiment of this specification. Detailed Implementation

[0012] In the era of rapid development of big data and artificial intelligence, Natural Language Processing (NLP), as a core technology for human-computer language interaction, has been widely applied in fields such as financial risk control, medical record mining, and intelligent customer service. In these applications, high-quality structured text data is the foundation for training relevant machine learning models and intelligent decision-making. Text annotation, as a key step in transforming unstructured text data into structured text data that machines can understand, has become an indispensable pre-process in NLP tasks. Furthermore, text annotation is a specific form of data annotation. Data annotation refers to adding semantically or structurally meaningful labels to raw data (such as text, images, audio, and video), enabling the corresponding machine learning models to better understand and learn. Most modern data annotation work relies on corresponding data annotation systems (also known as data annotation platforms). Currently, mainstream data annotation schemes used in data annotation systems, such as manual annotation and rule-based annotation (such as rule-based semi-automatic annotation), face challenges in handling massive amounts of data and maintaining consistency and high accuracy.

[0013] For example, some data annotation systems employ rule-based annotation schemes. These schemes rely on expert-defined annotation rules and patterns (such as regular expressions or specific string matching) for annotation. However, the creation and updating of these rules require significant human intervention and specialized knowledge, making them ill-suited to changing data environments. Furthermore, the lack of an automatic iterative update mechanism means the data annotation system cannot self-improve, resulting in poor flexibility and adaptability. While machine learning annotation schemes have been adopted in other data annotation systems with the development of artificial intelligence, these schemes typically rely on the extraction and classification of fixed features, using pre-trained machine learning models (such as deep learning models) to perform data annotation tasks. These models require a large amount of manually labeled data during the training phase, and are usually fixed in their design and not updated with new labeled data. Therefore, while machine learning annotation schemes may perform well initially, the lack of an automatic iteration mechanism prevents them from adapting to new data annotation patterns or task requirements. Regular manual updates and retraining of the machine learning model are necessary, resulting in high training costs and time consumption.

[0014] In summary, current data annotation systems lack the ability to adapt and iterate automatically, and cannot update and optimize themselves based on real-time data and feedback. Therefore, their data annotation performance is often limited in long-term applications.

[0015] To address the aforementioned issues, this specification provides a solution that employs a self-iterative data pre-labeling scheme based on a Large Language Model (LLM) in the data annotation system. Specifically, the basic idea is as follows: In the data annotation system, the LLM is guided to pre-label the data to be labeled by dynamically adjusting the prompts. The pre-labeling results are then fed back to the manual annotation results based on the pre-labeling results for comprehensive analysis, thereby continuously optimizing the prompts.

[0016] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments in this specification, and not all of them. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0017] It should be noted that, for ease of description, the accompanying drawings only show the parts related to the relevant technical solutions. Unless otherwise specified, the embodiments and features described in this specification can be combined with each other. Furthermore, the terms "first," "second," and "third" used in the embodiments of this specification are for informational purposes only and do not constitute any limitation. Moreover, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, the presence of other identical or equivalent elements in the process, method, product, or apparatus that includes the stated elements is not excluded. Furthermore, in this specification, unless explicitly stated otherwise, "receiving and transmitting data" does not necessarily mean direct receiving and transmitting; it can be indirect receiving and transmitting. For example, when A receives data sent by B, it can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, when B sends data to A, it can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0018] Furthermore, it should be noted that specific terms are used to describe embodiments of this specification. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different locations in this specification do not necessarily refer to the same embodiment. Moreover, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples, without contradiction. Although one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible order of execution among many steps, and does not represent the only possible order. Therefore, when the claims involve method steps, adjustments to the order of such steps, or parallel execution between steps, are also within the scope of protection of the claims.

[0019] Furthermore, it should be noted that the user data obtained in this manual is authorized by the user and does not involve user privacy.

[0020] The embodiments provided in this specification will be described below with reference to the accompanying drawings.

[0021] First, the terminology used in the embodiments of this specification will be explained. It should be understood that this explanation is for the purpose of providing a clearer understanding of the embodiments described herein and does not necessarily constitute a limitation on the embodiments of this specification.

[0022] Large Language Model (LLM): This is an artificial intelligence model (including machine learning models) designed to understand and generate human language. The characteristics of a Large Language Model (LLM) are its massive scale; it typically consists of millions or even billions of parameters, which help the model learn complex patterns in language data.

[0023] Text annotation is a type of data labeling, specifically a key process in Natural Language Processing (NLP) that involves identifying and classifying text data so that machine learning models can understand and process it. The goal of text annotation is to add structured labeling information to raw text data, making it suitable for analysis and processing by machine learning models.

[0024] The Model Context Protocol (MCP) is an open protocol that standardizes the way applications provide context to large language models. In other words, it allows applications to provide context to large language models in a unified and standardized manner. "Context" refers to the background information (such as dialogue history, external knowledge, and the user's current question) that the large language model relies on when generating answers or performing corresponding tasks.

[0025] A data annotation system, also known as a data annotation platform, is a software system or platform that integrates the management, allocation, and execution of data annotation tasks.

[0026] The technical solutions provided in the embodiments described below are all based on Figure 1 The technical architecture implementation is shown in the diagram. For example... Figure 1 As shown, the technical architecture mainly includes the following components: a prompt word repository 10, a data pre-labeling tool 11, a case-shot set 12, and a comprehensive analysis model for annotation results 13. Furthermore, the data processing steps implemented collaboratively by these components include: pre-labeling, manual labeling, annotation result feedback, comprehensive analysis of annotation results, and prompt word updating.

[0027] The following section details the above components and the data processing steps achieved through their collaboration.

[0028] 1) Prompt word repository 10 The prompt word repository 10 is used to store and update multiple prompt words. These prompt words are annotation hints designed for the data annotation needs of different data annotation projects, used to guide the data pre-labeling tool 11 (specifically, the data annotation model within it) to complete the data annotation task. A data annotation project typically includes, but is not limited to, the following core requirements: a dataset to be labeled (containing multiple data points to be labeled; these are the raw data that need to be labeled, such as text, images, audio, and video data), data annotation requirements (i.e., annotation rules, specifying what content needs to be labeled (e.g., sentiment, entities, objects), and the labeling system (e.g., positive / negative / neutral, people / cars / trees)).

[0029] For example, the text data annotation requirement for a certain text data annotation project is to determine sentiment. Then, the initial prompt word designed for this text data annotation project could be "Please determine whether the sentiment of the following text data is positive, negative, or neutral," and this prompt word would be stored in the prompt word repository 10. During storage, the prompt word can be associated with the corresponding project identifier for the data annotation project.

[0030] Based on the above example, in the prompt word repository 10, the storage format for prompt words can be "prompt word: project identifier of data annotation project".

[0031] For information on updating the prompt words in the prompt word repository 10, please refer to the relevant content given in the introduction of "Comprehensive Analysis Model 13 of Annotation Results" below, which will not be elaborated here. In addition, in this specification, the prompt word repository 10 is made available in the form of MCP resources so that data pre-annotation tools such as 11 can access the prompt word repository 10.

[0032] 2) Data pre-labeling tool 11 and pre-labeling The data pre-labeling tool 11 is used to read relevant prompt words from the prompt word repository 10 to pre-label the received target data based on the prompt words. "Pre-labeling" specifically refers to using the data pre-labeling tool 11 to perform initial, automated labeling of the target data before actual manual labeling begins, generating an "initial labeling result." The target data received by the data pre-labeling tool 11 is one of multiple unlabeled data sets in a specific data labeling project.

[0033] In this specification, the initial annotation result generated by the automatic annotation of target data using the data pre-labeling tool 11 is referred to as the first annotation result. Furthermore, the data pre-labeling tool 11 may include, but is not limited to, a data annotation model. For example, the data annotation model may be obtained by further fine-tuning and training a pre-trained Large Language Model (LLM) using an labeled dataset to enable it to learn to perform data annotation tasks. That is, the data annotation model can be a Large Language Model (LLM). When the data pre-labeling tool 11 receives target data to be labeled, it can first read suitable prompt words from the prompt word repository according to the data annotation project to which the target data belongs. Then, it inputs the target data and the read prompt words into the data annotation model, which performs automatic annotation processing on the target data based on the prompt words, outputting the first annotation result for this target data. The first annotation result is then used in subsequent manual annotation steps, providing a starting point for subsequent manual annotation, reducing manual workload, and improving the quality and efficiency of data annotation.

[0034] 3) Manual annotation After the target data is pre-annotated using the data pre-annotation tool 11, the first annotation result output by the tool is presented to the corresponding annotator. The annotator then corrects and / or supplements this first annotation result, or confirms its accuracy. "Correction" includes, but is not limited to: correcting errors (e.g., the annotator needs to correct any errors in the first annotation result) and adjusting the format (ensuring the annotation format conforms to specifications). "Supplementation" includes, but is not limited to: adding omissions (if some content that should be annotated is missing, the annotator manually adds it) and refining the annotation (providing more detailed annotation information for complex situations). In other words, the annotator needs to determine whether the first annotation result is accurate and complete, and modify it according to actual needs.

[0035] The manual annotation result formed after the annotator corrects and / or supplements the first annotation result is called the second annotation result.

[0036] 4) Annotation results feedback Here, "annotation result backflow" refers to backflowing the first and second annotation results output in 2) and 3) above to the Case-Shot centralized storage, so as to carry out comprehensive analysis of annotation results in the future, and to ensure that the data pre-annotation tool 11 can continuously learn and improve after updating the corresponding prompt words in the prompt word repository 10.

[0037] The Case-Shot set can be understood as a structured database. Each annotation record stored in the Case-Shot set includes a target data and its corresponding first and second annotation results. Therefore, the Case-Shot set can also be called an annotation record set.

[0038] For example, suppose the target data to be labeled is "**Company released X-T15 in 2003", then a labeling record for this target data in the Case-Shot set could be as follows: The data “**Company released X-T15 in 2003”: First annotation result “{Organization:**Company}, {Time “2003”, {Product: X-T15}”: Second annotation result “{Organization:**Company}, {Time “2003”, {Product: X-T15}, {Version number: 15}”.

[0039] Furthermore, in this specification, the Case-Shot set is publicly accessible via MCP (Multi-Channel Programming) and can be accessed by annotation result comprehensive analysis models (such as those built based on Large Language Models (LLM)). For example, it can be accessed periodically according to a preset reading cycle to obtain the first and second annotation results corresponding to multiple target data sets. By performing difference analysis on the first and second annotation results corresponding to these multiple target data sets, the corresponding prompt words in the prompt word repository can be iterated and optimized based on the annotation difference information obtained from the difference analysis. For a detailed description of the iterative optimization of prompt words, please refer to the prompt word update-related content described in section 5) below.

[0040] 5) Comprehensive analysis of annotation results and updating of prompt words Here, the comprehensive analysis of annotation results refers to using a comprehensive analysis model of annotation results (such as LLM) to perform a difference analysis on the first and second annotation results corresponding to each of the multiple target data read from the Case-Shot set, so as to determine whether the corresponding prompt words in the prompt word repository 10 need to be optimized and updated, and how to optimize and update them, based on the obtained annotation difference analysis information.

[0041] In practice, the comprehensive analysis model of annotation results can determine the quality of data annotation processing when using the data annotation model to perform automatic data annotation based on the obtained annotation difference information. This data annotation processing quality can also be referred to as pre-annotation quality or pre-annotation effect. Furthermore, the quality of data annotation processing can be measured by, but is not limited to, at least one of the following indicators: the modification rate of the first annotation result, the adoption rate of the first annotation result (also known as the confirmation rate), and the error types of the first annotation result (such as omissions, mislabeling, etc.). Further, based on the pre-annotation effect, it can be determined whether the corresponding prompt words in the prompt word repository need to be optimized and updated, and if so, the optimization and update of the corresponding prompt words in the prompt word repository can be triggered. The optimized and updated prompt words can be applied to the data annotation process of the next batch of data, thereby further improving the quality and efficiency of data annotation in the future.

[0042] For example, if the modification rate of the first annotation result is relatively low (e.g., less than 10%) and / or the adoption rate of the first annotation result is relatively high (e.g., greater than or equal to 90%), it indicates that the annotation quality of automatic data annotation using the data annotation model is high, the corresponding prompt words are effective, and no optimization of the prompt words is needed. However, if the modification rate of the first annotation result is relatively high (e.g., greater than or equal to 30%) and / or the adoption rate of the first annotation result is relatively low (e.g., less than 30%), it indicates that the annotation quality of automatic data annotation using the data annotation model is low, the prompt words may have problems, and the corresponding prompt words need to be optimized. When it is determined that the corresponding prompt words need optimization, for example, the prompt words in the prompt word repository can be optimized and updated based on the annotation error type of the first annotation result. For example, suppose that after analyzing the differences between the first and second annotation results corresponding to the most recent 100 mobile phone product review data, it is concluded that the main annotation error type in the process of automatic data annotation using the data annotation model (i.e., the pre-annotation process) is: omitting specific product models. Based on this type of labeling error, the original corresponding prompt in the prompt word repository, "Please extract the mobile phone product names mentioned in the following user comments and label the sentiment," will be updated to "Please extract the complete mobile phone product names (such as 'Xiaomi 14', 'iPhone 15 Pro') from the following user comments and further identify the specific models (such as '14', '15')."

[0043] In the above, the comprehensive analysis model for annotation results can be, for example, a pre-trained Large Language Model (LLM) that has been further fine-tuned and trained using a corresponding sample dataset to learn how to perform the task of analyzing annotation result differences. That is, the comprehensive analysis model for annotation results can be, but is not limited to, a Large Language Model.

[0044] Furthermore, for the Case-Shot set, the read operation is triggered only when preset read conditions are met. These preset read conditions include, but are not limited to, at least one of the following: reaching a preset read cycle (e.g., read once every 24 hours / 7 hours), or the cumulative number of labeled records in the Case-Shot set reaching a preset threshold. Here, the cumulative number of labeled records in the Case-Shot set reaching the preset threshold means that the number of newly added labeled records in the Case-Shot set since the last read has reached the preset threshold (e.g., 50, 100, or 200 records). By setting these read conditions, frequent and invalid access to the Case-Shot set can be avoided, effectively reducing computational resource consumption. Simultaneously, it ensures that the comprehensive analysis task of the labeled results is launched under the premise of sufficient data foundation and a reasonable time window, effectively improving the accuracy of prompt word optimization.

[0045] The technical architecture mentioned above is based on a server-side and client-side implementation. See also... Figure 1 As shown, the data pre-annotation tool 11 (specifically, the data annotation model) and the annotation result comprehensive analysis model in the technical architecture can be deployed on the server side. The server side can be a server, server cluster, virtual server, or cloud, etc. Manual annotation can be completed by annotators through a client. The client is the front-end interactive device of the data annotation system running on a terminal device. The terminal device can be, but is not limited to, smartphones, smart wearable devices, tablets, laptops, desktop computers, etc. The front-end interactive device of the data annotation system can take the form of an application or a web interface, which is configured to provide a graphical user interface (GUI) to display the target data to be annotated and the corresponding first annotation result (the initial annotation result generated using the text pre-annotation tool) to the annotator, and provide interactive controls to support the annotator in viewing, modifying, and confirming the first annotation result, and generating the final second annotation result (the manual annotation result).

[0046] thus, Figure 2 This specification also illustrates a data processing system (specifically a data annotation system) according to an embodiment, which includes a server 100 and a client 200. Server 100 is used to obtain target data; the target data is one of multiple unlabeled data included in a data annotation project; it calls a prompt word adapted to the data annotation project; the prompt word is used to guide the data annotation model to annotate the input target data; the target data and the prompt word are input into the data annotation model to obtain the first annotation result of the target data; Client 200 is used to display the first annotation result of the target data; in response to the manual annotation operation performed by the annotator on the target data based on the first annotation result, obtain the second annotation result of the target data; and send the second annotation result to the server 100. The server 100 is further configured to associate and store the target data with the first annotation result and the second annotation result; perform difference analysis on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; and optimize the prompt words based on the annotation difference information.

[0047] In addition to the above, the data processing system may also include: a prompt word repository, etc. The prompt word repository is used to store multiple prompt words. A prompt word is a prompt designed for the data annotation requirements of a data annotation project.

[0048] The specific implementation of the functions of server 100, client 200, and prompt word repository will be described in detail in the following method embodiments or in the relevant content of the aforementioned technical architecture, and will not be elaborated here.

[0049] The technical solutions provided in this specification will be described below by way of method embodiments.

[0050] Figure 3 This diagram illustrates a data processing method according to an embodiment of this specification. Some steps in this method (such as steps 102, 104, 106, 1010, 1012, and 1014) can be executed by a server in the aforementioned system, while the remaining steps (such as step 108) can be executed by a client in the aforementioned system. See also... Figure 3 As shown, the data processing method includes the following steps: 102. Obtain target data, wherein the target data is one of multiple data to be labeled contained in a data labeling project; 104. Invoke prompt words that are compatible with the data annotation project; the prompt words are used to guide the data annotation model to perform data annotation processing on the input target data; 106. Input the target data and the prompt words into the data annotation model, and output the first annotation result of the target data; 108. In response to the manual annotation operation performed by the annotator on the target data based on the first annotation result, a second annotation result for the target data is obtained; 1010. The target data, the first annotation result, and the second annotation result are stored together in a linked manner; 1012. Perform a difference analysis on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; 1014. Optimize the prompt words based on the annotation difference information.

[0051] In the aforementioned 102, data annotation projects can include, but are not limited to, medical data annotation projects, intelligent customer service data annotation projects, and 3D point cloud data annotation projects. Different data annotation projects contain multiple datasets to be annotated, and the data types can be text, images, audio, video, etc. Furthermore, the data annotation requirements for different data annotation projects often differ. Based on the data annotation requirements of different data annotation projects, different prompts can be designed in advance for each project to guide the data annotation model in performing automated annotation processing on the data to be annotated within the project.

[0052] Different prompt words designed in advance for different data annotation projects can be stored in a prompt word repository for later retrieval. Specifically, each prompt word stored in the prompt word repository is associated with a corresponding project identifier (such as project name or project number). In this solution, the prompt words in the prompt word repository will be automatically optimized and updated. For details on the implementation of prompt word optimization and updates, please refer to the relevant content below related to steps 1012 and 1014.

[0053] Furthermore, the acquired target data is one of multiple unlabeled data sets within a specific data annotation project. Based on the project identifier associated with the target data, prompt words with the same project identifier can be retrieved from a prompt word repository to guide the data annotation model in performing automatic annotation processing on the input target data.

[0054] Therefore, in a specific implementable solution, step 104 above, "calling the prompt word that matches the data annotation item," may include: 1041. Retrieve prompt words from the prompt word repository that are associated with the project identifier of the data-annotated project.

[0055] This prompt word repository stores multiple prompt words, and each prompt word is associated with a project identifier for a data annotation project. Furthermore, the prompt word repository can be publicly accessible, for example, via the MCP protocol.

[0056] For a detailed description of the prompt word repository, please refer to the aforementioned resources. Figure 1 The relevant content described.

[0057] Furthermore, the target data and the obtained prompt words are input into the data annotation model, which automatically annotates the target data based on the prompt words, thereby outputting the first annotation result of the target data (i.e., the initial annotation result).

[0058] In practice, the data annotation model could be, for example, a trained Large Language Model (LLM) with data annotation capabilities. For a detailed description of data annotation models, please refer to [link to relevant documentation / references]. Figure 1 The description includes relevant content. Furthermore, the data annotation model can first determine the target object to be annotated (such as product and product model, emotion, object, etc.) based on prompt words, and then extract or classify information from the target data based on this target object to achieve automatic annotation of the target data, thereby outputting the first annotation result of the target data.

[0059] That is, in a specific implementable solution, step 106 above, "inputting the target data and the prompt words into the data annotation model and outputting the first annotation result of the target data," may include: 1062. Input the target data and the prompt words into the data annotation model, and have the data annotation model execute: 10622. Based on the prompt words, determine the target objects that need to be labeled; 10624. Based on the target object, perform annotation processing on the target data to generate and output the first annotation result of the target data.

[0060] Based on the above, this embodiment designs an open prompt word repository to provide prompt words for the data annotation model while supporting the adaptive dynamic adjustment and updating of these prompt words. Ultimately, this enables the data annotation model to automatically annotate target data using dynamically adjusted and optimized prompt words, rather than relying on manual resetting of prompt words. Furthermore, by using dynamically adjustable prompt words to guide the data annotation model in the initial automatic data annotation, an automated and efficient annotation process is achieved. Additionally, the workload of manual annotation can be retrieved during the subsequent manual annotation stage, making the entire data annotation process more efficient.

[0061] The initial annotation results for the aforementioned target data will be displayed on the client interface. Specifically, this could be triggered by an annotator viewing the initial annotation results for the target data via the client interface. The target data can be displayed simultaneously with the initial annotation results. After reviewing the initial annotation results, the annotator can modify or confirm them as needed. Modification operations include correcting and / or supplementing the initial annotation results. Confirmation operations indicate that the current initial annotation results are correct. If the annotator modifies the initial annotation results, the modified initial annotation results will be designated as the second annotation results. If the annotator confirms the initial annotation results, the initial annotation results will be directly used as the second annotation results.

[0062] Based on the above, in one possible implementation, step 108, "in response to the manual annotation operation performed by the annotator on the target data based on the first annotation result, to obtain the second annotation result of the target data," may include: 1082. Display the first annotation result on the client interface.

[0063] 1084. In response to the annotator's modification operation on the first annotation result, the modified first annotation result is determined as the second annotation result of the target data; or, in response to the annotator's confirmation operation on the first annotation result, the first annotation result is directly used as the second annotation result of the target data.

[0064] For example, suppose the target data is: "**One 15 takes very clear photos, but the Pro version's telephoto lens is even better," and the first annotation result for this target data is: "**One 15" is labeled as "Product Model." After reviewing this first annotation result, the annotator finds a missing label and performs the corresponding modification operation: adding "Pro" to the label as "Product Model," and merging it with "**One 15" into the complete entity "**One 15 Pro." After the modification is completed, the annotator clicks "Submit," and the second annotation result "Product Model: **One 15 Pro" will be generated here.

[0065] Furthermore, the first and second annotation results of the aforementioned target data are back-stored to a set of annotation records, forming a knowledge set that supports the analysis of annotation result differences, for use in updating and optimizing the prompt words. Based on this, in one possible implementation, the aforementioned 1010 "associatively storing the target data, the first annotation result, and the second annotation result" may include the following steps: 10102. The target data, the first annotation result, and the second annotation result are associated and stored in an annotation record set. The annotation record set includes multiple annotation records, and each annotation record contains a target data and the corresponding first and second annotation results.

[0066] Here, the labeled record set is the Case-Shot set described in the other embodiments above. The Case-Shot set (i.e., the labeled record set) can be made publicly available, for example, via the MCP protocol.

[0067] For a detailed description of the labeled record set, please refer to [link to relevant documentation]. Figure 1 The description relates to the Case-Shot set.

[0068] This embodiment integrates resources from different stages (including the annotation record set mentioned here and the aforementioned prompt word repository) into resources that are made available to the outside world using the MCP protocol. This allows the resources to be made available to the corresponding large language models (such as data annotation models and annotation result comprehensive analysis models built based on large language models). This enables the large language models to flexibly select the required resource content for relevant analysis and processing, ultimately achieving continuous adjustment and optimization of prompt words and enhancing the adaptability of data annotation capabilities.

[0069] Furthermore, the "difference analysis of the first annotation result and the second annotation result corresponding to each of the stored target data" in step 1012 above may include the following steps: 10122. When a preset reading condition is detected, the plurality of labeled records are read from the labeled record set; 10124. Based on the multiple labeled records, trigger the step of performing difference analysis on the first and second labeled results corresponding to each of the multiple target data.

[0070] In the above 1022, satisfying the preset reading conditions may include, but is not limited to, at least one of the following: reaching a preset reading period, or the number of accumulated annotation records in the annotation record set being greater than or equal to a preset number threshold.

[0071] In the above 10124, a trained annotation result comprehensive analysis model can be used to perform difference analysis on the first and second annotation results corresponding to multiple target data, thereby outputting annotation difference information. The annotation result comprehensive analysis model can be, for example, a Large Language Model (LLM) with annotation result difference analysis capabilities.

[0072] For a detailed description of the implementation of steps 10122 and 10124 above, please refer to other embodiments. Figure 1 The relevant content described.

[0073] After obtaining the annotation difference information through step 1012 above, the annotation error types that exist when performing annotation processing using the data annotation model can be analyzed based on the annotation difference information, and then the prompt words can be optimized based on the annotation error types.

[0074] That is, in one feasible implementation, step 1014, "optimizing the prompt words based on the annotation difference information," may include: 10142. Based on the annotation difference information, determine the data annotation processing quality of the data annotation model; 10144. Optimize the prompt words based on the quality of the data annotation and processing.

[0075] Here, the metrics for measuring the quality of data annotation processing include, but are not limited to: the modification rate of the first annotation result and the types of annotation errors (such as omissions).

[0076] Furthermore, a specific implementation of step 10144 above may include: when it is determined that the prompt word needs to be optimized based on the modification rate, optimizing the prompt word according to the labeling error type.

[0077] For a detailed description of the implementation of steps 10142 and 10144 above, please refer to other embodiments. Figure 1 The description relates to the content related to "5) Comprehensive analysis of annotation results and update of prompt words".

[0078] In summary, this embodiment achieves feedback loops from the automatic annotation results (first annotation result) and the manual annotation results (second annotation result) of the target data using the data annotation model to the annotation record set. This ensures that the model can learn from past annotation data (including both automatic and manual annotation results of the target data) and continuously adaptively adjust the prompts used to guide the data annotation model in performing annotation processing. This improves the quality of automatic data annotation, reduces manual workload, and enhances overall data annotation efficiency. Furthermore, the adaptive adjustment of prompts also enhances the responsiveness to changes in the data to be annotated and new data annotation requirements.

[0079] Besides optimizing suggestion words based on annotation differences, other information can also be used to optimize and update suggestion words. For example, when the data annotation requirements of a data annotation project change, in order to ensure that suggestion words accurately reflect the latest data annotation goals, the corresponding suggestion words can be adaptively adjusted according to the changed data annotation requirements.

[0080] Based on this, the method provided in this embodiment may further include the following steps: 1016. When it is detected that the data annotation requirements of the data annotation project have been updated, the prompt words are optimized according to the updated data annotation requirements.

[0081] For example, suppose a data annotation project originally only required extracting the "product model". Later, the project administrator decided to add "color" and "storage capacity" as data analysis dimensions. Therefore, the data annotation requirement is updated to: extract the "product model" and annotate both "color" and "capacity". For this updated requirement, the original prompt "Please extract the product model from the comment data" will be optimized and updated. The optimized prompt could be "Please extract the product model, color, and storage capacity from the comment data".

[0082] Of course, the aforementioned optimization of prompt words based on the annotation difference information between the first and second annotation results also indirectly achieves the optimization of prompt words based on the updated data annotation requirements. This is because the updated content of the data annotation requirements is usually reflected in the second annotation results generated by manual annotation. Therefore, by analyzing the annotation difference information, we can identify the new annotation requirements in actual execution and optimize the prompt words accordingly, thereby achieving automatic adaptation to changes in data annotation requirements.

[0083] Furthermore, the method provided in this embodiment may also include the following steps: 1018. Store the optimized prompt words in the prompt word repository.

[0084] Here, when storing optimized suggestion words into the suggestion word repository, a relationship is established between the optimized suggestion words and the corresponding data annotation project identifier to ensure accurate retrieval in the future. Simultaneously, older versions of suggestion words that have been replaced in the suggestion word repository can be deleted to avoid version redundancy and free up storage resources.

[0085] Furthermore, considering that different target data in the same data annotation project may originate from different channels (such as from the internet or from textbooks, product manuals, etc.), and that data quality varies, the annotation performance will also differ when using the same prompt word-driven data annotation model to perform data annotation processing on target data with varying quantities and qualities. Specifically, when using the same prompt word-driven data annotation model to perform data annotation processing on different input target data, the annotation performance is often better on high-quality target data and worse on low-quality target data. To address this problem of "using one word for multiple purposes" (i.e., using the same prompt word-driven data annotation model to perform data annotation on different target data), which cannot adapt to changes in data quality and leads to a decline in overall annotation performance, this embodiment also introduces a data quality assessment mechanism to dynamically optimize the prompt word based on the data quality of the target data, thereby adapting to input data of different qualities.

[0086] Therefore, before performing step 106 above, which involves "inputting the target data and the prompt words into the data annotation model," the method provided in this embodiment may further include the following steps: S1. Perform quality analysis on the target data to obtain the data quality of the target data; S2. Based on the data quality, optimize the prompt words, input the target data and the optimized prompt words into the data annotation model, and output the first annotation result of the target data.

[0087] In S1 above, the data quality of the target data can be analyzed based on its source. For example, if the target data is obtained from the internet, it often contains typos, colloquial grammar, and misspellings, resulting in lower quantity and quality. If the target data is obtained from standardized resources such as product manuals, it is usually free of typos, grammatically correct, uses standard terminology, and has a consistent format, resulting in higher quantity and quality. Alternatively, the target data can be directly assessed for quality to determine its overall quality. The data quality can be measured by at least one of the following indicators: typo rate, number of grammatical errors, information completeness, noise ratio, and grammatical correctness. The noise ratio refers to the proportion of noisy content (such as emoticons, advertisements, garbled text, links, etc.) to the total number of characters or terms in the target data.

[0088] In S2 above, if the target data has low data quality, the optimization strategy for prompt words can be to enhance robustness and fault tolerance; if the target data has high data quality, the optimization strategy for prompt words can be to emphasize accuracy and format consistency, etc.

[0089] For example, suppose the prompt word P stored in the prompt word repository for a certain text annotation item is "Please extract the product model from the text", and the text data to be annotated obtained from this text annotation item is "**one 15 Pro Max equipped with A17 chip". If the analysis shows that the data quality of the text data is high, the optimized prompt word P could be, for example, "Please extract the complete product model from the text, including suffixes such as Pro, Max, Ultra, etc. The output must be consistent with the original text and must not be abbreviated or omitted." If the analysis shows that the data quality of the text data is low, the optimized prompt word could be, for example, "Please extract possible product models from the text, even if there are typos, abbreviations, or incomplete expressions. Please pay special attention to suffixes such as 'pro', 'max', 'plus', etc., and ignore noisy content such as emojis and advertising slogans." As can be seen, this embodiment dynamically optimizes prompt words based on the evaluated data quality of the target data, enabling the data annotation model to adapt to input data of different qualities, thereby improving the stability and accuracy of the annotation effect.

[0090] For specific implementation details of the steps described above in this embodiment, please refer to the relevant content in other embodiments, which will not be repeated here. Furthermore, the method provided in this embodiment may also include some steps disclosed in other embodiments, which can also be found in the relevant content in other embodiments, and will not be repeated here.

[0091] The above text combined Figure 3 Specific embodiments of the embodiments described herein have been described. It should be noted that other embodiments are within the scope of the appended claims. Furthermore, in some cases, the actions or steps described in the specification may be performed in a different order than those shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0092] The apparatus embodiments corresponding to the various method embodiments provided in this specification are described below.

[0093] Figure 4 A schematic diagram of the structure of a data processing apparatus provided in an exemplary embodiment of this specification is shown. For example... Figure 4 As shown, the device includes: an acquisition module 21, a calling module 22, an input module 23, a response module 24, an associated storage module 25, an analysis module 26, and an optimization module 27. Among them, The acquisition module 21 is used to acquire target data; the target data is one of multiple data to be labeled contained in a data labeling project; The calling module 22 is used to call prompt words that are compatible with the data annotation project; the prompt words are used to guide the data annotation model to perform data annotation processing on the input target data; Input module 23 is used to input the target data and the prompt words into the data annotation model to obtain the first annotation result of the target data; Response module 24 is used to respond to the manual annotation operation performed by the annotator on the target data based on the first annotation result, and obtain the second annotation result of the target data; The associated storage module 25 is used to associate and store the target data, the first annotation result, and the second annotation result together. Analysis module 26 is used to perform difference analysis on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; The optimization module 27 is used to optimize the prompt words based on the annotation difference information.

[0094] Optionally, the optimization module 27, when optimizing the prompt words based on the annotation difference information, is specifically used to: determine the data annotation processing quality of the data annotation model based on the annotation difference information; and optimize the prompt words based on the data annotation processing quality. The metrics for measuring the data annotation processing quality include: the modification rate of the first annotation result and the annotation error type. Furthermore, when optimizing the prompt words based on the data annotation processing quality, the optimization module 27 is specifically used to: when it is determined based on the modification rate that the prompt words need optimization, perform optimization of the prompt words based on the annotation error type.

[0095] Optionally, the optimization module 27 described above can also be used to: when the data annotation requirements of the data annotation project are updated, optimize the prompt words according to the updated data annotation requirements.

[0096] Optionally, when the acquisition module 21 is used to acquire prompt words that are compatible with the data annotation project, it is specifically used to: acquire prompt words associated with the project identifier of the data annotation project from the prompt word repository. The prompt word repository stores multiple prompt words, and each prompt word is associated with the project identifier of a data annotation project; the optimized prompt words are stored in the prompt word repository for subsequent use.

[0097] Optionally, the response module 24, when responding to the manual annotation operation performed by the annotator on the target data based on the first annotation result to obtain the second annotation result of the target data, is specifically used to: in response to the annotator's modification operation on the first annotation result, determine the modified first annotation result as the second annotation result; or, in response to the annotator's confirmation operation on the first annotation result, use the first annotation result as the second annotation result.

[0098] Optionally, the aforementioned associated storage module 25, when used to associate and store the target data, the first annotation result, and the second annotation result, is specifically used to: associate and store the target data, the first annotation result, and the second annotation result into an annotation record set. The annotation record set includes multiple annotation records, and each annotation record contains a target data and the corresponding first and second annotation results. Furthermore, the aforementioned analysis module 26, when used to perform difference analysis on the first and second annotation results corresponding to each of the stored multiple target data, is specifically used to: when a preset reading condition is detected, read the multiple annotation records from the annotation record set; based on the multiple annotation records, trigger the step of performing difference analysis on the first and second annotation results corresponding to each of the multiple target data; the preset reading condition includes at least one of the following: reaching a preset reading period, or the number of accumulated annotation records in the annotation record set being greater than or equal to a preset number threshold.

[0099] Optionally, the target data can be any of the following: text, image, audio, video, or 3D point cloud data.

[0100] It should be noted that the above-mentioned devices can implement the technical solutions described in the corresponding method embodiments. The specific implementation principles of each module or unit can be found in the relevant content of the corresponding method embodiments, and will not be elaborated further here. Furthermore, for ease of description, the above devices are described by function as various modules or units. Of course, when implementing one or more of this specification, the functions of each module or unit can be implemented in one or more software and / or hardware, or a module that implements the same function can be implemented by a combination of multiple sub-modules or sub-units, etc. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0101] Furthermore, embodiments of this specification also provide an electronic device. For example... Figure 5As shown, the electronic device 900 includes a memory 91 and a processor 92.

[0102] The aforementioned memory 91 can be implemented by at least one volatile or non-volatile storage device of any type, or a combination thereof, such as Static Random-Access Memory (SRAM), Electrically Erasable Programmable Read Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Furthermore, the memory, wholly or partially, can be integrated with the processor. The memory can contain both removable and non-removable components.

[0103] The processor 92 described above may include one or more general-purpose processors and / or special-purpose processors.

[0104] Furthermore, memory 91 may contain a non-transitory computer-readable medium storing executable program instructions 912 (e.g., compiled or uncompiled program logic and / or machine code). Processor 92 is capable of executing the program instructions 912 stored in memory to implement any methods, processes, or functions disclosed in this specification and / or the accompanying drawings. Additionally, execution of program instructions 912 by processor 92 may result in processor using corresponding data 911.

[0105] For example, the program instructions 912 described above may include an operating system 9122 (e.g., an operating system kernel, device drivers, and / or other modules) installed on the electronic device 900, and one or more application programs 9121 (e.g., a browser, social media application, or game application). Similarly, the data 911 described above may include operating system data 9112 and application data 9111. The operating system data 9112 is primarily accessible to the operating system 9122, while the application data 9111 is primarily accessible to one or more application programs 9121. The application data 9111 may reside in a file system visible or hidden from the user of the electronic device 900.

[0106] Application 9121 can communicate with operating system 9122 through one or more application programming interfaces (APIs). These APIs facilitate application 9121 in reading and / or writing application data, transmitting or receiving information via communication components, and receiving or displaying information on the user interface. In some terms, application 9121 may be simply referred to as "app". Furthermore, application 9121 can be downloaded to the electronic device through one or more online application stores or app markets. However, application 9121 can also be installed on electronic device 400 in other ways, such as through a web browser or a physical interface on electronic device 900 (e.g., a USB port). Furthermore, such as Figure 5 As shown, the electronic device also includes other components such as a communication component 93, a display 94, a power supply component 95, an audio component 96, and a user interface 97. Figure 5 The diagram only shows some components and does not mean that the electronic device 900 includes only these components. Figure 5 The components shown. Additionally... Figure 5 The components within the dashed box are optional, not mandatory, and their specific requirements depend on the product form of the electronic device 900. The electronic device 900 in this embodiment can be a terminal device such as a desktop computer, laptop computer, smartphone, or IoT device; it can also be a server-side device such as a conventional server, cloud server, or server array; or it can be an integrated device combining terminal and server-side devices. If the electronic device 900 in this embodiment is implemented as a terminal device such as a desktop computer, laptop computer, or smartphone, it may include... Figure 5 The components within the dashed box; if the electronic device 900 in this embodiment is implemented as a server-side device such as a conventional server, cloud server, or server array, then it may not include... Figure 5 The component within the dashed box.

[0107] The aforementioned communication component 93 is configured to facilitate wired or wireless communication between the device housing the communication component and other devices. The device housing the communication component 93 can access wireless networks based on communication standards, such as 2G, 3G, 4G / LTE, 5G, or combinations thereof. In one exemplary embodiment, the communication component 93 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. Specifically, the communication component 93 includes a communication interface that enables the electronic device 900 to communicate with other devices, access networks, and transmission networks via analog or digital modulation. For example, the communication interface may include a chipset and antenna for wireless communication with a radio access network or access point. Furthermore, the communication interface can be a wired interface such as Ethernet, Token Ring, or a USB port, or a wireless interface such as Wi-Fi, Bluetooth, Global Positioning System (GPS), or a wide-area wireless interface (e.g., WiMAX or LTE). Of course, the communication interface can also support other forms of physical layer interfaces and standard or proprietary communication protocols. The communication interface may also include multiple physical communication interfaces, such as Wi-Fi, Bluetooth, and wide-area wireless interfaces.

[0108] The aforementioned display 94 includes a screen, which may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a Touch Panel, the screen can be implemented as a touchscreen to receive input signals from the user. The Touch Panel includes one or more touch sensors to sense touches, swipes, and gestures on the Touch Panel. The touch sensors can sense not only the boundaries of touch or swipe actions but also the duration and pressure associated with the touch or swipe operation.

[0109] The power supply component 95 provides power to various components of the device in which it resides. The power supply component 95 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device in which the power supply component resides.

[0110] The aforementioned audio component 96 can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) configured to receive external audio signals when the device containing the audio component is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals can be further stored in memory or transmitted via a communication component. In some embodiments, the audio component also includes a speaker for outputting audio signals.

[0111] The user interface 97 described above includes receiving user input and providing output to the user. Therefore, the user interface 97 may include input components such as a keypad, keyboard, touch-sensitive or presence-sensitive panel, computer mouse, trackball, joystick, microphone, still camera, and video camera, and output components such as a display screen (which may be combined with a touch-sensitive panel), CRT, LCD, LED, display using DLP technology, printer, and other similar devices known or developed in the future. The user interface 97 may also generate auditory output via speakers, speaker jacks, audio output ports, audio output devices, headphones, and other similar devices known or developed in the future. In some embodiments, the user interface 97 may include software, circuitry, or other forms of logic capable of transmitting and receiving data from external user input / output devices. Additionally or alternatively, the electronic device 900 may support remote access from other devices via a communication interface or another physical interface (not shown). The user interface 97 may be configured to receive user input, the position and movement of which may be indicated by indicators or cursors described herein. The user interface 97 may also be configured as a display device for rendering or displaying text fragments.

[0112] Accordingly, embodiments of this specification also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, enables the processor to implement the steps in the above-described method embodiments. The computer-readable storage medium includes volatile or non-volatile or a combination thereof, and can be removable or non-removable. Examples of computer-readable storage media include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technologies, CD-ROM, Digital Video Disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium. Furthermore, embodiments of this specification also provide a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed in a computer, it causes the computer to perform actions such as... Figure 3 The method described.

[0113] This specification also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implements... Figure 3 The method described.

[0114] Those skilled in the art will recognize that the functions described in the various embodiments disclosed in this specification in one or more of the examples above can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0115] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of the multiple embodiments disclosed in this specification. It should be understood that the above descriptions are merely specific implementations of the multiple embodiments disclosed in this specification and are not intended to limit the protection scope of the multiple embodiments disclosed in this specification. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the multiple embodiments disclosed in this specification should be included within the protection scope of the multiple embodiments disclosed in this specification.

Claims

1. A data processing method, characterized in that, include: Obtain the target data; The target data is one of multiple unlabeled data sets included in a data labeling project; Invoke prompt words that are compatible with the data annotation project; the prompt words are used to guide the data annotation model to perform data annotation processing on the input target data; The target data and the prompt words are input into the data annotation model, and the first annotation result of the target data is output. In response to the manual annotation operation performed by the annotator on the target data based on the first annotation result, a second annotation result for the target data is obtained; The target data, the first annotation result, and the second annotation result are stored together in association; A difference analysis is performed on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; Based on the annotation difference information, optimize the prompt words.

2. The method according to claim 1, characterized in that, Optimize the prompt words based on the annotation difference information, including: Based on the annotation difference information, the data annotation processing quality of the data annotation model is determined; The prompt words are optimized based on the quality of the data annotation and processing.

3. The method according to claim 2, characterized in that, The metrics for measuring the quality of the data annotation processing include: the modification rate of the first annotation result and the types of annotation errors; Furthermore, based on the quality of the data annotation and processing, the prompt words are optimized, including: When it is determined that the prompt word needs to be optimized based on the modification rate, the prompt word is optimized according to the annotation error type.

4. The method according to any one of claims 1 to 3, characterized in that, Also includes: When an update to the data annotation requirements of the data annotation project is detected, the prompt words are optimized according to the updated data annotation requirements.

5. The method according to any one of claims 1 to 3, characterized in that, Before inputting the target data and the prompt words into the data annotation model, the method further includes: The target data is subjected to quality analysis to obtain the data quality of the target data; Based on the data quality, the prompt words are optimized so that the target data and the optimized prompt words are input into the data annotation model.

6. The method according to any one of claims 1 to 3, characterized in that, Obtaining prompt words that match the data annotation items, including: Retrieve prompt words from the prompt word repository that are associated with the project identifiers of the data-annotated projects; The prompt word repository stores multiple prompt words, and each prompt word is associated with a project identifier of a data annotation project; the optimized prompt words will be stored in the prompt word repository for subsequent use.

7. The method according to any one of claims 1 to 3, characterized in that, In response to a manual annotation operation performed by an annotator on the target data based on the first annotation result, a second annotation result for the target data is obtained, including: In response to the annotator's modification operation on the first annotation result, the modified first annotation result is determined as the second annotation result; or, In response to the annotator's confirmation of the first annotation result, the first annotation result is used as the second annotation result.

8. The method according to any one of claims 1 to 3, characterized in that, The target data, the first annotation result, and the second annotation result are associated and stored, including: The target data, the first annotation result, and the second annotation result are associated and stored in the annotation record set; The annotation record set includes multiple annotation records, and each annotation record contains a target data and a first annotation result and a second annotation result corresponding to the target data. Furthermore, a difference analysis is performed on the first annotation result and the second annotation result corresponding to each of the stored target data, including: When a preset reading condition is detected, the plurality of labeled records are read from the labeled record set; Based on the multiple labeled records, the step of performing a difference analysis on the first and second labeled results corresponding to each of the multiple target data is triggered; The preset reading conditions include at least one of the following: reaching a preset reading cycle, or the number of accumulated annotation records in the annotation record set being greater than or equal to a preset number threshold.

9. The method according to any one of claims 1 to 3, characterized in that, The target data can be any of the following: text, image, audio, video, or 3D point cloud data.

10. A data processing system, characterized in that, include: The server is used to obtain target data; the target data is one of multiple unlabeled data included in a data annotation project; it calls a prompt word that is compatible with the data annotation project; the prompt word is used to guide the data annotation model to annotate the input target data; the target data and the prompt word are input into the data annotation model to obtain the first annotation result of the target data; The client is used to display the first annotation result of the target data; In response to the manual annotation operation performed by the annotator on the target data based on the first annotation result, a second annotation result for the target data is obtained; The second annotation result is sent to the server. The server is also used to associate and store the target data with the first annotation result and the second annotation result; perform difference analysis on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; and optimize the prompt words based on the annotation difference information.

11. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire target data; The target data is one of multiple unlabeled data sets included in a data labeling project; The calling module is used to call prompt words that are compatible with the data annotation project; the prompt words are used to guide the data annotation model to annotate the input target data; The input module is used to input the target data and the prompt words into the data annotation model to obtain the first annotation result of the target data; The response module is used to respond to the manual annotation operation performed by the annotator on the target data based on the first annotation result, and obtain the second annotation result of the target data; An associated storage module is used to associate and store the target data, the first annotation result, and the second annotation result. The analysis module is used to perform difference analysis on the first annotation result and the second annotation result corresponding to each of the stored target data to obtain annotation difference information; An optimization module is used to optimize the prompt words based on the annotation difference information.

12. An electronic device, characterized in that, The method includes a memory and a processor, wherein the memory stores executable program instructions, and the processor executes the program instructions to implement the method of any one of claims 1 to 9.

13. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed in a computer, causes the computer to perform the method described in any one of claims 1 to 9.

14. A computer program product, characterized in that, The computer program product includes a computer program or instructions that, when executed by a processor, implement the method of any one of claims 1 to 9.

Images