Data labeling method, device and equipment and readable storage medium

Abstract

The invention relates to a data annotation method and device, equipment and a readable storage medium. The data annotation method comprises the following steps: firstly, inputting to-be-annotated data and cue words into a first pre-trained large language model, generating a first prediction data annotation result, analyzing the confidence coefficient of the first prediction data annotation result, and generating a first data annotation result based on a result conforming to a first confidence coefficient threshold value; and inputting a result which does not accord with the first confidence threshold value into a second pre-trained large language model for re-labeling, analyzing the confidence, and generating a second data labeling result. And performing manual annotation on the result which does not accord with the second confidence threshold, and finally combining the first, second and manual annotation results to determine a final annotation result. According to the method, through automatic annotation and confidence analysis of a multi-round pre-trained large language model, and in combination with manual annotation, low-confidence tasks are processed, so that the annotation accuracy and efficiency are remarkably improved, and the dependence on manual intervention is reduced.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and specifically, to a data annotation method, apparatus, device, and readable storage medium. Background Art

[0002] With the development of artificial intelligence and large language models (LLM), data labeling has become a key link in various artificial intelligence applications and model training. Data labeling is the basis for ensuring the accuracy of machine learning models and is widely used in natural language processing, image recognition, speech recognition and other fields. According to the type of annotator, data labeling is divided into two categories: manual labeling and machine labeling.

[0003] Manual labeling is to use humans as the labeling subject to check each element in the data set, which can ensure high accuracy. However, it is inefficient and easily affected by the subjective factors of the labeler and the external environment, resulting in inconsistency and errors in the labeling results.

[0004] To this end, existing data annotation methods still need to be improved. Summary of the invention

[0005] The present application aims to solve one of the technical problems in the related art at least to a certain extent. To this end, one purpose of the present application is to provide an efficient and accurate data annotation method.

[0006] Specifically, this application proposes the following technical solutions:

[0007] In the first aspect, the present application proposes a data labeling method. According to an embodiment of the present application, the method includes: inputting the data to be labeled and the prompt word into a first pre-trained large language model to determine the first predicted data labeling result; analyzing the first predicted data labeling result to determine the confidence of the first predicted data labeling result; determining the first data labeling result based on the first predicted data labeling result that meets the first confidence threshold; inputting the data to be labeled and the prompt word corresponding to the first predicted data labeling result that does not meet the first confidence threshold into a second pre-trained large language model to determine the second predicted data labeling result; analyzing the second predicted data labeling result to determine the confidence of the second predicted data labeling result; determining the second data labeling result based on the second predicted data labeling result that meets the second confidence threshold; manually labeling the data to be labeled corresponding to the second predicted labeling result that does not meet the second confidence threshold to determine the manual data labeling result; merging the first data labeling result, the second data labeling result and the manual data labeling result to determine the final labeling result of the data to be labeled.

[0008] In some examples of this application, the aforementioned method significantly improves the accuracy and efficiency of annotation and reduces the reliance on manual intervention by combining multiple rounds of automatic annotation and confidence analysis of pre-trained large language models with manual annotation to handle low-confidence tasks. This method can flexibly select different models for annotation, optimize the annotation process, improve the adaptability and generalization ability of the model in different fields, and effectively filter out low-confidence erroneous annotations to ensure the accuracy of the final annotation results, and is suitable for large-scale data annotation tasks.

[0009] In the second aspect, the present application proposes a data labeling device. According to an embodiment of the present application, the device includes: a first prediction data labeling module, which is used to input the data to be labeled and the prompt word into the first pre-trained large language model to determine the first prediction data labeling result; a first confidence determination module, which is used to analyze the first prediction data labeling result and determine the confidence of the first prediction data labeling result; a first data labeling result determination module, which is used to determine the first data labeling result based on the first prediction data labeling result that meets the first confidence threshold; a second prediction data labeling module, which is used to input the data to be labeled and the prompt word corresponding to the first prediction data labeling result that does not meet the first confidence threshold into the second pre-trained large language model. , determine the second prediction data labeling result; a second confidence determination module, used to analyze the second prediction data labeling result and determine the confidence of the second prediction data labeling result; a second data labeling result determination module, used to determine the second data labeling result based on the second prediction data labeling result that meets the second confidence threshold; a manual labeling module, used to manually label the data to be labeled corresponding to the second prediction labeling result that does not meet the second confidence threshold, and determine the manual data labeling result; a final labeling result output module, used to merge the first data labeling result, the second data labeling result and the manual data labeling result, and determine the final labeling result of the data to be labeled.

[0010] In some examples of the present application, the aforementioned device significantly improves the efficiency and accuracy of data annotation through multi-level automated processing and confidence analysis combined with manual annotation. The device uses multiple pre-trained large language models to gradually annotate data, which can effectively handle annotation tasks with different confidence levels, and further ensure the accuracy of annotation through manual annotation. The final annotation result is a fusion of data from multiple sources, which reduces the possibility of single model errors, is suitable for complex or large-scale data annotation tasks, reduces dependence on manual annotation, and improves the level of automation.

[0011] In a third aspect, the present application proposes a computer program product. According to an embodiment of the present application, the computer program product includes: computer instructions; when part or all of the computer instructions are run on a computer, the data labeling method described in the first aspect of the present application is executed.

[0012] In a fourth aspect, the present application proposes a computing device. According to an embodiment of the present application, the computing device includes: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program to implement the data labeling method as described in the first aspect of the present application.

[0013] In a fifth aspect, the present application proposes a computer-readable storage medium. According to an embodiment of the present application, the computer-readable storage medium stores computer instructions or programs, and when the computer instructions or programs are run on a computer, the data labeling method described in the first aspect of the present application is executed.

[0014] The aforementioned computer program product, computing device, and computer-readable storage medium automatically execute the data labeling method through computer instructions, thereby achieving efficient automation and improving labeling efficiency and accuracy. Secondly, the characteristics of the instructions make the above method highly consistent, stable, and reliable in different application scenarios.

[0015] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0017] Figure 1 A schematic diagram of the text data annotation process provided for the specific implementation method of this application;

[0018] Figure 2 A schematic diagram of a data annotation device provided for a specific implementation of the present application;

[0019] Figure 3 A schematic diagram of an electronic device provided for a specific embodiment of the present application;

[0020] Figure 4 A schematic diagram of the chemical literature fragment data annotation process provided in the embodiments of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0022] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In the present application, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two.

[0023] In this article, unless otherwise specified, "data annotation" refers to the process of classifying, marking or identifying raw data (such as text, images, audio, video, etc.) to provide structured information. In some examples of this application, data annotation is performed by using a multi-round progressive large language model and manual annotation collaboration.

[0024] Currently, there are machine learning methods for automatic labeling of large-scale data. However, current automatic labeling systems often fail to correctly handle complex or ambiguous data, especially when deep semantic understanding or domain expertise is required, and machine labeling performs worse than manual labeling. Moreover, the combination of manual and machine labeling is still not mature enough and lacks an effective collaboration mechanism. After the automatic labeling is completed, many labeling systems rely on manual review to correct the results, but this model easily leads to inefficiency and duplication of work. In addition, existing automatic labeling systems usually use a single model or fixed algorithm for labeling tasks, lack a multi-model collaborative optimization mechanism, and cannot fully utilize the advantages of different models to improve labeling accuracy.

[0025] To this end, the present application proposes a data annotation method, a data annotation device, a computer program product, a computing device, and a computer-readable storage medium. The following describes each of them:

[0026] Data annotation method

[0027] On the one hand, the present application proposes a data annotation method, which includes:

[0028] 1) Inputting the data to be labeled and the prompt word (Prompt) into the first pre-trained large language model to determine the first predicted data labeling result.

[0029] In some examples of the present application, the aforementioned data to be annotated is selected from text data, such as patents, documents, research reports, news articles, social media comments, emails, customer feedback, etc. In some preferred examples, the data to be annotated in the present application is chemical literature, patents or research reports.

[0030] The aforementioned prompt word refers to an instruction or question provided to a large language model during data annotation or model reasoning, which is intended to guide the model to generate a specific output or complete a task. The prompt word is usually described in natural language, containing clear task requirements or constraints, so that the model can accurately process the input data according to the prompt and generate the required results. For example, in text data annotation, the prompt word may be "mark all compound names in the text". In some examples of the present application, prompt words are designed based on the data to be annotated. The prompt word is optimized using standard data annotation results by active learning. Among them, the aforementioned standard data annotation results are obtained by manual annotation.

[0031] In some examples of the present application, the aforementioned first pre-trained large language model is selected from at least one of chatGLM2 and chatGLM3.

[0032] The inventors have found that selecting multiple different first pre-trained large language models can improve data annotation accuracy and adaptability to different data types. Selecting multiple identical first pre-trained large language models can be used to avoid computer hallucinations.

[0033] Exemplarily, the present method is described in detail below by taking chemical data to be annotated as an example:

[0034] (1) Examples of data to be annotated

[0035] A chemistry research report stated: "In this study, we synthesized new catalysts to improve the conversion rate of phenol. By using an improved palladium catalyst, the selectivity and yield of the reaction were significantly improved. The experimental results showed that the catalyst can maintain good activity even at lower temperatures."

[0036] (2) Examples of prompt words

[0037] Identify the catalyst type;

[0038] extracting reactants and products;

[0039] Determine reaction conditions (e.g., temperature, pressure);

[0040] (3) Example of the first pre-trained large language model used

[0041] Select models, such as chatGLM2 and chatGLM3, to process the above text data.

[0042] (4) Data labeling results

[0043] Catalyst type: modified palladium catalyst;

[0044] Reactants: phenol;

[0045] Product: Not directly mentioned (needs further inference or annotation);

[0046] Reaction conditions: lower temperature.

[0047] 2) Analyze the aforementioned first prediction data labeling result to determine the confidence level of the first prediction data labeling result.

[0048] In some examples of the present application, the confidence of the first prediction data annotation result is calculated in the following manner:

[0049] Con(S llm-i )=C(S llm-i ) / N;

[0050] Among them, Con(S llm-i ) represents the confidence level; S llm-i Represents the data annotation result of the i-th pre-trained large language model; Con(S llm-i ) indicates S llm-i The number of times a word appears in the answers of N pre-trained large language models; the number of N pre-trained large language models.

[0051] 3) Determine a first data labeling result based on the first predicted data labeling result that meets a first confidence threshold.

[0052] It is understandable that the first confidence threshold can be a value (such as 60%) or a range (such as 60%-80%). The first prediction data annotation result that meets the first confidence threshold is the first data annotation result, that is, at least part of the final output result.

[0053] In some examples of the present application, the aforementioned first data annotation result is the first predicted data annotation result with the highest confidence. In some examples, there may be multiple annotation results that meet the confidence threshold, and the first predicted data annotation result with the highest confidence is used as the first data annotation result.

[0054] 4) Inputting the data to be annotated and the prompt word corresponding to the first predicted data annotation result that does not meet the first confidence threshold into the second pre-trained large language model to determine the second predicted data annotation result.

[0055] In some examples of the present application, the aforementioned second pre-trained large language model is selected from at least one of LLaMA3-8b and chatGPT (3.5).

[0056] A second round of labeling is performed on the unlabeled data that does not reach the preset confidence threshold (such as erroneous data or uncertain results) through a more accurate pre-trained large language model to improve the accuracy of data labeling.

[0057] Example: Compound extraction from chemical literature

[0058] (1) First round of annotation

[0059] Data to be annotated: A chemical literature describing a new compound and its application.

[0060] Prompt words: Mark all the compounds mentioned in the text and their relevant properties (such as chemical formula, structure, name).

[0061] The first pre-trained large language model: use chatGLM2 and chatGLM3 models to automatically annotate compounds in the literature.

[0062] First prediction data annotation result: The model extracted 5 compound names, but the annotation confidence of two of them was low and did not reach the set confidence threshold (such as 60%). These two compounds may be due to unclear descriptions in the literature or the model's poor performance in processing complex chemical symbols.

[0063] (2) Second round of annotation

[0064] Handling data that does not meet the confidence threshold: re-enter the system with information of two compounds with lower confidence in the first round of annotation.

[0065] The prompt word remains unchanged: it is still required to mark the compounds in the text and their properties.

[0066] The second pre-trained large language model: Use the higher-precision chatGLM2 and chatGLM3 models to re-annotate these low-confidence compounds and output the second prediction data annotation results.

[0067] 5) Analyze the second prediction data labeling result to determine the confidence level of the second prediction data labeling result.

[0068] The confidence level of the second prediction data labeling result is determined in the same manner, as shown below:

[0069] Con(S llm-i )=C(S llm-i ) / N;

[0070] Among them, Con(S llm-i ) represents the confidence level; S llm-i Represents the data annotation result of the i-th pre-trained large language model; Con(S llm-i ) indicates S llm-i The number of times a word appears in the answers of N pre-trained large language models; the number of N pre-trained large language models.

[0071] 6) Determine a second data labeling result based on the second predicted data labeling result that meets the second confidence threshold.

[0072] Similarly, the second confidence threshold can be a value (such as 60%) or a range (such as 60%-80%). The second prediction data annotation result that meets the second confidence threshold is the second data annotation result, that is, at least part of the final output result.

[0073] In some examples of the present application, the aforementioned second data annotation result is the first predicted data annotation result with the highest confidence. In some examples, there may be multiple annotation results that meet the confidence threshold, and the second predicted data annotation result with the highest confidence is used as the second data annotation result.

[0074] Those skilled in the art can understand that after the second pre-trained large language model is labeled, the method further includes: inputting the data to be labeled and the prompt word corresponding to the second predicted data labeling result that does not meet the second confidence threshold into the third pre-trained large language model to determine the third predicted data labeling result. In some examples of the present application, some higher-precision pre-trained large language models can be added based on calculation accuracy and operating resources.

[0075] In some examples of the present application, the third pre-trained large language model is selected from at least one of LLaMA3-70b and chatGPT4.

[0076] It can be understood that, in the Nth round of pre-trained large language model labeling, the Nth round predicted data labeling result that meets the confidence threshold and has the highest confidence is also selected as the Nth round data labeling result.

[0077] In other examples of the present application, a larger language model with higher accuracy may be added or a multimodal model may be used for prediction.

[0078] 7) Manually labeling the data to be labeled corresponding to the second predicted labeling result that does not meet the second confidence threshold, and determining a manual data labeling result.

[0079] After multiple rounds of data annotation, the data to be annotated that still does not meet the confidence threshold is annotated manually. In some examples of the present application, the human beings used for data annotation are professional technicians in the relevant field to ensure the accuracy of the annotation.

[0080] 8) Combining the first data labeling result, the second data labeling result and the manual data labeling result to determine the final labeling result of the data to be labeled.

[0081] This method improves data annotation accuracy and efficiency and reduces manual annotation costs through a progressive group voting mechanism and human-machine collaborative annotation.

[0082] For ease of understanding, refer to Figure 1 , taking text data as an example, the aforementioned data annotation method is described in detail.

[0083] The text to be annotated is obtained, and the text to be annotated and the prompt words (task requirement questions) are input into multiple first pre-trained large language models, the first data prediction annotation results are determined, and the annotation results are scored to calculate the confidence.

[0084] The obtained confidence level is compared with a first predetermined threshold value, and if it meets the first predetermined threshold value, the first predicted data labeling result is considered to be the first data labeling result (ie, at least part or all of the final labeling result).

[0085] If the confidence does not meet the first predetermined threshold, the data to be labeled and the prompt words that do not meet the first predetermined threshold are input into multiple second pre-trained large language models, the modified labeling results (second predicted data labeling results) are determined, and the labeling results are scored to calculate the confidence.

[0086] The obtained confidence level is compared with a second predetermined threshold, and if it meets the second predetermined threshold, the second predicted data labeling result is considered to be the second data labeling result (ie, at least part or all of the final labeling result).

[0087] If the confidence level does not meet the second predetermined threshold, the to-be-annotated data that does not meet the second predetermined threshold is manually annotated to determine the manual annotation result.

[0088] The first annotation result, the second annotation result and the manual annotation result are combined to obtain the final annotation result.

[0089] Data annotation device

[0090] On the other hand, the present application proposes a data annotation device, referring to Figure 2 The device includes: a first prediction data labeling module 100, a first confidence determination module 200, a first data labeling result determination module 300, a second prediction data labeling module 400, a second confidence determination module 500, a second data labeling result determination module 600, a manual labeling module 700 and a final labeling result output module 800. Among them,

[0091] Module 100, used to input the data to be labeled and the prompt word into the first pre-trained large language model to determine the first predicted data labeling result;

[0092] In some examples of the present application, the module 100 further includes a prompt word optimization submodule, which is used to optimize the prompt word using a standard data annotation result obtained by manual annotation.

[0093] In some examples of the present application, the aforementioned first pre-trained large language model is selected from at least one of chatGLM2 and chatGLM3.

[0094] Module 200 is used to analyze the first prediction data labeling result and determine the confidence level of the first prediction data labeling result;

[0095] In some examples of the present application, the aforementioned confidence is calculated by the following formula:

[0096] Con(S llm-i )=C(S llm-i ) / N;

[0097] Among them, Con(S llm-i ) represents the confidence level; S llm-i Represents the data annotation result of the i-th pre-trained large language model; Con(S llm-i ) indicates S llm-i The number of times a word appears in the answers of N pre-trained large language models; the number of N pre-trained large language models.

[0098] Module 300, configured to determine a first data labeling result based on the first predicted data labeling result that meets a first confidence threshold;

[0099] In some examples of the present application, the aforementioned first data labeling result is the first predicted data labeling result with the highest confidence.

[0100] Module 400 is used to input the data to be annotated and the prompt word corresponding to the first prediction data annotation result that does not meet the first confidence threshold into the second pre-trained large language model to determine the second prediction data annotation result;

[0101] In some examples of the present application, the aforementioned second pre-trained large language model is selected from at least one of LLaMA3-8b and chatGPT (3.5).

[0102] Module 500 is used to analyze the second prediction data labeling result and determine the confidence level of the second prediction data labeling result;

[0103] In some examples of this application, the confidence in this module is also calculated by the following formula:

[0104] Con(S llm-i )=C(S llm-i ) / N;

[0105] Among them, Con(S llm-i ) represents the confidence level; S llm-i Represents the data annotation result of the i-th pre-trained large language model; Con(S llm-i ) indicates S llm-i The number of times a word appears in the answers of N pre-trained large language models; the number of N pre-trained large language models.

[0106] Module 600 is used to determine a second data annotation result based on the second predicted data annotation result that meets a second confidence threshold;

[0107] In some examples of the present application, the aforementioned second data labeling result is the second predicted data labeling result with the highest confidence.

[0108] In some examples of the present application, after the second pre-trained large language model is labeled, the method further includes: inputting the data to be labeled and the prompt word corresponding to the second predicted data labeling result that does not meet the second confidence threshold into a third pre-trained large language model to determine the third predicted data labeling result. In some examples of the present application, some higher-precision pre-trained large language models can be added based on calculation accuracy and operating resources.

[0109] In some examples of the present application, the third pre-trained large language model is selected from at least one of LLaMA3-70b and chatGPT4.

[0110] Module 700 is used to manually label the data to be labeled corresponding to the second predicted labeling result that does not meet the second confidence threshold, and determine the manual data labeling result;

[0111] Module 800 is used to merge the first data labeling result, the second data labeling result and the manual data labeling result to determine the final labeling result of the data to be labeled.

[0112] In some examples of the present application, the aforementioned modules are connected, wherein the aforementioned connection includes physical connection and network connection.

[0113] The data annotation device adopts modular design, uses multiple rounds of pre-trained large language model prediction, and combines confidence analysis and manual intervention to significantly improve the accuracy and reliability of data annotation. By introducing large language models of different precisions, the device shows strong adaptability and robustness in complex data annotation scenarios. Finally, by integrating automatic and manual annotation results, efficient and accurate final annotation output is ensured.

[0114] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Figure 2 The device shown can execute the embodiment of the above-mentioned data labeling method, and the operations and / or functions performed by each module in the device correspond to the method embodiment, which will not be repeated here for the sake of brevity.

[0115] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0116] Computer program product, computing device, and computer readable storage medium

[0117] In another aspect of the present application, the present application proposes a computer program product, a computing device, and a computer-readable storage medium. Based on the aforementioned computer program product, computing device, or computer-readable storage medium, the aforementioned data labeling method is executed.

[0118] Descriptions of computer program products, computing devices, or computer-readable storage media may refer to each other. This article takes electronic devices as an example for detailed description. The so-called electronic devices are intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. Computing devices can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0119] like Figure 3 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a ROM (Read-Only Memory) 502 or a computer program loaded from a storage unit 508 to a RAM (Random Access Memory) 503. In the RAM 503, various programs and data required for the operation of the device 500 can also be stored. The computing unit 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An I / O (Input / Output) interface 505 is also connected to the bus 504.

[0120] A number of components in the device 500 are connected to the I / O interface 505, including: an input unit 506, such as a keyboard, a mouse, etc.; an output unit 507, such as various types of displays, speakers, etc.; a storage unit 508, such as a disk, an optical disk, etc.; and a communication unit 509, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 509 allows the device 500 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0121] The computing unit 501 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a CPU (Central Processing Unit), a GPU (Graphic Processing Units), various dedicated AI (Artificial Intelligence) computing chips, various computing units running machine learning model algorithms, a DSP (Digital Signal Processor), and any appropriate processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as a data annotation method. For example, in some embodiments, the data annotation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via the ROM 502 and / or the communication unit 509. When the computer program is loaded into the RAM 503 and executed by the computing unit 501, one or more steps of the method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to execute the aforementioned data labeling method in any other appropriate manner (eg, by means of firmware).

[0122] In the present application, the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the aforementioned program may be printed, because the aforementioned program may be obtained electronically, for example, by optically scanning the paper or other medium, and then editing, interpreting or processing in other suitable ways as necessary, and then storing it in a computer memory. The various computer-readable storage media described in the present invention may represent one or more devices and / or other machine-readable storage media for storing information. The term "machine-readable storage medium" may include, but is not limited to, wireless channels and various other media capable of storing, containing and / or carrying instructions and / or data.

[0123] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0124] A person of ordinary skill in the art may understand that all or part of the steps in the method of the above-mentioned embodiment may be completed by instructing related hardware through a program, and the aforementioned program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.

[0125] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

[0126] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0127] It should be noted that the features and technical effects described in this article for different aspects can be used as reference for each other and will not be repeated here.

[0128] The embodiments of the present application will be described in more detail below, and examples of the embodiments are shown in the accompanying drawings. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, but cannot be understood as limiting the present application.

[0129] Example 1: Chemical Literature Fragment Data Annotation

[0130] This example uses chemical literature fragments as an example for data annotation, aiming to illustrate the accuracy and feasibility of the method of this application. Figure 4 , as follows:

[0131] Data to be annotated: Under the conditions of reaction temperature of 120℃ and pressure of 1.5atm, ethanol (C2H5OH) and acetic acid (CH3COOH) react with concentrated sulfuric acid (H2SO4) to produce ethyl acetate (C4H8O2) and water (H2O). The reaction lasted for 3 hours.

[0132] Prompt: What are the reactants in the experiment?

[0133] Correct answer: ethanol, acetic acid.

[0134] The first pre-trained large language model: LLaMA3-8B, GPT-2, Text-to-Text TransferTransformer, DistilBERT, and ALBERT;

[0135] The second pre-trained large language model: ChatGLM2, ChatGLM3, GPT-3, ChatGPT 3.5 and RoBERTa;

[0136] The first confidence threshold and the second confidence threshold are both 0.8;

[0137] Annotator: A professional with some background knowledge in chemistry.

[0138] After two rounds of pre-trained large language model annotation combined with manual annotation, the expected results were finally obtained (correct answers: ethanol, acetic acid).

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0140] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in the field can change, modify, replace and modify the above embodiments within the scope of the present application without departing from the principles and purpose of the present application.

Claims

1. A data labeling method, characterized in that: include: Inputting the data to be labeled and the prompt word into the first pre-trained large language model to determine the first predicted data labeling result; Analyzing the first prediction data labeling result to determine the confidence level of the first prediction data labeling result; Determining a first data labeling result based on the first predicted data labeling result that meets a first confidence threshold; Inputting the data to be annotated and the prompt word corresponding to the first predicted data annotation result that does not meet the first confidence threshold into the second pre-trained large language model to determine the second predicted data annotation result; Analyzing the second prediction data labeling result to determine the confidence level of the second prediction data labeling result; Determining a second data labeling result based on the second predicted data labeling result that meets a second confidence threshold; Manually annotating the data to be annotated corresponding to the second predicted annotation result that does not meet the second confidence threshold, and determining a manual data annotation result; The first data labeling result, the second data labeling result and the manual data labeling result are combined to determine a final labeling result of the data to be labeled.

2. The method according to claim 1, characterized in that The first data labeling result is the first predicted data labeling result with the highest confidence; Optionally, the second data labeling result is the second predicted data labeling result with the highest confidence.

3. The method according to claim 1, characterized in that The first pre-trained large language model is selected from at least one of chatGLM2 and chatGLM3.

4. The method according to claim 1, characterized in that The second pre-trained large language model is selected from at least one of LLaMA3-8b and chatGPT (3.5).

5. The method according to claim 1, characterized in that Further including: Optimizing the prompt words using standard data annotation results; Optionally, the standard data annotation result is obtained through manual annotation.

6. The method according to any one of claims 1 to 5, characterized in that: The confidence level is calculated using the following formula: Con(S llm-i )=C(S llm-i ) / N; Among them, Con(S llm-i ) represents the confidence level; S llm-i Represents the data annotation result of the i-th pre-trained large language model; Con(S llm-i ) indicates S llm-i The number of times a word appears in the answers of N pre-trained large language models; the number of N pre-trained large language models.

7. The method according to claim 1, characterized in that After the second pre-trained large language model is labeled and before the manual labeling, the method further includes: inputting the data to be labeled and the prompt word corresponding to the second predicted data labeling result that does not meet the second confidence threshold into the third pre-trained large language model to determine the third predicted data labeling result; Optionally, the third pre-trained large language model is selected from at least one of LLaMA3-70b and chatGPT4.

8. A data labeling device, characterized in that: include: A first prediction data labeling module, used for inputting the data to be labeled and the prompt word into the first pre-trained large language model to determine the first prediction data labeling result; A first confidence determination module, used to analyze the first prediction data labeling result and determine the confidence of the first prediction data labeling result; a first data labeling result determining module, configured to determine a first data labeling result based on the first predicted data labeling result that meets a first confidence threshold; A second prediction data labeling module, used for inputting the data to be labeled and the prompt word corresponding to the first prediction data labeling result that does not meet the first confidence threshold into a second pre-trained large language model to determine a second prediction data labeling result; A second confidence determination module, used to analyze the second prediction data labeling result and determine the confidence of the second prediction data labeling result; A second data labeling result determining module, configured to determine a second data labeling result based on the second predicted data labeling result that meets a second confidence threshold; A manual labeling module, used for manually labeling the data to be labeled corresponding to the second predicted labeling result that does not meet the second confidence threshold, and determining a manual data labeling result; The final labeling result output module is used to merge the first data labeling result, the second data labeling result and the manual data labeling result to determine the final labeling result of the data to be labeled.

9. A computing device, characterized in that include: Processor and memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the data labeling method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions or programs. When the computer instructions or programs are executed on a computer, the data labeling method according to any one of claims 1 to 7 is executed.

Images