Data set labeling method and device based on multiple LLMs, equipment and medium

Abstract

The invention relates to a data set labeling method and device based on multiple LLMs, equipment and a medium, and the method comprises the following steps: obtaining a reference data set, and carrying out the manual labeling; selecting a plurality of LLM models and constructing an LLM annotation combination based on logic calculation; based on the labeling result of the reference data set, the labeling accuracy of each LLM labeling combination is calculated, and the combination with the highest labeling accuracy is selected as the optimal combination scheme; and obtaining a to-be-labeled data set, and labeling the to-be-labeled data set by using the optimal combination scheme. Compared with the prior art, the method has the advantages of high accuracy of automatic labeling results and the like.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data set annotation method, device, equipment and medium based on multiple LLMs. Background Art

[0002] Existing classification models perform poorly in classification. For example, in the case of minority languages, traditional methods are usually trained on a single language, resulting in low accuracy. If you want to develop a multilingual classification model that meets business needs, you will face the following technical problems:

[0003] 1. Open source classification models may not meet specific business needs for the following reasons: inconsistent classification definitions due to different regulatory requirements, for example, the definition and degree of toxicity vary in different regions; there is no corresponding open source model in the language of the business requirement: if the language of the open source model remains unchanged, the workflow of first translation and then prediction is adopted, that is, the content to be identified is translated into a language that the model can recognize, which will result in a large loss of accuracy.

[0004] 2. The model performance produced by the traditional single-language training method needs to be improved, and the cost of training models in multiple languages ​​is high.

[0005] 3. It is difficult to find low-cost, high-quality manually annotated data for multilingual minority languages.

[0006] The above is a typical demand scenario. In other fields, there is also the problem of insufficient classification accuracy of models in complex scenarios. The general solution to this type of problem is to use automatic strategies to annotate corpora and develop models based on the annotated corpora. The more corpora that are automatically annotated and the higher the accuracy, the better the classification effect of the model.

[0007] As the capabilities of large language models (LLMs) increase, they play an increasingly important role in data annotation. Generally speaking, LLM annotation includes the following steps:

[0008] (1) Model selection: Select an LLM;

[0009] (2) Prompt Engineering: Create a prompt to guide LLM to complete the annotation of input text;

[0010] (3) Annotation: Use the created prompts to annotate large amounts of text;

[0011] (4) Post-processing: converting LLM’s answers into structured annotation results.

[0012] The above prior art has the following disadvantages:

[0013] 1. The accuracy of a single LLM annotation is limited;

[0014] 2. Different LLMs have different emphases in terms of language, task type, semantic understanding, etc., and their complementary advantages are not utilized.

[0015] Therefore, a method is needed to automatically and accurately annotate corpora. Summary of the invention

[0016] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a data set annotation method, device, equipment and medium based on multiple LLMs. With a small number of manually annotated samples as a benchmark, the annotation scheme with the best performance can be automatically selected. This scheme has significantly improved the annotation performance compared to a single LLM.

[0017] The purpose of the present invention can be achieved by the following technical solutions:

[0018] According to a first aspect of the present invention, a data set annotation method based on multiple LLMs is provided, the method comprising the following steps:

[0019] Obtain benchmark datasets and perform manual annotation;

[0020] Select multiple LLM models and construct LLM annotation combinations based on logical calculations;

[0021] Based on the annotation results of the benchmark dataset, the annotation accuracy of each LLM annotation combination is calculated, and the combination with the highest annotation accuracy is selected as the optimal combination solution;

[0022] Obtain the data set to be labeled, and use the optimal combination solution to label the data set to be labeled.

[0023] As a preferred technical solution, the selecting of multiple LLM models specifically includes: selecting multiple LLM models for solving similar labeling problems and constructing an LLM model set.

[0024] As an optimal technical solution, the construction of LLM annotation combinations based on logical calculation is specifically as follows: using logical AND and logical OR to construct a logical combination scheme of all elements in the LLM model set, assuming that the number of selected LLM models is a, a≥2, the logical combination scheme includes at least a single model scheme, a*(a-1) / 2 dual model schemes based on logical AND and a*(a-1) / 2 dual model schemes based on logical OR; when a≥3, the logical combination scheme also includes a multi-model scheme based on logical OR, a multi-model scheme based on logical AND and a multi-model scheme combining logical OR and logical AND.

[0025] As a preferred technical solution, the annotation accuracy is calculated based on the proportion of samples in the annotation results of the LLM annotation combination solution that are consistent with the annotation results of the benchmark data set.

[0026] As a preferred technical solution, the calculation formula for the marking accuracy is:

[0027]

[0028] Among them, label_F_i represents the labeling result of the i-th sample obtained by labeling the benchmark dataset using the LLM labeling combination scheme F, label_Ds_i represents the manual labeling result of the i-th sample of the benchmark dataset Ds, and n represents the number of samples.

[0029] As a preferred technical solution, the format of the data set includes one or more of text, picture, audio and video.

[0030] As a preferred technical solution, the types of annotation include text generation and label classification.

[0031] According to a second aspect of the present invention, a data set annotation device based on multiple LLMs is provided, for implementing the above method, the device comprising:

[0032] Benchmark data set acquisition and annotation module: used to acquire benchmark data sets and perform manual annotation;

[0033] LLM annotation combination construction module: used to select multiple LLM models and construct LLM annotation combinations based on logical calculations;

[0034] Optimal combination scheme determination module: used to calculate the annotation accuracy of each LLM annotation combination based on the annotation results of the benchmark data set, and select the combination with the highest annotation accuracy as the optimal combination scheme;

[0035] Labeling module: used to obtain the data set to be labeled and label the data set using the optimal combination solution.

[0036] According to a third aspect of the present invention, there is provided an electronic device, comprising a memory and a processor, wherein a computer program is stored in the memory, and the method described above is implemented when the processor executes the program.

[0037] According to a fourth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, wherein the program implements the method described when executed by a processor.

[0038] Compared with the prior art, the present invention uses a small number of manually annotated samples as a benchmark and can automatically select the best-performing annotation scheme. This scheme has significantly improved the annotation performance of a single LLM and can generate high-quality, low-cost annotated data. This method can be applied to the annotation of data in multiple modalities and has a wide range of applications. When this method is applied to text annotation, the annotated samples produced by this method can be used for multi-language mixed training, and the output accuracy and generalization are both optimal for multi-language classification models. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 is a flow chart of the method of the present invention;

[0040] Figure 2 The figure is a schematic diagram of the process of selecting the optimal combination scheme in an embodiment. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0042] This embodiment provides a data set annotation method based on multiple LLMs, such as Figure 1 As shown, the method comprises the following steps:

[0043] S1, obtain the benchmark dataset Ds and perform manual annotation.

[0044] In this step, only a small amount of manual data annotation is required for the benchmark dataset.

[0045] S2, select multiple LLM models and construct LLM annotation combinations based on logical calculations.

[0046] Select multiple LLM models for solving similar annotation problems and build an LLM model set. The types of annotation problems include but are not limited to text generation and label classification.

[0047] A logical combination scheme of all elements in the LLM model set is constructed using logical AND and logical OR. Assuming that the number of selected LLM models is a, a≥2, the logical combination scheme includes at least a single-model scheme, a*(a-1) / 2 dual-model schemes based on logical AND, and a*(a-1) / 2 dual-model schemes based on logical OR; when a≥3, the logical combination scheme also includes a multi-model scheme based on logical OR, a multi-model scheme based on logical AND, and a multi-model scheme combining logical OR and logical AND.

[0048] This embodiment takes category labeling as an example. Assuming that the number of selected LLM models is 3, represented by a, b, and c respectively, then the LLM model set Z = {a, b, c}. Assuming that the requirement is to label the content as "yes" or "no" to indicate whether the content belongs to a certain category, the combination schemes are {a}, {b}, {c}, {a AND b}, {a AND c}, {b AND c}, {a AND b AND c}, {a ORb}, {a OR c}, {b OR c}, {a OR b OR c}, {a AND bOR c}, {a AND c OR b}, {b AND c ORa}. Among them, AND means "logical AND" and OR means "logical OR".

[0049] S3, based on the annotation results of the benchmark dataset, calculate the annotation accuracy of each LLM annotation combination, and select the combination with the highest annotation accuracy as the optimal combination solution.

[0050] The annotation accuracy is calculated based on the proportion of samples in the annotation results of the LLM annotation combination scheme that are consistent with the annotation results of the benchmark dataset. The calculation formula is:

[0051]

[0052] Among them, label_F_i represents the labeling result of the i-th sample obtained by labeling the benchmark dataset using the LLM labeling combination scheme F, label_Ds_i represents the manual labeling result of the i-th sample of the benchmark dataset Ds, and n represents the number of samples.

[0053] S4, obtaining a dataset D to be labeled, and labeling the dataset using the optimal combination scheme.

[0054] In this embodiment, the format of the data set includes but is not limited to one or more of text, picture, audio and video.

[0055] The present invention has been verified in multilingual toxic content annotation, and compared with the single LLM annotation effect, the consistency with the manual annotation results has been improved by more than 5%.

[0056] This embodiment also provides a data set annotation device based on multiple LLMs, which is used to implement the above method, and the device includes:

[0057] Benchmark data set acquisition and annotation module: used to acquire benchmark data sets and perform manual annotation;

[0058] LLM annotation combination construction module: used to select multiple LLM models and construct LLM annotation combinations based on logical calculations;

[0059] Optimal combination scheme determination module: used to calculate the annotation accuracy of each LLM annotation combination based on the annotation results of the benchmark data set, and select the combination with the highest annotation accuracy as the optimal combination scheme;

[0060] Labeling module: used to obtain the data set to be labeled and label the data set using the optimal combination solution.

[0061] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the described module can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0062] The electronic device of the present invention includes a central processing unit (CPU), which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) or loaded from a storage unit into a random access memory (RAM). In the RAM, various programs and data required for device operation can also be stored. The CPU, ROM and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0063] Multiple components in the device are connected to the I / O interface, including: input units, such as keyboards, mice, etc.; output units, such as various types of displays, speakers, etc.; storage units, such as disks, optical disks, etc.; and communication units, such as network cards, modems, wireless communication transceivers, etc. The communication unit allows the device to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunication networks.

[0064] The processing unit performs the various methods and processes described above, such as methods S1 to S4. For example, in some embodiments, methods S1 to S4 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit. In some embodiments, part or all of the computer program may be loaded and / or installed on the device via a ROM and / or a communication unit. When the computer program is loaded into the RAM and executed by the CPU, one or more steps of methods S1 to S4 described above may be performed. Alternatively, in other embodiments, the CPU may be configured to execute methods S1 to S4 in any other appropriate manner (e.g., by means of firmware).

[0065] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0066] The program code for implementing the method of the present invention can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0067] In the context of the present invention, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0068] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A dataset annotation method based on multiple LLMs, characterized in that: The method comprises the following steps: Obtain benchmark datasets and perform manual annotation; Select multiple LLM models and construct LLM annotation combinations based on logical calculations; Based on the annotation results of the benchmark dataset, the annotation accuracy of each LLM annotation combination is calculated, and the combination with the highest annotation accuracy is selected as the optimal combination solution; Obtain the data set to be labeled, and use the optimal combination solution to label the data set to be labeled.

2. According to the multi-LLM-based dataset annotation method of claim 1, it is characterized in that: The selecting of multiple LLM models specifically includes: selecting multiple LLM models for solving similar labeling problems and constructing an LLM model set.

3. According to the multi-LLM-based dataset annotation method of claim 2, it is characterized in that: The method of constructing an LLM annotation combination based on logical calculation is specifically as follows: constructing a logical combination scheme of all elements in an LLM model set using logical AND and logical OR. Assuming that the number of selected LLM models is a, a≥2, the logical combination scheme includes at least a single model scheme, a*(a-1) / 2 dual model schemes based on logical AND, and a*(a-1) / 2 dual model schemes based on logical OR; when a≥3, the logical combination scheme also includes a multi-model scheme based on logical OR, a multi-model scheme based on logical AND, and a multi-model scheme combining logical OR and logical AND.

4. The method for labeling a data set based on multiple LLMs according to claim 1, characterized in that: The annotation accuracy is calculated based on the percentage of samples in the annotation results of the LLM annotation combination scheme that are consistent with the annotation results of the benchmark dataset.

5. According to the multi-LLM-based dataset annotation method of claim 4, it is characterized in that: The calculation formula of the marking accuracy is: Among them, label_∑_i represents the labeling result of the i-th sample obtained by labeling the benchmark dataset using the LLM labeling combination scheme F, label_Ds_i represents the manual labeling result of the i-th sample of the benchmark dataset Ds, and n represents the number of samples.

6. The method for labeling a data set based on multiple LLMs according to claim 1, characterized in that: The format of the data set includes one or more of text, picture, audio and video.

7. The method for labeling a data set based on multiple LLMs according to claim 1, characterized in that: The types of annotation include text generation and label classification.

8. A data set annotation device based on multiple LLMs, characterized in that: For implementing the method according to any one of claims 1 to 7, the device comprises: Benchmark data set acquisition and annotation module: used to acquire benchmark data sets and perform manual annotation; LLM annotation combination construction module: used to select multiple LLM models and construct LLM annotation combinations based on logical calculations; Optimal combination scheme determination module: used to calculate the annotation accuracy of each LLM annotation combination based on the annotation results of the benchmark data set, and select the combination with the highest annotation accuracy as the optimal combination scheme; Labeling module: used to obtain the data set to be labeled and label the data set using the optimal combination solution.

9. An electronic device comprising a memory and a processor, wherein a computer program is stored in the memory, wherein: When the processor executes the program, the method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images