Method, apparatus, medium and program product for annotating data

Abstract

The invention provides a method and device for annotating data, electronic equipment, a computer readable medium and a computer program product. The large model training method comprises the steps of performing predefined processing in a target large language model; inputting a preset labeling rule into the target large language model; and inputting the to-be-labeled data into the target large language model, and enabling the target large language model to obtain corresponding labeling result information based on predefined human setting and site information and the labeling rule. According to the method, the target large language model is used for obtaining the corresponding labeling result information based on the predefined human setting and standing field information and the labeling rule, the consistency of the machine labeling result obtained by the large language model and the manual labeling result is improved, and the accuracy of machine labeling is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium, and computer program product for annotating data. Background Art

[0002] Based on the existing technology, when training a large model for machine-reviewed text, a large number of complex positive and negative samples need to be annotated with explanatory annotations, which consumes a lot of human annotation costs and time costs, so machine annotation needs to be introduced. However, due to differences in annotation scale, knowledge reserves, position setting and other factors, the consistency between machine annotation and manual annotation is low, and the accuracy is poor. Summary of the invention

[0003] Multiple aspects of the present application provide a method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for annotating data.

[0004] In one aspect of the present application, a method for labeling data is provided, wherein the method comprises:

[0005] Performing pre-defined processing in the target large language model, the pre-defined processing including defining a persona and defining stance information that matches the defined persona;

[0006] Inputting preset annotation rules into the target large language model, wherein the annotation rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and position indication information of the objects to be reviewed;

[0007] The data to be annotated is input into the target large language model, so that the target large language model obtains corresponding annotation result information based on the predefined personality and stance information and the annotation rules.

[0008] In one aspect of the present application, a device for annotating data is provided, wherein the device comprises:

[0009] means for performing predefined processing in a target large language model, the predefined processing comprising defining a persona and defining stance information that conforms to the defined persona;

[0010] A device for inputting preset annotation rules into a target large language model, wherein the annotation rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and position indication information of the objects to be reviewed;

[0011] A device for inputting the data to be annotated into a target large language model, so that the target large language model obtains corresponding annotation result information based on predefined personality and stance information and the annotation rules.

[0012] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of an embodiment of the present application.

[0013] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0014] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0015] The solution provided in the embodiment of the present application obtains corresponding annotation result information based on predefined personality and stance information and the annotation rules by using the target large language model, thereby improving the consistency between the machine annotation results and the manual annotation results obtained by the large language model, improving the accuracy of machine annotation, reducing the labor cost of manual annotation, and improving data annotation efficiency; the annotation rules are converted into an executable program through a reasoning method based on a thinking program, and complex annotation problems are structured, which further improves the consistency between the machine annotation results and the manual annotation, improves the accuracy, and can trace and reverse the results according to the results of each step of the program execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0017] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0018] Figure 1 A schematic diagram of a process for labeling data according to an embodiment of the present application is shown;

[0019] Figure 2 A schematic diagram of the structure of a device for marking data provided by an embodiment of the present application is shown;

[0020] Figure 3 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0021] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0024] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0025] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0026] Figure 1 A schematic flow chart of a method for annotating data provided in an embodiment of the present application is shown. The method at least includes step S101, step S102, step S103 and step S104.

[0027] In actual scenarios, the execution subject of the method can be a network device, or an application running on a network device, wherein the network device includes but is not limited to a network host, a single network server, a plurality of network server sets, or a collection of computers based on cloud computing, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computers.

[0028] The concepts involved in the embodiments of the present application are explained below.

[0029] Large Language Model (LLM): Large language model is a commonly used model in the field of natural language processing. It is used to process a variety of natural language tasks, such as text classification, question answering, dialogue, etc., to generate natural language text or understand the meaning of language text. Large language model is also a general term for deep learning models trained with large amounts of text data. For example, GPT-3, PaLM, Galactica, LLaMA and other models are all large language models commonly used by technicians in this field.

[0030] GPT: GPT (Generative Pre-trained Transformer), such as OPENAI's ChatGPT or Google's Grad, can provide users with a visual conversation interface and generate corresponding answers based on user input, giving users an intelligent user experience.

[0031] Program-of-Thoughts: Program-of-Thoughts (PoT) formulates the reasoning process behind question-answering into an executable program, with the program interpreter output as part of the final answer. Program-of-Thoughts (PoT) is a unique LLM reasoning method. Rather than just generating natural language answers, it requires creating an executable program that can be run on a program interpreter such as Python to produce actual results. Compared to direct models, this approach emphasizes the ability to decompose reasoning into sequential steps and associate semantics with variables.

[0032] In the machine review scenario, the method of the embodiment of the present application is implemented by a generative large language model. The embodiment of the present application sets a person in the generative large language model, sets it as a community review expert, and hopes that it can judge whether there are sensitive elements involved in the relevant violation categories in the comment content and the manuscript content, and the emotional tendency of the comment content to these sensitive elements based on the given definition of the relevant violation categories, the comment content, and the manuscript content, and then give the final violation line judgment and explanation through the reasoning steps of program thinking (PoT) based on these judgments as the corresponding annotation results.

[0033] In some embodiments, a prompt framework based on the method of the present application is designed based on machine annotation rules based on programmatic thinking (PoT), and the prompt framework is used to enable the target large language model to obtain corresponding annotation results based on predefined personality and stance information and the annotation rules.

[0034] Reference Figure 1 , in step S101, predefined processing is performed in the target large language model.

[0035] Specifically, the predefined processing is performed by inputting the instruction text into the target large language model.

[0036] The pre-defined processing includes defining a persona and defining position information that conforms to the defined persona.

[0037] The target large language model is a generative large language model. The target large language model according to the embodiment of the present application is used to annotate data and output corresponding annotation results.

[0038] Among them, character setting refers to the personality characteristics, appearance description, behavior habits, character traits, etc. set for the character during the process of virtual character creation, text creation, role-playing, etc. The definition of character setting can be achieved by specifying the character's gender, age, occupation, hobbies, language style and other factors.

[0039] For example, in a large language model, the character's personality traits can be set by inputting command text or gradually revealed during the conversation, making it more in line with user needs during the conversation.

[0040] Optionally, the predefined processing also includes defining data annotation requirements.

[0041] The data annotation requirements are used to indicate the aspects and information from which the target large language model needs to make judgments, and what conclusions it ultimately gives.

[0042] Optionally, the method indicates the data format or data structure of the annotation results that need to be output by the target large language model by defining data annotation requirements.

[0043] Explicit instructions will enable the target large language model to correctly understand the instructions and generate expected results. It can also help narrow the scope of generated text and avoid generating text that is irrelevant to the topic or contains a lot of irrelevant information.

[0044] The stance information includes various information that can be used by the target large language model to learn the stance and attitude of a predefined persona towards a specific thing or behavior.

[0045] According to the first example, in a scenario of reviewing comments on a manuscript, it is desired to use a large language model to output annotation results.

[0046] The following dialogue is input into the large language model to define the persona: "If you are a community review expert, please judge whether the comments and manuscripts contain any illegal or illegal content and the position of the comments based on the definitions of illegal or illegal content, comment content, manuscript content, and sentiment tendency, and then make a safety judgment based on the return value of the Python code." By inputting the above dialogue into the large language model, the large language model is set as the persona of a security review expert, limiting the specific field, style, and type of the generated results, making the generated explanatory annotations more focused.

[0047] In the large language model, the sensitive element is defined by inputting the following dialogue: "Relevant speech involving illegal and irregular issues, mainly referring to supporting and advocating various behaviors that violate Chinese laws and regulations."

[0048] The following dialogue is input into the large language model to define the position that matches the persona (review expert): "Prohibit all kinds of illegal and irregular behaviors." It can be seen that based on this definition, if the comment deliberately mentions various illegal behaviors or incites encouragement, it is inconsistent with this position; if it is critical of various illegal and irregular behaviors, it is consistent with this position.

[0049] Through the above settings and processing, the large language model can determine whether there are relevant violation categories and sensitive elements involved in the comments and manuscript content of the input video manuscripts to be reviewed, and whether the emotional tendency of the comments on the existing sensitive elements is consistent with the set position.

[0050] Next, continue Figure 1 To illustrate, in step S102, the preset labeling rules are input into the target large language model.

[0051] The marking rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and the position indication information of the objects to be reviewed. The objects to be reviewed are preset data that needs to be judged as illegal or in violation of regulations. For example, for a video manuscript to be reviewed, the objects to be reviewed may include the manuscript content and the comment content.

[0052] The position indication information is used to indicate whether the position of the object to be reviewed is consistent with the position corresponding to the predefined persona.

[0053] The method determines whether the position of the audit object is consistent with the position corresponding to the predefined persona through the following steps:

[0054] The target large language model obtains the emotional tendency information of the text content of the target sample towards the element to be audited by performing sentiment analysis on the object to be audited. The emotional tendency information includes various information that can be used to indicate emotional attitudes towards specific things or behaviors. For example, "support", "oppose", "criticism", "disgust", etc. The sentiment analysis process can be achieved by identifying whether predetermined keywords are included. Then, the position of the object to be audited is analyzed based on the emotional tendency information to obtain the position indication information of the object to be audited, and then determine whether it is consistent with the predefined position information of the persona.

[0055] According to one embodiment, the method converts the audit rules into an executable program based on the rule logic contained in the annotation rules and the reasoning method of PoT. The executable program can be run on a program interpreter such as Python.

[0056] Among them, PoT is a reasoning framework that simulates the human decision-making process. It breaks down complex thinking and judgment tasks into a series of orderly steps. Each step corresponds to a clear semantic unit and decision variable, making the entire reasoning process clearer and easier to track. Those skilled in the art should be familiar with setting or updating audit rules according to different tasks and requirements and adjusting the PoT reasoning steps accordingly.

[0057] Continuing to explain the first example, the rules of this example based on whether the comment content violates the rules, whether the manuscript content violates the rules, and whether the comment content complies with the official position are shown in Table 1 below:

[0058] Table 1

[0059]

[0060] Based on the rules shown in the table above, the content elements, manuscript content elements, and stance tendencies are taken as three features, and based on the judgment results obtained under different values, the following decision tree-like PoT program code is designed:

[0061] def main(comment content, manuscript content, comment stance):

[0062] If the comment content == "illegal and illegal" and the comment stance == "inconsistent with the preset stance":

[0063] ans="Rejected due to illegality or irregularity"

[0064] else:

[0065] If the content of the article == "illegal and illegal" and the comment position == "inconsistent with the preset position":

[0066] ans="Rejected due to illegal or irregular behavior"

[0067] else:

[0068] ans="Pass"

[0069] return ans

[0070] The large language model can obtain the annotation results based on these judgment results and the above PoT program code. The PoT in this example provides a clearer, more expressive and basic answer derivation model, which improves accuracy and understanding, especially for mathematical logic problems that require numerical calculations.

[0071] Continue to refer to Figure 1 To illustrate, in step S103, the data to be annotated is input into the target large language model, so that the target large language model obtains corresponding annotation result information based on the predefined personality and position information and the annotation rules. The annotation result information includes the judgment of the violation of the data to be annotated and the corresponding explanation.

[0072] Among them, those skilled in the art should be familiar with the fact that the predefined personality, stance information or labeling rules in the target large language model can be modified so that the target large language model obtains corresponding labeling result information based on the updated personality, stance information and / or labeling rules.

[0073] Continuing to explain the first example above, based on the above operations, a prompt framework based on machine annotation rules of program thinking (PoT) is obtained, and the prompt framework can be used to annotate large quantities of data. In addition, the accuracy rate can be calculated, the wrongly annotated data can be checked, and the prompt framework can be modified by sampling the obtained annotation results and handing them over to manual annotation and re-annotation.

[0074] According to one embodiment, the method further includes step S104.

[0075] In step S104, the tagging result output by the target large language model is normalized by adjusting the instruction text input to the target large language model.

[0076] Through the normalization process, the annotation results output by the target large language model have a fixed output style and format.

[0077] Specifically, step S104 includes step S1041 and step S1042.

[0078] In step S1041, the target sample is input into the target large language model, so that the target large language model obtains the labeling result corresponding to the target sample based on the predefined personality and stance information and the review rules.

[0079] The target samples include representative positive and negative samples of various categories.

[0080] In step S1042, the annotation result after the obtained annotation result is reviewed is obtained, wherein the review process makes the annotation result have a fixed output style and format, and the annotation result after the review process is used as an example.

[0081] Continuing with the first example, representative positive and negative samples of each category are selected from the database. Based on the definition of personality, needs, sensitive elements, official stance and the program code of PoT, the large language model is run to generate annotation results. The annotation results are modified to a fixed output style and format through manual review, and the reviewed annotation results are used as examples, so that the annotation explanations generated by the large language model are more accurate and reasonable.

[0082] According to the method of the embodiment of the present application, by training the large language model, the target large language model learns how to obtain the labeling results based on the predefined personality and stance information and the labeling rules, thereby improving the consistency between the machine labeling results and the manual labeling results obtained by the large language model, improving the accuracy of machine labeling, reducing the manpower cost of manual labeling, and improving the efficiency of data labeling; by converting the labeling rules into an executable program through a reasoning method based on the thinking program, the complex labeling problems are structured, further improving the consistency between the machine labeling results and the manual labeling, and improving the accuracy, and the results can be traced and checked back according to the results of each step of the program execution.

[0083] In addition, the embodiment of the present application also provides a device for annotating data, the structure of which is shown in the figure. The device includes: a device for performing predefined processing in a target large language model (hereinafter referred to as "predefinition device 101"), a device for inputting preset annotation rules into the target large language model (hereinafter referred to as "rule input device 102"), and a device for inputting the data to be annotated into the target large language model, so that the target large language model obtains corresponding annotation result information based on predefined personality and position information and the annotation rules (hereinafter referred to as "data annotation device 103").

[0084] Reference Figure 2, the predefinition device 101 performs predefinition processing in the target large language model.

[0085] Specifically, the predefined processing is performed by inputting the instruction text into the target large language model.

[0086] The pre-defined processing includes defining a persona and defining position information that conforms to the defined persona.

[0087] The target large language model is a generative large language model. The target large language model according to the embodiment of the present application is used to annotate data and output corresponding annotation results.

[0088] Among them, character setting refers to the personality characteristics, appearance description, behavior habits, character traits, etc. set for the character during the process of virtual character creation, text creation, role-playing, etc. The definition of character setting can be achieved by specifying the character's gender, age, occupation, hobbies, language style and other factors.

[0089] For example, in a large language model, the character's personality traits can be set by inputting command text or gradually revealed during the conversation, making it more in line with user needs during the conversation.

[0090] Optionally, the predefined processing also includes defining data annotation requirements.

[0091] The data annotation requirements are used to indicate the aspects and information from which the target large language model needs to make judgments, and what conclusions it ultimately gives.

[0092] Optionally, the predefining device 101 indicates the data format or data structure of the annotation results that need to be output by the target large language model by defining data annotation requirements.

[0093] Explicit instructions will enable the target large language model to correctly understand the instructions and generate expected results. It can also help narrow the scope of generated text and avoid generating text that is irrelevant to the topic or contains a lot of irrelevant information.

[0094] The stance information includes various information that can be used by the target large language model to learn the stance and attitude of a predefined persona towards a specific thing or behavior.

[0095] According to the first example, in a scenario of reviewing comments on a manuscript, it is desired to use a large language model to output annotation results.

[0096] The pre-definition device 101 defines the persona by inputting the following dialogue into the large language model: "If you are a community review expert, please judge whether the comment content and the manuscript content involve illegal and illegal content and the comment position based on the definitions of illegal and illegal, comment content, manuscript content, and emotional tendency, and then make a security judgment based on the return value of the Python code." By inputting the above dialogue into the large language model, the large language model is set as the persona of the security review expert, limiting the specific field, style and type of the generated results, so that the generated explanatory annotations are more focused.

[0097] The pre-definition device 101 defines the sensitive element "speech related to illegal and irregular issues, mainly referring to supporting and advocating various behaviors that violate Chinese laws and regulations" in the large language model by inputting the following dialogue.

[0098] The pre-definition device 101 defines the position that matches the persona (review expert) by inputting the following dialogue in the large language model: "Prohibit all kinds of illegal and irregular behaviors." It can be seen that based on this definition, if the comment deliberately mentions various illegal behaviors or incites and encourages, it is inconsistent with this position; if it criticizes and persuades various illegal and irregular behaviors, it is consistent with this position.

[0099] Through the above settings and processing, the large language model can determine whether there are relevant violation categories and sensitive elements involved in the comments and manuscript content of the input video manuscripts to be reviewed, and whether the emotional tendency of the comments on the existing sensitive elements is consistent with the set position.

[0100] Next, continue Figure 1 To explain, the rule input device 102 inputs the preset annotation rules into the target large language model.

[0101] The marking rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and the position indication information of the objects to be reviewed. The objects to be reviewed are preset data that needs to be judged as illegal or in violation of regulations. For example, for a video manuscript to be reviewed, the objects to be reviewed may include the manuscript content and the comment content.

[0102] The position indication information is used to indicate whether the position of the object to be reviewed is consistent with the position corresponding to the predefined persona.

[0103] The device determines whether the position of the audit object is consistent with the position corresponding to the predefined persona by the following operations:

[0104] The target large language model obtains the emotional tendency information of the text content of the target sample towards the element to be audited by performing sentiment analysis on the object to be audited. The emotional tendency information includes various information that can be used to indicate emotional attitudes towards specific things or behaviors. For example, "support", "oppose", "criticism", "disgust", etc. The sentiment analysis process can be achieved by identifying whether predetermined keywords are included. Then, the position of the object to be audited is analyzed based on the emotional tendency information to obtain the position indication information of the object to be audited, and then determine whether it is consistent with the predefined position information of the persona.

[0105] According to one embodiment, the device comprises a program conversion device.

[0106] The rule input device 102 converts the audit rule into an executable program based on the rule logic contained in the annotation rule and the reasoning method of PoT. The executable program can be run on a program interpreter such as Python.

[0107] Among them, PoT is a reasoning framework that simulates the human decision-making process. It breaks down complex thinking and judgment tasks into a series of orderly steps. Each step corresponds to a clear semantic unit and decision variable, making the entire reasoning process clearer and easier to track. Those skilled in the art should be familiar with setting or updating audit rules according to different tasks and requirements and adjusting the PoT reasoning steps accordingly.

[0108] Continuing with the first example, the rules of this example based on whether the comment content violates the rules, whether the manuscript content violates the rules, and whether the comment content complies with the official position are shown in Table 1 above.

[0109] Based on the rules shown in the table above, the rule conversion device takes content elements, manuscript content elements, and position tendencies as three features, and designs the following PoT program code of the class decision tree based on the judgment results obtained under different values:

[0110] def main(comment content, manuscript content, comment stance):

[0111] If the comment content == "illegal and illegal" and the comment stance == "inconsistent with the preset stance":

[0112] ans="Rejected due to illegal or irregular behavior"

[0113] else:

[0114] If the content of the article == "illegal and illegal" and the comment position == "inconsistent with the preset position":

[0115] ans="Rejected due to illegal or irregular behavior"

[0116] else:

[0117] ans="Pass"

[0118] return ans

[0119] The large language model can obtain the annotation results based on these judgment results and the above PoT program code. The PoT in this example provides a clearer, more expressive and basic answer derivation model, which improves accuracy and understanding, especially for mathematical logic problems that require numerical calculations.

[0120] Continue to refer to Figure 1 To illustrate, the data annotation device 103 inputs the data to be annotated into the target large language model, so that the target large language model obtains corresponding annotation result information based on the predefined personality and position information and the annotation rules. The annotation result information includes the judgment of the violation of the data to be annotated and the corresponding explanation.

[0121] Among them, those skilled in the art should be familiar with the fact that the predefined personality, stance information or labeling rules in the target large language model can be modified so that the target large language model obtains corresponding labeling result information based on the updated personality, stance information and / or labeling rules.

[0122] Continuing to explain the first example, based on the above operation, a prompt framework of machine annotation rules based on program thinking (PoT) is obtained, and the data annotation device 103 can use the prompt framework to annotate a large amount of data. In addition, the accuracy rate can be calculated, the wrongly annotated data can be checked, and the prompt framework can be modified by sampling the obtained annotation results and handing them over to manual annotation and re-annotation.

[0123] According to one embodiment, the device further comprises result normalization means.

[0124] The result normalization device normalizes the annotation results output by the target large language model by adjusting the instruction text input to the target large language model.

[0125] Through the normalization process, the annotation results output by the target large language model have a fixed output style and format.

[0126] Specifically, the result standardization device includes a sample input device and a review processing device.

[0127] The sample input device inputs the target sample into the target large language model, so that the target large language model obtains the labeling result corresponding to the target sample based on the predefined personality and stance information and the review rules.

[0128] The target samples include representative positive and negative samples of various categories.

[0129] The review processing device obtains the annotation result after review processing of the obtained annotation result, wherein the review processing makes the annotation result have a fixed output style and format, and takes the annotation result after the review processing as an example.

[0130] Continuing with the first example, representative positive and negative samples of each category are selected from the database. Based on the definition of personality, needs, sensitive elements, official stance and the program code of PoT, the large language model is run to generate annotation results. The annotation results are modified to a fixed output style and format through manual review, and the reviewed annotation results are used as examples, so that the annotation explanations generated by the large language model are more accurate and reasonable.

[0131] According to the device of the embodiment of the present application, by training the large language model, the target large language model learns how to obtain the labeling results based on the predefined personality and stance information and the labeling rules, thereby improving the consistency between the machine labeling results and the manual labeling results obtained by the large language model, improving the accuracy of machine labeling, reducing the manpower cost of manual labeling, and improving the efficiency of data labeling; through the reasoning method based on the thinking program, the labeling rules are converted into an executable program, the complex labeling problems are structured, and the consistency between the machine labeling results and the manual labeling is further improved, and the accuracy is improved, and the results can be traced and checked back according to the results of each step of the program execution.

[0132] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for annotating data in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0133] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0134] Figure 3 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0135] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0136] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0137] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0138] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0139] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0140] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0141] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0142] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0146] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0147] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0149] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for annotating data, wherein: The method comprises: Performing pre-defined processing in the target large language model, the pre-defined processing including defining a persona and defining stance information that matches the defined persona; Inputting preset annotation rules into the target large language model, wherein the annotation rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and position indication information of the objects to be reviewed; The data to be annotated is input into the target large language model, so that the target large language model obtains corresponding annotation result information based on predefined personality and stance information and the annotation rules.

2. The method according to claim 1, wherein: The training method further comprises: According to the rule logic contained in the annotation rule, the annotation rule is converted into an executable program based on the reasoning method of the thinking program.

3. The method according to claim 1, wherein: The method further comprises: By adjusting the instruction text input to the target large language model, the labeling results output by the target large language model are normalized.

4. The method according to claim 3, wherein: The step of normalizing the labeling result output by the target large language model by adjusting the instruction text input to the target large language model includes: Input the target sample into the target large language model, and let the target large language model obtain the labeling result corresponding to the target sample based on the predefined personality and position information and the review rules; The annotation result after the obtained annotation result is reviewed is obtained, wherein the review process makes the annotation result have a fixed output style and format, and the annotation result after the review process is used as an example.

5. The method according to claim 1 or 2, wherein: The method determines whether the position of the audit object is consistent with the position corresponding to the predefined persona through the following steps: The target large language model performs sentiment analysis on the object to be reviewed to obtain the sentiment tendency information of the text content of the target sample to the element to be reviewed; The position of the object to be audited is analyzed based on the emotional tendency information to obtain the position indication information of the object to be audited, and then it is determined whether it is consistent with the predefined position information of the person.

6. The method according to claim 1 or 2, wherein: The predefined processing in the target large language model includes: Predefined processing is performed by inputting instruction text into a target large language model.

7. According to the method of claim 1 or 2, the predefined processing further includes defining data annotation requirements, and the predefined processing in the target large language model includes: The data annotation requirements are defined by inputting instruction text into the target large language model, and the data annotation requirements are used to indicate from which aspects and based on what information the target large language model needs to make judgments, and what kind of conclusions are finally given.

8. A device for annotating data, wherein: The device comprises: means for performing predefined processing in a target large language model, the predefined processing comprising defining a persona and defining stance information that conforms to the defined persona; A device for inputting preset annotation rules into a target large language model, wherein the annotation rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and position indication information of the objects to be reviewed; A device for inputting the data to be annotated into a target large language model, so that the target large language model obtains corresponding annotation result information based on predefined personality and stance information and the annotation rules.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images