Method, apparatus, medium, and program product for annotating data

Abstract

The present application provides a method, apparatus, electronic device, computer-readable medium, and computer program product for labeling data. The large model training method according to the present application includes: performing predefined processing in a target large language model; inputting preset labeling rules into the target large language model; inputting the data to be labeled into the target large language model, so that the target large language model obtains corresponding labeling result information based on predefined personality and stance information and the labeling rules. The present application improves the consistency between the machine labeling results obtained by the large language model and the manual labeling results by using the target large language model to obtain corresponding labeling result information based on predefined personality and stance information and the labeling rules, thereby improving the accuracy of machine labeling.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium, and computer program product for annotating data. Background Art

[0002] Existing solutions require interpretative annotation of a large number of complex positive and negative examples when training large machine-based text review models. This consumes significant human annotation costs and time, necessitating the introduction of machine annotation. However, due to differences in annotation scale, knowledge base, and perspective, machine annotation is less consistent and accurate than manual annotation. Summary of the Invention

[0003] Various aspects of the present application provide a method, apparatus, electronic device, computer-readable medium, and computer program product for annotating data.

[0004] In one aspect of the present application, a method for annotating data is provided, wherein the method comprises:

[0005] Performing pre-definition processing in the target large language model, the pre-definition processing including defining a persona and defining stance information that conforms to the defined persona;

[0006] Inputting preset labeling rules into the target large language model, wherein the labeling rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and the position indication information of the objects to be reviewed;

[0007] The data to be annotated is input into the target large language model, so that the target large language model obtains corresponding annotation result information based on the predefined personality and stance information and the annotation rules.

[0008] In one aspect of the present application, a device for annotating data is provided, wherein the device comprises:

[0009] means for performing pre-defined processing in a target large language model, the pre-defined processing comprising defining a persona and defining stance information consistent with the defined persona;

[0010] means for inputting preset labeling rules into a target large language model, wherein the labeling rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and information indicating the position of the objects to be reviewed;

[0011] A device for inputting the data to be annotated into a target large language model, so that the target large language model obtains corresponding annotation result information based on predefined personality and stance information and the annotation rules.

[0012] Another aspect of the present application provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of an embodiment of the present application.

[0013] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0014] In another aspect of the present application, a computer program product is provided, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0015] The solution provided in the embodiment of the present application obtains corresponding annotation result information based on predefined personality and stance information and the annotation rules using a target large language model, thereby improving the consistency between the machine annotation results and the manual annotation results obtained by the large language model, improving the accuracy of machine annotation, reducing the labor cost of manual annotation, and improving data annotation efficiency; through a reasoning method based on a thinking program, the annotation rules are converted into an executable program, and complex annotation problems are structured, further improving the consistency between the machine annotation results and the manual annotation, improving accuracy, and the results can be traced and checked back according to the results of each step of the program execution. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction is given below to the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0018] Figure 1 A schematic diagram of a process for labeling data according to an embodiment of the present application is shown;

[0019] Figure 2 A schematic structural diagram of a device for annotating data provided by an embodiment of the present application is shown;

[0020] Figure 3 A structural diagram of a device suitable for implementing the solution in the embodiments of the present application is shown.

[0021] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0022] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0023] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0024] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0025] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for information storage. The information can be computer program instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0026] Figure 1 A flow chart of a method for annotating data provided by an embodiment of the present application is shown, wherein the method comprises at least step S101, step S102, step S103 and step S104.

[0027] In actual scenarios, the execution subject of this method can be a network device, or an application running on a network device. The network device includes but is not limited to a network host, a single network server, a set of multiple network servers, or a set of computers based on cloud computing, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing). Cloud computing is a type of distributed computing, which is a virtual computer composed of a group of loosely coupled computers.

[0028] The concepts involved in the embodiments of this application are explained below.

[0029] Large Language Model (LLM): Large language models are commonly used in natural language processing (NLP) to handle a variety of natural language tasks, such as text classification, question answering, and conversation. They are used to generate natural language text or understand the meaning of text. Large language models are also a general term for deep learning models trained using large amounts of text data. Models such as GPT-3, PaLM, Galactica, and LLaMA are all commonly used by those skilled in the art.

[0030] GPT: GPT (Generative Pre-trained Transformer), such as OPENAI's ChatGPT or Google's Grad, can provide users with a visual conversational interface and generate corresponding answers based on user input, giving users an intelligent user experience.

[0031] Program-of-Thoughts (PoT) formulates the reasoning process behind question-answering into an executable program, with the output of a program interpreter as part of the final answer. PoT is a unique LLM reasoning approach. Rather than simply generating natural language answers, it requires creating an executable program that can be run on a program interpreter like Python to produce practical results. Compared to direct models, this approach emphasizes the ability to decompose reasoning into sequential steps and associate semantics with variables.

[0032] In the machine review scenario, the method of the embodiment of the present application is implemented by a generative large language model. The embodiment of the present application sets a human setting in the generative large language model, setting it as a community review expert, and hopes that it can judge whether there are sensitive elements involved in the relevant violation categories in the comment content and manuscript content, as well as the emotional tendency of the comment content towards these sensitive elements, based on the given definition of the relevant violation categories, comment content, and manuscript content. Then, based on these judgments, the final violation line judgment and explanation are given through the reasoning steps of program thinking (PoT) as the corresponding annotation results.

[0033] In some embodiments, a prompt framework based on machine annotation rules based on procedural thinking (PoT) is designed based on the method of the present application. Through this prompt framework, the target large language model obtains corresponding annotation results based on predefined personality and stance information and the annotation rules.

[0034] Reference Figure 1 ,In step S101, predefined processing is performed in the target large language model.

[0035] Specifically, predefined processing is performed by inputting instruction text into a target large language model.

[0036] The pre-defined processing includes defining a persona and defining stance information that conforms to the defined persona.

[0037] The target large language model is a generative large language model. The target large language model according to the embodiment of the present application is used to annotate data and output corresponding annotation results.

[0038] Character design refers to the personality traits, appearance, behavior, and character traits that are set for a character during the creation, writing, and role-playing of a virtual character. Character design can be achieved by specifying various factors, such as the character's gender, age, occupation, hobbies, and language style.

[0039] For example, in a large language model, the character's personality traits can be set or gradually revealed during the conversation by inputting command text, making it more in line with user needs during the conversation.

[0040] Optionally, the predefined process also includes defining data annotation requirements.

[0041] The data annotation requirements are used to indicate the aspects and information from which the target large language model needs to make judgments, and what conclusions it ultimately gives.

[0042] Optionally, the method indicates the data format or data structure of the annotation results that need to be output by the target large language model by defining data annotation requirements.

[0043] Explicit instructions will enable the target large language model to correctly understand the instructions and generate expected results. It can also help narrow the scope of generated text and avoid generating text that is irrelevant to the topic or contains a lot of irrelevant information.

[0044] The stance information includes various information that can be used by the target large language model to learn the stance and attitude of a predefined persona towards specific things or behaviors.

[0045] According to the first example, in a scenario where comments on a manuscript are reviewed, it is desired to use a large language model to output annotation results.

[0046] The following dialogue is input into the large language model to define the persona: "If you are a community review expert, please determine whether the comments and manuscripts contain any violations of laws and regulations, as well as the comment's stance, based on the definitions of violations, comment content, manuscript content, and sentiment. Then, make a safety assessment based on the return value of the Python code." By inputting this dialogue into the large language model, the large language model is set as the persona of a security review expert, limiting the generated results to a specific domain, style, and type, making the generated explanatory annotations more focused.

[0047] The following dialogue is input into the large language model to define sensitive elements: "Remarks related to illegal and irregular issues, mainly referring to support and promotion of various behaviors that violate Chinese laws and regulations."

[0048] The following dialogue was input into the large language model to define the stance that matches the persona (review expert): "All illegal and irregular behaviors are prohibited." Based on this definition, comments that deliberately mention illegal behaviors or encourage them are inconsistent with this stance; comments that criticize and encourage illegal and irregular behaviors are consistent with this stance.

[0049] Through the above settings and processing, the large language model can determine whether there are relevant violation categories and sensitive elements involved in the violations in the comments and manuscript content of the input video manuscripts for review, and determine whether the emotional tendency of the comments towards the existing sensitive elements is consistent with the set position.

[0050] Next, continue Figure 1 To illustrate, in step S102 , the preset labeling rules are input into the target large language model.

[0051] The marking rules are determined based on whether one or more objects to be reviewed violate laws and regulations, as well as information indicating the position of the objects to be reviewed. The objects to be reviewed are pre-defined data that needs to be determined to be illegal or in violation of regulations. For example, for a video manuscript to be reviewed, the objects to be reviewed may include the manuscript content and the comments.

[0052] The position indication information is used to indicate whether the position of the subject to be reviewed is consistent with the position corresponding to the predefined personality.

[0053] The method determines whether the position of the audit object is consistent with the position corresponding to the predefined persona through the following steps:

[0054] The target large language model obtains the emotional tendency information of the text content of the target sample towards the element to be reviewed by performing sentiment analysis on the object to be reviewed. The emotional tendency information includes various information that can be used to indicate emotional attitudes towards specific things or behaviors. For example, "support", "oppose", "criticism", "disgust", etc. The emotional analysis process can be achieved by identifying whether predetermined keywords are included. Then, based on the emotional tendency information, the position of the object to be reviewed is analyzed to obtain the position indication information of the object to be reviewed, and then determine whether it is consistent with the predefined position information of the person.

[0055] According to one embodiment, the method converts the audit rules into an executable program based on the rule logic contained in the annotation rules and the reasoning method of PoT. The executable program can be run on a program interpreter such as Python.

[0056] PoT is a reasoning framework that simulates the human decision-making process, breaking down complex thinking and judgment tasks into a series of ordered steps. Each step corresponds to a clear semantic unit and decision variable, making the entire reasoning process clearer and easier to track. Those skilled in the art will be familiar with setting or updating audit rules and adjusting PoT reasoning steps accordingly based on different tasks and needs.

[0057] Continuing with the first example, the rules for determining whether the comment content violates regulations, whether the manuscript content violates regulations, and whether the comment content complies with the official position are shown in Table 1 below:

[0058] Table 1

[0059]

[0060] Based on the rules shown in the table above, we take content elements, manuscript content elements, and stance as three features, and design the following decision tree-like PoT program code based on the judgment results obtained under different values:

[0061] def main(comment content, manuscript content, comment stance):

[0062] If the comment content == "illegal and in violation of regulations" and the comment stance == "inconsistent with the pre-set stance":

[0063] ans="Rejected due to illegal or irregular behavior"

[0064] else:

[0065] If the article content == "illegal and in violation of regulations" and the comment stance == "inconsistent with the pre-set stance":

[0066] ans="Rejected due to illegal or irregular behavior"

[0067] else:

[0068] ans="pass"

[0069] return ans

[0070] The large language model can generate annotations based on these judgments and the PoT program code above. The PoT in this example provides a clearer, more expressive, and more fundamental model for answer derivation, improving accuracy and comprehension, especially for mathematical logic problems that require numerical calculations.

[0071] Continue to refer to the following Figure 1 To illustrate, in step S103, the data to be annotated is input into the target large language model, and the target large language model obtains corresponding annotation result information based on the predefined personality and stance information and the annotation rules. The annotation result information includes the judgment of the violation of the data to be annotated and the corresponding explanation.

[0072] Among them, those skilled in the art should be familiar with the fact that the predefined personality, stance information or labeling rules in the target large language model can be modified so that the target large language model can obtain corresponding labeling result information based on the updated personality, stance information and / or labeling rules.

[0073] Continuing with the first example, we can now create a prompt framework based on programmatic thinking (PoT) machine annotation rules. This prompt framework can be used to annotate large amounts of data. Furthermore, by sampling the annotation results and submitting them for manual re-annotation, we can calculate accuracy, identify incorrectly annotated data, and modify the prompt framework.

[0074] According to one embodiment, the method further includes step S104.

[0075] In step S104 , the tagging result output by the target large language model is normalized by adjusting the instruction text input to the target large language model.

[0076] Through the normalization process, the annotation results output by the target large language model have a fixed output style and format.

[0077] Specifically, step S104 includes step S1041 and step S1042.

[0078] In step S1041 , the target sample is input into the target large language model, and the target large language model is enabled to obtain a labeling result corresponding to the target sample based on predefined personality and stance information and the review rules.

[0079] The target samples include representative positive and negative samples of various categories.

[0080] In step S1042 , the annotation result after the obtained annotation result is reviewed is obtained. The review process makes the annotation result have a fixed output style and format, and the annotation result after the review process is used as an example.

[0081] Continuing with the first example, we selected representative positive and negative examples from each category from the database. Based on the definitions of personality, requirements, sensitive elements, official stance, and the PoT program code, we ran the large language model to generate annotation results. Through manual review, the annotation results were modified to a fixed output style and format, and the reviewed annotation results were used as examples, making the annotation interpretations generated by the large language model more accurate and reasonable.

[0082] According to the method of the embodiment of the present application, by training the large language model, the target large language model learns how to obtain labeling results based on predefined personality and stance information and the labeling rules, thereby improving the consistency between the machine labeling results and the manual labeling results obtained by the large language model, improving the accuracy of machine labeling, reducing the labor cost of manual labeling, and improving data labeling efficiency; through the reasoning method based on the thinking program, the labeling rules are converted into an executable program, the complex labeling problems are structured, and the consistency between the machine labeling results and the manual labeling is further improved, and the accuracy is improved, and the results can be traced and checked back according to the results of each step of the program execution.

[0083] In addition, an embodiment of the present application further provides a device for annotating data, the structure of which is shown in the figure. The device includes: a device for performing predefined processing in a target large language model (hereinafter referred to as "predefinition device 101"), a device for inputting preset annotation rules into the target large language model (hereinafter referred to as "rule input device 102"), and a device for inputting the data to be annotated into the target large language model, so that the target large language model obtains corresponding annotation result information based on predefined personality and position information and the annotation rules (hereinafter referred to as "data annotation device 103").

[0084] Reference Figure 2, the predefinition device 101 performs predefinition processing in the target large language model.

[0085] Specifically, predefined processing is performed by inputting instruction text into a target large language model.

[0086] The pre-defined processing includes defining a persona and defining stance information that conforms to the defined persona.

[0087] The target large language model is a generative large language model. The target large language model according to the embodiment of the present application is used to annotate data and output corresponding annotation results.

[0088] Character design refers to the personality traits, appearance, behavior, and character traits that are set for a character during the creation, writing, and role-playing of a virtual character. Character design can be achieved by specifying various factors, such as the character's gender, age, occupation, hobbies, and language style.

[0089] For example, in a large language model, the character's personality traits can be set or gradually revealed during the conversation by inputting command text, making it more in line with user needs during the conversation.

[0090] Optionally, the predefined process also includes defining data annotation requirements.

[0091] The data annotation requirements are used to indicate the aspects and information from which the target large language model needs to make judgments, and what conclusions it ultimately gives.

[0092] Optionally, the predefining device 101 indicates the data format or data structure of the annotation results that need to be output by the target large language model by defining data annotation requirements.

[0093] Explicit instructions will enable the target large language model to correctly understand the instructions and generate expected results. It can also help narrow the scope of generated text and avoid generating text that is irrelevant to the topic or contains a lot of irrelevant information.

[0094] The stance information includes various information that can be used by the target large language model to learn the stance and attitude of a predefined persona towards specific things or behaviors.

[0095] According to the first example, in a scenario where comments on a manuscript are reviewed, it is desired to use a large language model to output annotation results.

[0096] The pre-definition device 101 defines a persona by inputting the following dialogue into the large language model: "If you are a community review expert, please determine whether the comments and manuscripts contain any content related to violations of laws and regulations, as well as the position of the comments, based on the definitions of violations of laws and regulations, comment content, manuscript content, and sentiment. Then, make a security judgment based on the return value of the Python code." By inputting the above dialogue into the large language model, the large language model is set as the persona of a security review expert, limiting the specific domain, style, and type of the generated results, making the generated explanatory annotations more focused.

[0097] The predefining device 101 defines the sensitive element "speech related to violations of laws and regulations, mainly referring to support and promotion of various behaviors that violate Chinese laws and regulations" in the large language model by inputting the following dialogue.

[0098] The pre-definition device 101 defines a stance consistent with the persona (review expert) by inputting the following dialogue into the large language model: "All illegal and irregular behaviors are prohibited." Based on this definition, comments that deliberately mention illegal behaviors or incite them are inconsistent with this stance; comments that criticize and encourage illegal and irregular behaviors are consistent with this stance.

[0099] Through the above settings and processing, the large language model can determine whether there are relevant violation categories and sensitive elements involved in the violations in the comments and manuscript content of the input video manuscripts for review, and determine whether the emotional tendency of the comments towards the existing sensitive elements is consistent with the set position.

[0100] Next, continue Figure 1 To illustrate, the rule input device 102 inputs the preset tagging rules into the target large language model.

[0101] The marking rules are determined based on whether one or more objects to be reviewed violate laws and regulations, as well as information indicating the position of the objects to be reviewed. The objects to be reviewed are pre-defined data that needs to be determined to be illegal or in violation of regulations. For example, for a video manuscript to be reviewed, the objects to be reviewed may include the manuscript content and the comments.

[0102] The position indication information is used to indicate whether the position of the subject to be reviewed is consistent with the position corresponding to the predefined personality.

[0103] The device determines whether the position of the audit object is consistent with the position corresponding to the predefined persona by performing the following operations:

[0104] The target large language model obtains the emotional tendency information of the text content of the target sample towards the element to be reviewed by performing sentiment analysis on the object to be reviewed. The emotional tendency information includes various information that can be used to indicate emotional attitudes towards specific things or behaviors. For example, "support", "oppose", "criticism", "disgust", etc. The emotional analysis process can be achieved by identifying whether predetermined keywords are included. Then, based on the emotional tendency information, the position of the object to be reviewed is analyzed to obtain the position indication information of the object to be reviewed, and then determine whether it is consistent with the predefined position information of the person.

[0105] According to one embodiment, the apparatus comprises program conversion means.

[0106] The rule input device 102 converts the audit rule into an executable program based on the rule logic contained in the annotation rule and the reasoning method of PoT. The executable program can be run on a program interpreter such as Python.

[0107] PoT is a reasoning framework that simulates the human decision-making process, breaking down complex thinking and judgment tasks into a series of ordered steps. Each step corresponds to a clear semantic unit and decision variable, making the entire reasoning process clearer and easier to track. Those skilled in the art will be familiar with setting or updating audit rules and adjusting PoT reasoning steps accordingly based on different tasks and needs.

[0108] Continuing with the first example, the rules for this example based on whether the comment content violates regulations, whether the manuscript content violates regulations, and whether the comment content complies with the official position are shown in Table 1 above.

[0109] Based on the rules shown in the table above, the rule conversion device takes content elements, manuscript content elements, and stance as three features, and designs the following PoT program code based on the judgment results obtained under different values, which is a class decision tree:

[0110] def main(comment content, manuscript content, comment stance):

[0111] If the comment content == "illegal and in violation of regulations" and the comment stance == "inconsistent with the pre-set stance":

[0112] ans="Rejected due to illegal or irregular behavior"

[0113] else:

[0114] If the article content == "illegal and in violation of regulations" and the comment stance == "inconsistent with the pre-set stance":

[0115] ans="Rejected due to illegal or irregular behavior"

[0116] else:

[0117] ans="pass"

[0118] return ans

[0119] The large language model can generate annotations based on these judgments and the PoT program code above. The PoT in this example provides a clearer, more expressive, and more fundamental model for answer derivation, improving accuracy and comprehension, especially for mathematical logic problems that require numerical calculations.

[0120] Continue to refer to the following Figure 1 To illustrate, the data annotation device 103 inputs the data to be annotated into the target large language model, and the target large language model obtains corresponding annotation result information based on the predefined personality and position information and the annotation rules. The annotation result information includes the illegality judgment and corresponding explanation of the data to be annotated.

[0121] Among them, those skilled in the art should be familiar with the fact that the predefined personality, stance information or labeling rules in the target large language model can be modified so that the target large language model can obtain corresponding labeling result information based on the updated personality, stance information and / or labeling rules.

[0122] Continuing with the first example, based on the aforementioned operations, a prompt framework for machine annotation rules based on programmatic thinking (PoT) is obtained. Data annotation device 103 can use this prompt framework to annotate large quantities of data. Furthermore, by sampling the obtained annotation results and submitting them for manual re-annotation, the accuracy rate can be calculated, incorrectly annotated data can be identified, and the prompt framework can be modified.

[0123] According to one embodiment, the apparatus further comprises result normalization means.

[0124] The result normalization device normalizes the annotation results output by the target large language model by adjusting the instruction text input to the target large language model.

[0125] Through the normalization process, the annotation results output by the target large language model have a fixed output style and format.

[0126] Specifically, the result standardization device includes a sample input device and a review processing device.

[0127] The sample input device inputs the target sample into the target large language model, so that the target large language model obtains the labeling result corresponding to the target sample based on the predefined personality and stance information and the review rules.

[0128] The target samples include representative positive and negative samples of various categories.

[0129] The review processing device obtains the annotation result after review processing is performed on the obtained annotation result, wherein the review processing makes the annotation result have a fixed output style and format, and uses the annotation result after review processing as an example.

[0130] Continuing with the first example, we selected representative positive and negative examples from each category from the database. Based on the definitions of personality, requirements, sensitive elements, official stance, and the PoT program code, we ran the large language model to generate annotation results. Through manual review, the annotation results were modified to a fixed output style and format, and the reviewed annotation results were used as examples, making the annotation interpretations generated by the large language model more accurate and reasonable.

[0131] According to the device of the embodiment of the present application, by training the large language model, the target large language model learns how to obtain labeling results based on predefined personality and stance information and the labeling rules, thereby improving the consistency between the machine labeling results and the manual labeling results obtained by the large language model, improving the accuracy of machine labeling, reducing the labor cost of manual labeling, and improving data labeling efficiency; through the reasoning method based on the thinking program, the labeling rules are converted into an executable program, the complex labeling problems are structured, and the consistency between the machine labeling results and the manual labeling is further improved, and the accuracy is improved. In addition, the results can be traced and checked back according to the results of each step of the program execution.

[0132] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application. The method corresponding to the electronic device may be the method for annotating data in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0133] The electronic device may be a user device, or a device formed by integrating a user device and a network device via a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablets, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer collections, and can be used to implement some of the processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computers.

[0134] Figure 3 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown. The device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 into the random access memory (RAM) 1203. Various programs and data required for system operation are also stored in RAM 1203. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through a bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0135] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, and a speaker; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, and a semiconductor memory; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1209 performs communication processing via a network such as the Internet.

[0136] In particular, the methods and / or embodiments of the present application can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above-mentioned functions defined in the method of the present application are performed.

[0137] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.

[0138] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0139] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0140] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0141] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0142] The flow chart or block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the equipment, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code include one or more executable instructions for realizing the logical function of the specification. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs the function or operation of the specification, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0144] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0145] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0146] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0147] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0149] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

Claims

1. A method for labeling data, wherein: The method comprises: Performing pre-definition processing in the target large language model, the pre-definition processing including defining a persona and defining stance information that conforms to the defined persona; Inputting preset labeling rules into the target large language model, wherein the labeling rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and the position indication information of the objects to be reviewed; Input the data to be annotated into the target large language model, and let the target large language model obtain corresponding annotation result information based on the predefined personality and stance information and the annotation rules; The method determines whether the position of the audit object is consistent with the position corresponding to the predefined persona through the following steps: The target large language model performs sentiment analysis on the object to be reviewed to obtain the sentiment tendency information of the text content of the target sample towards the object to be reviewed; The stance of the subject to be audited is analyzed based on the emotional tendency information to obtain stance indication information of the subject to be audited, and then it is determined whether it is consistent with the stance information of the predefined personality.

2. The method according to claim 1, wherein The method further comprises: According to the rule logic contained in the annotation rule, the annotation rule is converted into an executable program based on the reasoning method of the thinking program.

3. The method according to claim 1, wherein The method further comprises: By adjusting the instruction text input to the target large language model, the annotation results output by the target large language model are normalized.

4. The method according to claim 3, wherein: The normalizing of the labeling results output by the target large language model by adjusting the instruction text input to the target large language model includes: Input the target sample into the target large language model, and let the target large language model obtain the labeling result corresponding to the target sample based on the predefined personality and stance information and the labeling rules; Obtain an annotation result after reviewing the obtained annotation result, wherein the review process makes the annotation result have a fixed output style and format, and use the annotation result after review as an example.

5. The method according to claim 1 or 2, wherein: The predefined processing in the target large language model includes: Predefined processing is performed by inputting instruction text into the target large language model.

6. The method according to claim 1 or 2, wherein the predefined processing further comprises defining data annotation requirements, and the predefined processing in the target large language model comprises: The data annotation requirements are defined by inputting instruction text into the target large language model. The data annotation requirements are used to indicate which aspects and information the target large language model needs to judge based on, and what conclusion it ultimately gives.

7. A device for labeling data, wherein: The device comprises: means for performing pre-defined processing in a target large language model, the pre-defined processing comprising defining a persona and defining stance information consistent with the defined persona; means for inputting preset labeling rules into a target large language model, wherein the labeling rules are determined based on whether one or more objects to be reviewed are illegal or in violation of regulations and information indicating the position of the objects to be reviewed; A device for inputting the data to be annotated into the target large language model, so that the target large language model obtains corresponding annotation result information based on predefined personality and stance information and the annotation rules; The following operations are used to determine whether the position of the audited object is consistent with the position corresponding to the predefined persona: The target large language model performs sentiment analysis on the object to be reviewed to obtain the sentiment tendency information of the text content of the target sample towards the object to be reviewed; The stance of the subject to be audited is analyzed based on the emotional tendency information to obtain stance indication information of the subject to be audited, and then it is determined whether it is consistent with the stance information of the predefined personality.

8. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images