Method and device for generating labels, medium and program product

Abstract

The invention provides a method and device for generating labels, electronic equipment, a computer readable medium and a computer program product. The method comprises the following steps: according to preset sensitive query information, obtaining first violation case information matched with the sensitive query information; performing case expansion processing on the first violation case information to obtain second violation case information; inputting background knowledge rule information matched with the first violation case information and / or the second violation case information into a target large language model; and for violation sample data, using the target large language model to carry out annotation processing based on the first violation case information, the second violation case information and the corresponding background knowledge rule information, and generating corresponding violation annotation information. According to the method, the recall capability of the large model of the generative machine-review text is improved, and the training requirements of the large model of the generative machine-review text are effectively met.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device, computer-readable medium, and computer program product for generating annotations. Background Art

[0002] Based on the existing technology, in the training of large generative machine-reviewed text models, the labeling of sensitive information becomes extremely difficult due to its obscurity. The labelers are required to have a complete knowledge background, and their number is relatively small, which makes it difficult to meet the data requirements for model training. Summary of the invention

[0003] Multiple aspects of the present application provide a method, an apparatus, an electronic device, a computer-readable medium, and a computer program product for generating annotations.

[0004] In one aspect of the present application, a method for generating a label is provided, wherein the method comprises:

[0005] According to preset sensitive query information, obtaining first violation case information matching the sensitive query information;

[0006] By performing case expansion processing on the first violation case information, second violation case information is obtained;

[0007] inputting background knowledge rule information matching the first violation case information and / or the second violation case information into the target large language model;

[0008] For the violation sample data, the target large language model is used to perform labeling processing based on the first violation case information, the second violation case information and the corresponding background knowledge rule information to generate corresponding violation labeling information.

[0009] In one aspect of the present application, a device for generating a label is provided, wherein the device comprises:

[0010] A device for obtaining first violation case information matching the sensitive query information according to preset sensitive query information;

[0011] A device for obtaining second violation case information by performing case expansion processing on the first violation case information;

[0012] means for inputting background knowledge rule information matching the first violation case information and / or the second violation case information into a target large language model;

[0013] A device for generating corresponding violation annotation information by using the target large language model to perform annotation processing on violation sample data based on first violation case information, second violation case information and corresponding background knowledge rule information.

[0014] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0015] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0016] In the solution provided in the embodiment of the present application, violation cases related to sensitive data are obtained as first case information, and the first case information is expanded to obtain more violation cases as second case information, and the background knowledge and rules matching the violation cases are informed to the generative large language model, so that it can generate accurate annotations and detailed explanations for the sample data based on the first case information, the second case information, the background knowledge and the rules, thereby improving the recall capability of the generative machine-reviewed text large model and effectively meeting the training needs of the generative machine-reviewed text large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 A schematic diagram of a process for generating annotations according to an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of the structure of a device for generating annotations provided in an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0022] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0024] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0025] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0026] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0027] Figure 1 A schematic flow chart of a method for generating annotations provided in an embodiment of the present application is shown. The method at least includes step S101, step S102, step S103 and step S104.

[0028] In actual scenarios, the execution subject of the method can be a network device, or an application running on a network device, wherein the network device includes but is not limited to a network host, a single network server, a plurality of network server sets, or a collection of computers based on cloud computing, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0029] In the machine review scenario, the method of the embodiment of the present application is implemented by a generative large language model. The embodiment of the present application obtains violation cases related to sensitive data, expands the violation cases to obtain more violation cases, and informs the generative large language model of the background knowledge and rules matching the violation cases, so that it can generate accurate annotations and detailed explanations for the sample data based on the violation cases, background knowledge and rules, effectively meeting the needs of generative machine review text large model training.

[0030] The concepts involved in the embodiments of the present application are explained below.

[0031] Large Language Model (LLM): Large language model is a commonly used model in the field of natural language processing. It is used to process a variety of natural language tasks, such as text classification, question answering, dialogue, etc., to generate natural language text or understand the meaning of language text. Large language model is also a general term for deep learning models trained with large amounts of text data. For example, GPT-3, PaLM, Galactica, LLaMA and other models are all large language models commonly used by technicians in this field.

[0032] Reference Figure 1 In step S101, according to preset sensitive query information, first violation case information matching the sensitive query information is obtained.

[0033] The sensitive query information includes various information that can be used to query violation cases.

[0034] The violation case information may include various types of text information, such as a text description of the violation.

[0035] Optionally, violation cases in the embodiments of the present application include, but are not limited to, cases in which reported comments, reported barrages, and reported tags are judged to be violations based on preset rules or review agencies.

[0036] Optionally, the violation case may also include source information or type information, etc. The source information may include but is not limited to all source cases, system banned cases, etc. Type information includes but is not limited to all type cases, comment cases, barrage cases, tag cases, or submission cases, etc.

[0037] Optionally, the sensitive query information includes preset sensitive words and violation tags. In step S101, a query is performed in a predetermined case library based on the preset sensitive words and violation tags, and the matching violation cases obtained by the query are used as the first violation case information.

[0038] Optionally, the obtained violation case is processed according to a predetermined format, and the processed case information in the predetermined format is used as the first violation case information.

[0039] For example, for violation cases of comment barrage, the corresponding video manuscript information (video title, tags, introduction) is added, and this information and the text information of the illegal comment barrage content are used as the first violation case information.

[0040] In step S102, second violation case information is obtained by performing case expansion processing on the first violation case information.

[0041] The method of expanding the first violation case information includes but is not limited to:

[0042] 1) Expand the case by rewriting the first violation case information;

[0043] Specifically, according to this method, step S102 includes step S1021.

[0044] In step S1021, a predetermined number of violation cases similar to the first violation case are generated as second violation case information by rewriting part of the violation case information.

[0045] There are many ways to rewrite. For example, some keywords in the text of the violation case can be replaced with their synonyms or near synonyms through synonym replacement. Another example is to replace the words or phrases in the original text with popular online memes or expressions commonly used in specific communities through Internet meme replacement. Another example is to change the background or environment of the violation case through context conversion.

[0046] 2) Perform case expansion processing by defining personas in the target large language model;

[0047] Specifically, according to this method, step S102 includes step S1022 and step S1023.

[0048] According to an embodiment, in step S1022, a target persona is defined in the target large language model. Optionally, the target persona includes various people who are good at processing text, such as a writer, a network expert, etc.

[0049] Among them, character setting refers to the personality characteristics, appearance description, behavior habits, character traits, etc. set for the character during the process of virtual character creation, text creation, role-playing, etc. The definition of character setting can be achieved by specifying the character's gender, age, occupation, hobbies, language style and other factors.

[0050] For example, in a large language model, the character's personality traits can be gradually revealed by inputting command text during the conversation, making it more in line with user needs during the conversation.

[0051] In step S1023, based on the target persona, a predetermined number of violation cases similar to the first violation case that conform to the target persona are generated in the target large language model as second violation case information.

[0052] 3) Case expansion processing is performed by combining the persona defined in the target large language model and rewriting the first violation case information;

[0053] Specifically, according to this method, step S102 includes step S1024 and step S1025.

[0054] According to one embodiment, in step S1024, a target persona is defined in a target large language model.

[0055] In step S1025, by inputting instructions to the target large language model, the target large language model is caused to rewrite part of the information of the first violation case based on the target persona, and generate a predetermined number of violation cases similar to the first violation case as second violation cases.

[0056] According to the first example of this application, in the scenario of machine review of comments on video manuscripts, this example screens in the case library based on preset sensitive words and violation tags to obtain matching violation cases, and uses a large language model to generate more violation cases based on the obtained violation cases. The violation cases in this example include the text information of the video manuscript comments that are judged to be illegal.

[0057] In this example, a prompt is input into the large language model to enable the large language model to generate similar violation comments to the violation comments in the violation case, and then generate a new violation case based on the obtained similar violation comments.

[0058] The specific rules are as follows: set up a seed comment group (the comments themselves are illegal), and then for the same manuscript, if the manuscript has more than 5 comments, add all the comments in the prompt sample demonstration of generating a violation case as shown below. If the manuscript has less than 5 comments, randomly select comments from the seed comment group to make up 5 comments.

[0059] In this example, the prompt structure for generating a new violation case is as follows:

[0060] The following is a piece of background material: {Fill in the text of the background material}.

[0061] The background material is used to introduce what kind of speech is illegal in the online environment, such as attacking or satirizing specific matters.

[0062] There is now a video from the xx partition.

[0063] The title is: {Fill in the title text content};

[0064] The tags are: {Fill in the tag text content};

[0065] The description is: {Fill in the description text content};

[0066] The root comment is: {Fill in the root comment text content};

[0067] Some viewers have left the following comments:

[0068] Comment 1: {Fill in the Comment 1 text content};

[0069] Comment 2: {Fill in the Comment 2 text content};

[0070] Comment 3: {Fill in the Comment 3 text content};

[0071] Comment 4: {Fill in the Comment 4 text content};

[0072] Comment 5: {Fill in the Comment 5 text content}.

[0073] The ways for the large language model to generate similar违规 comments in this example include:

[0074] 1) Define the persona of a surfing expert in the large language model, and let the large language model generate similar违规 comments based on this persona and in combination with the input background-related materials. Specifically, by inputting: "According to the background materials, the above 5 comments all need further review. If you are a surfing expert who likes to play with memes, please write 10 other违规 comments in combination with the background materials."

[0075] 2) Define the persona of a surfing expert in the large language model, and let the large language model generate similar违规 comments based on this persona, the input background-related materials, and in combination with variant descriptions of internet memes.

[0076] By inputting into the large language model "According to the background materials, the above 5 comments all need further review. If you are a surfing expert, your ability is to be good at playing with memes and you will describe internet memes in variants, for example, a variant of 'forever the god' can be 'yyds', and you can also create variants of other internet memes by yourself. Please write 10 other违规 comments in combination with the background materials", new违规 comments output by the large language model are obtained.

[0077] Continue to refer to the following Figure 1 For illustration, in step S103, the background knowledge rule information that matches the first违规 case information and / or the second违规 case information is input into the target large language model.

[0078] The background knowledge rule information includes various types of background knowledge information and rule information that can be used by the target large language model to judge violations, for example, text data such as material information, factual knowledge, etc. from different sources such as news, blogs, forums, etc.

[0079] Optionally, the method analyzes the first violation case and / or the second violation case to determine one or more sensitive elements involved, and then collects data related to the sensitive elements as background knowledge rule information matching the first violation case and the second violation case.

[0080] For example, for violation cases involving sensitive elements, relevant historical data and the latest developments are collected. Based on the collected data, existing rules are refined to ensure that they can accurately cover different types of violations. In addition, a detailed background knowledge base is established and maintained, which contains information such as political figures, events, organizations, and related terms. This knowledge base will serve as the basis for identifying and judging illegal content.

[0081] According to one embodiment, the method periodically obtains background knowledge rule information that matches the first violation case information and / or the second violation case information, and updates the background knowledge rule information input into the target large language model based on the obtained information.

[0082] In step S104, the target large language model is used to perform annotation processing on the violation sample data based on the first violation case information, the second violation case information and the corresponding background knowledge rule information to generate corresponding violation annotation information.

[0083] The violation marking information includes a violation determination result indicating a violation or non-violation. Optionally, the violation marking information also includes explanation and description information of the corresponding violation point. The explanation and description information may include specific description information of the violation point and the basis for determining the violation.

[0084] Among them, the data annotated by the target large language model can be used as training data to train the large model used for machine review.

[0085] Optionally, data annotation requirements are predefined in the target large language model.

[0086] The data annotation requirements are used to indicate the aspects and information from which the target large language model needs to make judgments, and what conclusions it ultimately gives.

[0087] Optionally, the method indicates the data format or data structure of the illegal annotation information that needs to be output by the target large language model by defining data annotation requirements. Optionally, the data format or data structure of the illegal annotation information is presented in a format that is easy for manual review and modification, facilitating manual verification and annotation processes.

[0088] Continuing to explain the first example above, a large amount of sample data is input into the large language model, and the large language model is allowed to generate violation annotation information of the comments based on the violation cases and the corresponding background knowledge / rules. Among them, the violation annotation information includes the violation annotation results and the explanation description of the violation points. Among them, the violation cases include the violation cases obtained based on the preset sensitive words and violation labels and the new violation cases generated by the large language model. The annotated data can be used for the training of the generative large model.

[0089] According to the method of the embodiment of the present application, by obtaining violation cases related to sensitive data as the first case information, and expanding the first case information to obtain more violation cases as the second case information, and informing the generative large language model of the background knowledge and rules matching the violation cases, it generates accurate annotations and detailed explanations for the sample data based on the first case information, the second case information, the background knowledge and the rules, thereby improving the recall capability of the generative machine-reviewed text large model and effectively meeting the training needs of the generative machine-reviewed text large model.

[0090] In addition, the present application also provides a device for generating annotations, the structure of which is as follows: Figure 2 As shown. The device includes: a device for obtaining first violation case information matching the sensitive query information according to preset sensitive query information (hereinafter referred to as "case obtaining device 101"), a device for obtaining second violation case information by performing case expansion processing on the first violation case information (hereinafter referred to as "case expansion device 102"), a device for inputting background knowledge rule information matching the first violation case information and / or the second violation case information into a target large language model (hereinafter referred to as "knowledge rule input device 103"), and a device for using the target large language model to perform annotation processing on violation sample data based on the first violation case information, the second violation case information and the corresponding background knowledge rule information to generate corresponding violation annotation information (hereinafter referred to as "annotation generation device 104").

[0091] Reference Figure 2 The case acquisition device 101 acquires first violation case information matching the sensitive query information according to the preset sensitive query information.

[0092] The sensitive query information includes various information that can be used to query violation cases.

[0093] The violation case information may include various types of text information, such as a text description of the violation.

[0094] Optionally, violation cases in the embodiments of the present application include, but are not limited to, cases in which reported comments, reported barrages, and reported tags are judged to be violations based on preset rules or review agencies.

[0095] Optionally, the violation case may also include source information or type information, etc. The source information may include but is not limited to all source cases, system banned cases, etc. Type information includes but is not limited to all type cases, comment cases, barrage cases, tag cases or submission cases, etc.

[0096] Optionally, the sensitive query information includes preset sensitive words and violation labels. The case acquisition device 101 searches a predetermined case library based on the preset sensitive words and violation labels, and uses the matching violation cases obtained by the search as the first violation case information.

[0097] Optionally, for the acquired violation case, the case acquisition device 101 processes it according to a predetermined format, and uses the processed case information in the predetermined format as the first violation case information.

[0098] For example, for violation cases of comment barrage, the corresponding video manuscript information (video title, tags, introduction) is added, and this information and the text information of the illegal comment barrage content are used as the first violation case information.

[0099] The case expansion device 102 obtains second violation case information by performing case expansion processing on the first violation case information.

[0100] The method in which the case expansion device 102 performs case expansion processing on the first violation case information includes but is not limited to:

[0101] 1) Expand the case by rewriting the first violation case information;

[0102] Specifically, the case expansion device 102 according to this method rewrites part of the information of the violation case to generate a predetermined number of violation cases similar to the first violation case as the second violation case information.

[0103] There are many ways to rewrite. For example, some keywords in the text of the violation case can be replaced with their synonyms or near synonyms through synonym replacement. Another example is to replace the words or phrases in the original text with popular online memes or expressions commonly used in specific communities through Internet meme replacement. Another example is to change the background or environment of the violation case through context conversion.

[0104] 2) Perform case expansion processing by defining personas in the target large language model;

[0105] Specifically, the case expansion device 102 according to this method defines a target persona in the target large language model. Optionally, the target persona includes various people who are good at processing text, such as a writer, a network expert, and the like.

[0106] Among them, character setting refers to the personality characteristics, appearance description, behavior habits, character traits, etc. set for the character during the process of virtual character creation, text creation, role-playing, etc. The definition of character setting can be achieved by specifying the character's gender, age, occupation, hobbies, language style and other factors.

[0107] For example, in a large language model, the character's personality traits can be gradually revealed by inputting command text during the conversation, making it more in line with user needs during the conversation.

[0108] Next, the case expansion device 102 generates a predetermined number of violation cases similar to the first violation case that conform to the target persona in the target large language model based on the target persona as second violation case information.

[0109] 3) Case expansion processing is performed by combining the persona defined in the target large language model and rewriting the first violation case information;

[0110] Specifically, the case expansion device 102 of this method defines a target persona in the target large language model. Then, by inputting instructions to the target large language model, the target large language model is made to rewrite part of the information of the first violation case based on the target persona, and generate a predetermined number of violation cases similar to the first violation case as the second violation case.

[0111] According to the first example of this application, in the scenario of machine review of comments on video manuscripts, this example screens in the case library based on preset sensitive words and violation tags to obtain matching violation cases, and uses a large language model to generate more violation cases based on the obtained violation cases. The violation cases in this example include the text information of the video manuscript comments that are judged to be illegal.

[0112] In this example, the case expansion device 102 inputs prompts to the large language model, so that the large language model generates similar violation comments to the violation comments in the violation case, and then generates a new violation case based on the obtained similar violation comments.

[0113] The specific rules are as follows: set up a seed comment group (the comments themselves are illegal), and then for the same manuscript, if the manuscript has more than 5 comments, add all the comments in the prompt sample demonstration of generating a violation case as shown below. If the manuscript has less than 5 comments, randomly select comments from the seed comment group to make up 5 comments.

[0114] In this example, the prompt structure for generating a new violation case is as follows:

[0115] The following is a piece of background material: {Fill in the text of the background material}.

[0116] The background material is used to introduce what kind of speech is illegal in the online environment, such as attacking or satirizing specific matters.

[0117] Now there is a video from the xx partition.

[0118] The title is: {fill in the title text};

[0119] The label is: {fill in the label text content};

[0120] The introduction is: {fill in the introduction text};

[0121] The root comment is: {fill in the root comment text};

[0122] An audience member made the following comment:

[0123] Comment 1: {fill in the text content of Comment 1};

[0124] Comment 2: {fill in the text content of Comment 2};

[0125] Comment 3: {fill in the text content of Comment 3};

[0126] Comment 4: {fill in the text content of comment 4};

[0127] Comment 5: {fill in the text content of Comment 5}.

[0128] In this example, the large language model generates similar illegal comments in the following ways:

[0129] 1) Define the persona of a surfing expert in the large language model, and let the large language model generate similar违规 comments based on this persona and in combination with the input background-related materials. Specifically, by inputting: "According to the background materials, the above 5 comments all require further review. If you are a surfing expert who likes to play with memes, please write 10 other违规 comments in combination with the background materials."

[0130] 2) Define the persona of a surfing expert in the large language model, and let the large language model generate similar违规 comments based on this persona, the input background-related materials, and in combination with variant descriptions of internet memes.

[0131] By inputting into the large language model "According to the background materials, the above 5 comments all require further review. If you are a surfing expert, your ability is to be good at playing with memes and will describe internet memes in variants, for example, the variant of 'forever the god' can be 'yyds', and you can also describe other internet memes in variants by yourself. Please write 10 other违规 comments in combination with the background materials", new违规 comments output by the large language model are obtained.

[0132] The following continues to refer to Figure 2 for illustration. The knowledge rule input device 103 inputs background knowledge rule information that matches the first违规 case information and / or the second违规 case information into the target large language model.

[0133] Among them, the background knowledge rule information includes various types of background knowledge information and rule information for the target large language model to judge违规, such as text data such as material information and factual knowledge from different sources such as news, blogs, and forums.

[0134] Optionally, the device analyzes the first违规 case and / or the second违规 case to determine one or more sensitive elements involved. Then, data related to this sensitive element is collected as background knowledge rule information that matches the first违规 case and the second违规 case.

[0135] For example, for违规 cases involving sensitive elements, relevant historical data and the latest event developments are collected. Based on the collected data, existing rules are refined to ensure that they can accurately cover different types of违规 situations. And, a detailed background knowledge base is established and maintained, which contains information such as political figures, events, organizations, and related terms. This knowledge base will serve as the basis for identifying and judging违规 content.

[0136] According to an embodiment, the device periodically obtains background knowledge rule information that matches the first违规 case information and / or the second违规 case information, and updates the background knowledge rule information input into the target large language model based on the obtained information.

[0137] The annotation generating device 104 uses the target large language model to perform annotation processing on the violation sample data based on the first violation case information, the second violation case information and the corresponding background knowledge rule information to generate corresponding violation annotation information.

[0138] The violation marking information includes a violation determination result indicating a violation or non-violation. Optionally, the violation marking information also includes explanation and description information of the corresponding violation point. The explanation and description information may include specific description information of the violation point and the basis for determining the violation.

[0139] Among them, the data annotated by the target large language model can be used as training data to train the large model used for machine review.

[0140] Optionally, data annotation requirements are predefined in the target large language model.

[0141] The data annotation requirements are used to indicate the aspects and information from which the target large language model needs to make judgments, and what conclusions it ultimately gives.

[0142] Optionally, the device indicates the data format or data structure of the illegal annotation information that needs to be output by the target large language model by defining data annotation requirements. Optionally, the data format or data structure of the illegal annotation information is presented in a format that is easy for manual review and modification, facilitating manual verification and annotation processes.

[0143] Continuing to explain the first example, the knowledge rule input device 103 inputs a large amount of sample data into the large language model, and the large language model generates violation annotation information of the comments based on the violation cases and the corresponding background knowledge / rules. The violation annotation information includes the violation annotation results and the explanation description of the violation points. The violation cases include the violation cases obtained based on the preset sensitive words and violation labels and the new violation cases generated by the large language model. The annotated data can be used for the training of the generative large model.

[0144] According to the device of the embodiment of the present application, by obtaining violation cases related to sensitive data as first case information, and expanding the first case information to obtain more violation cases as second case information, and informing the generative large language model of the background knowledge and rules matching the violation cases, it generates accurate annotations and detailed explanations for the sample data based on the first case information, the second case information, the background knowledge and the rules, thereby improving the recall capability of the generative machine-reviewed text large model and effectively meeting the training needs of the generative machine-reviewed text large model.

[0145] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the method for generating annotations in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0146] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0147] Figure 3 The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0148] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0149] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0150] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0151] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0152] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0153] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0154] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0156] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0157] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0158] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0160] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0161] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0162] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A method for generating annotations, wherein: The method comprises: According to preset sensitive query information, obtaining first violation case information matching the sensitive query information; By performing case expansion processing on the first violation case information, second violation case information is obtained; inputting background knowledge rule information matching the first violation case information and / or the second violation case information into the target large language model; For the violation sample data, the target large language model is used to perform labeling processing based on the first violation case information, the second violation case information and the corresponding background knowledge rule information to generate corresponding violation labeling information.

2. The method according to claim 1, wherein: The second violation case information obtained by performing case expansion processing on the first violation case information includes: Define the target persona in the target large language model; Based on the target persona, a predetermined number of violation cases similar to the first violation case that conform to the target persona are generated in the target large language model as second violation case information.

3. The method according to claim 2, wherein: The generating, in the target large language model based on the target persona, a predetermined number of violation cases that are similar to the first violation case and conform to the target persona as second violation case information includes: By inputting instructions into the target large language model, the target large language model is caused to rewrite part of the information of the first violation case based on the target persona, and generate a predetermined number of violation cases similar to the first violation case as the second violation cases.

4. The method according to claim 1, wherein: The second violation case information obtained by performing case expansion processing on the first violation case information includes: A predetermined number of violation cases similar to the first violation case are generated as second violation case information by rewriting part of the violation case information.

5. The method according to any one of claims 1 to 4, wherein: The method further comprises: Analyze the first violation case and / or the second violation case to determine the one or more sensitive elements involved; Data related to the sensitive element is collected as background knowledge rule information matching the first violation case and the second violation case.

6. The method according to claim 5, wherein: The method further comprises: Periodically acquiring background knowledge rule information matching the first violation case information and / or the second violation case information; The background knowledge rule information input to the target large language model is updated based on the acquired information.

7. The method according to claim 1, wherein: The sensitive query information includes preset sensitive words and violation labels, and obtaining first violation case information matching the sensitive query information according to the preset sensitive query information includes: A query is performed in a predetermined case library based on preset sensitive words and violation tags, and matching violation cases obtained through the query are used as first violation case information.

8. A device for generating annotations, wherein: The device comprises: A device for obtaining first violation case information matching the sensitive query information according to preset sensitive query information; Device for obtaining second violation case information by performing case expansion processing on the first violation case information; means for inputting background knowledge rule information matching the first violation case information and / or the second violation case information into a target large language model; A device for generating corresponding violation annotation information by using the target large language model to perform annotation processing on violation sample data based on first violation case information, second violation case information and corresponding background knowledge rule information.

9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 7.

10. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 7.

11. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images