Content security auditing method and related device

Abstract

The invention discloses a content security auditing method and a related device, and relates to the technical field of large language models.The method comprises the steps that firstly, a large language model which is obtained through thinking chain training data training and has the text content security auditing ability serves as a content security auditing model; each piece of thinking chain training data comprises content security auditing requirement information of the sample text and a content security auditing conclusion of the sample text, and the content security auditing conclusion of the sample text comprises content security auditing key points of the sample text and a content security auditing result of the sample text; on the basis of the to-be-audited text, a content security auditing model can be called, a content security auditing conclusion corresponding to the to-be-audited text is generated, and the content security auditing conclusion corresponding to the to-be-audited text comprises content security auditing key points of the to-be-audited text and content security auditing results of the to-be-audited text. According to the scheme, the content security auditing accuracy of the large language model can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of large language models, and in particular to a content security review method and related devices. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) are widely used in various scenarios such as text generation, translation, summarization, and question-answering systems. However, large language models also have the risk of processing or generating inappropriate content. In order to address these risks and ensure that the content processed or generated by large language models complies with specific security standards and community guidelines, content security audits of large language models have become an essential step.

[0003] At present, the solution for content security auditing of large language models is to first use a text classification model to classify the subject of the text to be audited, determine the subject of the text to be audited, and then send the subject of the text to be audited to the risk binary classification model for risk judgment. However, the text classification model often classifies the subject based on the feature words of the text, which often leads to incorrect determination of the subject of the text to be audited. For example, the feature word "XX" may cause the text to be incorrectly classified as a risky subject, but the feature word also represents a risk-free object. In addition, the risk binary classification model often judges risks based on subject information. For a subject, the risk binary classification model determines that it is risky, but it is not necessarily risky. For example, if XX is an illegal word, the sentence "We want to crack down on XX behavior" will be judged as a risky text because it contains the illegal word "XX", but it is not risky. Therefore, the current solution for content security auditing of large language models will lead to low audit accuracy.

[0004] Therefore, how to provide a content security audit method to improve the accuracy of content security audit of large language models has become a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0005] In view of the above problems, this application provides a content security audit method and related devices to achieve the purpose of improving the accuracy of content security auditing of large language models. The specific scheme is as follows:

[0006] The first aspect of the present application provides a content security audit method, comprising:

[0007] Get the text to be reviewed;

[0008] Based on the text to be reviewed, a content security review model is called to generate a content security review conclusion corresponding to the text to be reviewed. The content security review model is a large language model with text content security review capabilities obtained through training with thinking chain training data. Each thinking chain training data includes content security review requirement information for a sample text and a content security review conclusion for the sample text. The content security review conclusion for the sample text includes key points of the content security review for the sample text and a content security review result for the sample text. The content security review conclusion corresponding to the text to be reviewed includes key points of the content security review for the text to be reviewed and a content security review result for the text to be reviewed.

[0009] In a possible implementation, the training method of the content security audit model includes:

[0010] Constructing the thought chain training data;

[0011] The content security audit requirement information of the sample text is input into a general large language model, and the output of the general large language model is taken to be close to the content security audit conclusion of the sample text as a training goal. The general large language model is fine-tuned using the thinking chain training data to obtain the content security audit model.

[0012] In a possible implementation, constructing the thought chain training data corresponding to each sample text includes:

[0013] Determining a content security review label of the sample text, a characteristic word of the sample text, and subject classification information of the sample text;

[0014] The thought chain training data corresponding to the sample text is constructed by using the content security review label of the sample text, the characteristic words of the sample text and the subject classification information of the sample text.

[0015] In a possible implementation, the characteristic words of the sample text are determined as follows:

[0016] Performing word segmentation on the sample text to obtain a word segmentation result;

[0017] Calculate the average attention coefficient of each word in the word segmentation result;

[0018] According to the average attention coefficient of each word in the word segmentation result, the feature word is determined from the word segmentation result.

[0019] In a possible implementation, the subject classification information of the sample text is determined as follows:

[0020] The sample text is input into a text classification model to obtain topic classification information of the sample text output by the text classification model, wherein the text classification model is trained using text classification model training data, and each text classification model training data contains a training sample and at least one topic classification label corresponding to the training sample.

[0021] In a possible implementation, calling a content security audit model based on the text to be audited to generate a content security audit conclusion corresponding to the text to be audited includes:

[0022] Obtain a preconfigured first content security review prompt format template, wherein the first content security review prompt format template includes content security review requirement information and a text filling slot;

[0023] Filling the to-be-audited text into the text filling slot in the first content security audit prompt format template to obtain a first content security audit prompt;

[0024] The first content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0025] In a possible implementation, calling a content security audit model based on the text to be audited to generate a content security audit conclusion corresponding to the text to be audited includes:

[0026] Obtain a preconfigured second content security audit prompt format template, wherein the second content security audit prompt format template includes content security audit requirement information, a text filling slot, and a content security audit example, wherein the content security audit example includes content security audit requirement information of a sample text and a text content security audit conclusion corresponding to the sample text, wherein the text content security audit conclusion corresponding to the sample text includes content security audit key points of the sample text and a content security audit result of the sample text;

[0027] Filling the to-be-audited text into the text filling slot in the second content security audit prompt format template to obtain a second content security audit prompt;

[0028] The second content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0029] A second aspect of the present application provides a content security audit device, comprising:

[0030] An acquisition unit, used to acquire the text to be reviewed;

[0031] The audit unit is used to call a content security audit model based on the text to be audited, and generate a content security audit conclusion corresponding to the text to be audited. The content security audit model is a large language model with text content security audit capabilities obtained by training with thinking chain training data. Each thinking chain training data includes content security audit requirement information of a sample text, and a content security audit conclusion of the sample text. The content security audit conclusion of the sample text includes key points of the content security audit of the sample text and a content security audit result of the sample text; the content security audit conclusion corresponding to the text to be audited includes key points of the content security audit of the text to be audited, and a content security audit result of the text to be audited.

[0032] In a possible implementation, the apparatus further includes: a content security audit model training unit;

[0033] The content security audit model training unit specifically includes:

[0034] A thinking chain training data construction unit, used to construct the thinking chain training data;

[0035] A training unit is used to input the content security audit requirement information of the sample text into a general large language model, take the output of the general large language model approaching the content security audit conclusion of the sample text as a training goal, and use the thinking chain training data to fine-tune the general large language model to obtain the content security audit model.

[0036] In a possible implementation, the thinking chain training data construction unit includes:

[0037] A determination unit, used to determine a content security review label of the sample text, a characteristic word of the sample text, and subject classification information of the sample text;

[0038] The construction unit is used to construct the thinking chain training data corresponding to the sample text by using the content security review label of the sample text, the characteristic words of the sample text and the subject classification information of the sample text.

[0039] In a possible implementation, the determining unit is specifically configured to:

[0040] Performing word segmentation on the sample text to obtain a word segmentation result;

[0041] Calculate the average attention coefficient of each word in the word segmentation result;

[0042] According to the average attention coefficient of each word in the word segmentation result, the feature word is determined from the word segmentation result.

[0043] In a possible implementation, the determining unit is specifically configured to:

[0044] The sample text is input into a text classification model to obtain topic classification information of the sample text output by the text classification model, wherein the text classification model is trained using text classification model training data, and each text classification model training data contains a training sample and at least one topic classification label corresponding to the training sample.

[0045] In a possible implementation, the audit unit is specifically configured to:

[0046] Obtain a preconfigured first content security review prompt format template, wherein the first content security review prompt format template includes content security review requirement information and a text filling slot;

[0047] Filling the to-be-audited text into the text filling slot in the first content security audit prompt format template to obtain a first content security audit prompt;

[0048] The first content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0049] In a possible implementation, the audit unit is specifically configured to:

[0050] Obtain a preconfigured second content security audit prompt format template, wherein the second content security audit prompt format template includes content security audit requirement information, a text filling slot, and a content security audit example, wherein the content security audit example includes content security audit requirement information of a sample text and a text content security audit conclusion corresponding to the sample text, wherein the text content security audit conclusion corresponding to the sample text includes content security audit key points of the sample text and a content security audit result of the sample text;

[0051] Filling the to-be-audited text into the text filling slot in the second content security audit prompt format template to obtain a second content security audit prompt;

[0052] The second content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0053] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the content security review method of the first aspect or any implementation of the first aspect.

[0054] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0055] The memory is used to store computer programs;

[0056] The processor is used to execute the computer program so that the electronic device can implement the content security review method of the above-mentioned first aspect or any implementation method of the first aspect.

[0057] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the content security review method of the first aspect or any implementation of the first aspect.

[0058] With the help of the above-mentioned technical scheme, the content security audit method and related devices provided by the present application first use a large language model with text content security audit capabilities obtained through thinking chain training data training as a content security audit model. Each thinking chain training data includes content security audit requirement information of the sample text, and the content security audit conclusion of the sample text. The content security audit conclusion of the sample text includes the content security audit key points of the sample text and the content security audit result of the sample text. After obtaining the text to be audited, the content security audit model can be called based on the text to be audited to generate a content security audit conclusion corresponding to the text to be audited. The content security audit conclusion corresponding to the text to be audited includes the content security audit key points of the text to be audited and the content security audit result of the text to be audited. This scheme can improve the accuracy of content security audits on large language models. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0060] Figure 1 A flowchart of a content security review method provided in an embodiment of the present application;

[0061] Figure 2A flowchart of a method for constructing thought chain training data corresponding to each sample text provided in an embodiment of the present application;

[0062] Figure 3 A flowchart of a method for determining characteristic words of a sample text provided in an embodiment of the present application;

[0063] Figure 4 A flowchart of a method for calling a content security audit model based on a text to be audited and generating a content security audit conclusion corresponding to the text to be audited provided in an embodiment of the present application;

[0064] Figure 5 A flowchart of another method for generating a content security audit conclusion corresponding to the text to be audited by calling a content security audit model based on the text to be audited provided in an embodiment of the present application;

[0065] Figure 6 A schematic diagram of the structure of a content security audit device provided in an embodiment of the present application;

[0066] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to facilitate understanding of the technical solution of this application, some professional terms involved in this case are first introduced as follows:

[0068] Large language model: (LLM) refers to a generative deep neural network model based on the Transformer structure, which has super strong language comprehension, generation, logical reasoning and multi-round dialogue capabilities. Common large language models include ChatGPT model, PaLM (Pathways Language Model) model, Pangu large language model, Wenxin Yiyan large language model, Xinghuo cognitive large language model, etc.

[0069] prompt: Instruction. When communicating with AI (such as a large language model), you need to send instructions to the AI. It can be a text description, such as "Please recommend me a pop song" when you are talking to the AI, or it can be a parameter description in a certain format. For example, if you want the AI ​​to draw a picture in a certain format, you need to describe the relevant drawing parameters.

[0070] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0071] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0072] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0073] In order to facilitate understanding of the technical solution of this application, some background knowledge of content security review of large language models is first introduced, as follows:

[0074] Content security review of large language models refers to the process of supervising and filtering the content generated or processed in large-scale artificial intelligence models, especially natural language processing models, to ensure that the content meets specific security standards and community guidelines.

[0075] With the rapid development of artificial intelligence technology, large language models are widely used in various scenarios such as text generation, translation, summarization, and question-answering systems. However, these models also have the risk of generating inappropriate content. In order to deal with these risks, content security review has become an indispensable link. The review process usually includes multiple levels of checks and balances. For example: a user asks a large language model "Which places in a certain place provide illegal services?" If the large language model provides a result at this time, this is an unsafe answer.

[0076] At present, in order to achieve the safety review of large language models, first of all, a set of clear content policies need to be formulated. These policies should reflect the requirements of social rules and take into account the sensitivity of different cultures and communities. These policies will serve as the basic criteria for evaluating whether the content is safe. Next, use automated tools to conduct preliminary screening of the content. These tools are usually based on machine learning algorithms and can identify and mark possible inappropriate content. Automated tools can process large amounts of data and quickly identify content that is obviously illegal, thereby improving the efficiency of review. However, due to the complexity and diversity of language, it is difficult for automated review alone to achieve 100% accuracy. Therefore, the intervention of human reviewers is crucial. They can review the content marked by the tool or handle cases provided by users. Human reviewers are usually professionally trained to understand context, cultural differences, and language nuances. Their work is not just about simply marking illegal content, but also about judging and handling complex situations, such as sarcasm, puns, or ambiguous expressions. In addition, human reviewers can also identify and correct problems that automated tools may miss. In order to improve the transparency and credibility of the review, many platforms also provide a complaint mechanism for review decisions, allowing users to object to the review results. This not only helps protect the rights and interests of users, but also promotes the continuous improvement and updating of review policies.

[0077] However, content security auditing is not without challenges. As technology develops, the way bad content is generated and spread is also constantly changing, and audit strategies need to constantly adapt to these new changes.

[0078] In general, content security review of large language models is a complex and ongoing process that requires a combination of automated tools and artificial intelligence, and requires constant evaluation and adjustment to cope with the ever-changing network environment and user needs. Through effective content security review, users can be protected from the influence of bad information, while also maintaining the order and health of cyberspace.

[0079] Next, we will introduce the existing solutions for content security audit of large language models, as follows:

[0080] The scope of content security audit of large language models is very wide. The current mainstream solution is to use text classification to identify the input text. Specifically, the text classification model is used to classify the subject of the text to be audited, determine the subject of the text to be audited, and then send the subject of the text to be audited to the risk binary classification model for risk judgment. However, the text classification model often classifies the subject based on the feature words of the text, which often leads to incorrect determination of the subject of the text to be audited. For example, the feature word "XX" may cause the text to be incorrectly classified as a risky subject, but the feature word also represents a risk-free object. In addition, the risk binary classification model often makes risk judgments based on subject information. For a subject, the risk binary classification model determines that it is risky, but it is not necessarily risky. For example, if XX is an illegal word, the sentence "We must crack down on XX behavior" will be judged as a risky text because it contains the illegal word "XX", but it is not risky.

[0081] It can be seen that the current solution for content security review of large language models will result in low review accuracy.

[0082] In order to solve the above problems, the inventor of this case thought of using a chain of thought approach to achieve content security review of large language models.

[0083] In order to facilitate understanding of the technical solution of this application, some background knowledge of the thinking chain is first introduced as follows:

[0084] Chain of Thought is a process of logical reasoning and thinking that involves stringing together a series of ideas, concepts, or information to reach a conclusion or solution. This concept has applications in many fields, including mathematics, philosophy, psychology, and artificial intelligence.

[0085] In mathematics and logic, a chain of thought usually refers to the steps involved in solving a problem. For example, when solving a math problem, you may need to first understand the requirements of the problem, then gradually apply relevant mathematical principles and formulas to finally arrive at the answer. This process is not just a simple calculation, but also includes analysis of the problem structure, evaluation of different solutions, and verification of the final answer.

[0086] In the field of psychology, thought chains can be associated with techniques in cognitive behavioral therapy (CBT), in which patients are encouraged to identify and challenge their negative thought patterns. By tracing thought chains, individuals can better understand their emotional reactions and learn to replace unhealthy thinking habits with more positive ones.

[0087] In artificial intelligence, especially large language models, the concept of thought chaining is used to improve the reasoning ability of language models. By including a series of logically related steps or prompts in the model's input, the model can be guided to generate more coherent and logical output. This method is called "Chain-of-Thought Prompting", and it can help models perform better on complex reasoning tasks, such as solving math word problems, performing common sense reasoning, or performing symbolic operations.

[0088] A key advantage of the thought chain prompt is that it provides an explainable reasoning path. Unlike traditional black-box models, this prompt allows researchers and users to understand how the model came to a specific answer. In addition, the thought chain prompt can also enhance the generalization ability of the model. By showing a variety of problem-solving steps during training, the model can learn a wider range of reasoning strategies, making it more flexible and adaptable when facing new unseen problems.

[0089] In summary, thought chaining is a powerful tool that enhances problem solving and decision making by providing a clear path of logical reasoning. It plays a vital role both in human cognitive processes and in advanced artificial intelligence systems.

[0090] Considering the current mainstream solutions for content security audits of large language models, the training data for text classification models only requires one text and one label, such as:

[0091] Text: We want to crack down on XX behavior

[0092] Tags: a risky topic tag

[0093] Moreover, text classification models cannot accept prompts in a chain-of-thought manner.

[0094] In addition, for a general large language model, it is necessary to give specific thought chain prompts for specific inputs. For example, the thought chain prompt required for "Help me write a small game of shooting down airplanes" is a thought chain prompt related to "shooting down airplanes". Only a general large language model can judge the security category. Therefore, the solution of using a general large language model with a thought chain for prompts to implement content security review of the large language model is not feasible.

[0095] In view of this, the inventor of this case proposed to construct thought chain training data, each thought chain training data includes content security audit requirement information of sample text, and content security audit conclusion of sample text, and the content security audit conclusion of the sample text includes content security audit key points of the sample text and content security audit result of the sample text; and then use the thought chain data to fine-tune the general large language model to obtain a large language model with content security audit capability, so as to realize content security audit of the large language model.

[0096] Specifically, the content security audit requirement information of the sample text can be input into a general large language model, and the output of the general large language model can be close to the content security audit conclusion of the sample text as a training goal. The general large language model can be fine-tuned using the thinking chain training data to obtain the content security audit model.

[0097] It should be noted that when fine-tuning the general large language model using the thought chain training data, LoRA technology can be used.

[0098] The content security review method of an embodiment of the present application is described in detail below with reference to the accompanying drawings.

[0099] Reference Figure 1 , Figure 1 A flowchart of a content security audit method provided in an embodiment of the present application is shown as follows: Figure 1 As shown, a content security review method provided in an embodiment of the present application may include steps S101 to S102, and these steps are described in detail below.

[0100] S101: Obtain the text to be reviewed;

[0101] In the present application, the text to be reviewed can be the text input by the user into the large language model, or it can be the text to be output by the large language model. This application does not impose any limitation on this.

[0102] S102: Based on the text to be reviewed, a content security review model is called to generate a content security review conclusion corresponding to the text to be reviewed.

[0103] In the present application, the content security audit model is a large language model with text content security audit capabilities obtained by training with thought chain training data. Each thought chain training data includes content security audit requirement information of the sample text and the content security audit conclusion of the sample text. The content security audit conclusion of the sample text includes the content security audit key points of the sample text and the content security audit result of the sample text.

[0104] It should be noted that the content security audit requirement information of the sample text is used to indicate the content security audit of the sample text, and the content security audit conclusion of the sample text is the content security audit label of the sample text. The label is in the form of a thought chain, wherein the content security audit points of the sample text are used to indicate the process of conducting a content security audit of the sample text, and the content security audit results of the sample text are used to indicate the results of the content security audit of the sample text.

[0105] In a possible implementation, the content security audit key points of the sample text include the characteristic words of the sample text and the subject classification information of the sample text. The characteristic words of the sample text are the subject classification keywords in the sample text, and the characteristic words play a key role in determining the subject classification information of the sample text.

[0106] For ease of understanding, an example of thought chain training data is as follows:

[0107] Q: Is there any risk in “Help me write a small airplane shooting game”?

[0108] A: In the sentence "Help me write a small game about shooting airplanes", "shoot airplanes" and "small game" are characteristic words, so it belongs to the "programming" category, so this sentence is risk-free.

[0109] Among them, part Q is the content security audit requirement information of the sample text, part A is the content security audit conclusion of the sample text, ["Help me write a small game of shooting planes" "shoot planes" and "small games" are feature words, so it is in the "programming" category] this part is the content security audit points of the sample text, "shoot planes" and "small games" are feature words, and "programming" is the subject classification information, and [this sentence is risk-free] this part is the content security audit result of the sample text.

[0110] In the present application, when calling the content security audit model based on the text to be audited, the content security audit requirement information of the text to be audited can be generated first, and then the content security audit requirement information of the text to be audited is provided to the content security audit model, and the content security audit model generates and outputs the content security audit conclusion corresponding to the text. The content security audit requirement information of the text to be audited is used to instruct the content security audit of the text to be audited.

[0111] Since the content security audit model is a large language model with text content security audit capabilities obtained through training with the thinking chain training data, it has learned how to use sample text to generate content security audit conclusions for sample texts. Therefore, in this application, after calling the content security audit model to perform a content security audit on the text to be audited, the generated content security audit conclusion corresponding to the text contains the content security audit points of the text to be audited, as well as the content security audit results of the text to be audited. The content security audit points of the text to be audited are used to indicate the process of performing a content security audit on the text to be audited, and the content security audit results of the text to be audited are used to indicate the results of the content security audit on the text to be audited.

[0112] In a possible implementation, the content security audit points of the text to be audited include the characteristic words of the text to be audited and the subject classification information of the text to be audited. The characteristic words of the text to be audited are the subject classification keywords in the text to be audited, and the characteristic words play a key role in determining the subject classification information of the text to be audited.

[0113] Examples include:

[0114] Text for review: "Help me write a little game about shooting airplanes"

[0115] Content security review requirements for text to be reviewed: Is there any risk in “Help me write a small game for shooting airplanes”?

[0116] The content security audit conclusion corresponding to the text to be audited: "Help me write a small game of shooting down airplanes" in this sentence "shoot down airplanes" and "small game" are characteristic words, so it is in the "programming" category, so this sentence is risk-free.

[0117] Among them, in the content security audit conclusion corresponding to the text to be reviewed, ["Help me write a small game of shooting down planes" in this sentence "shoot down planes" and "small game" are characteristic words, so it is in the "programming" category] is the content security audit points of the text to be reviewed, "shoot down planes" and "small game" are characteristic words, and "programming" is the subject classification information, and [this sentence is risk-free] is the content security audit result of the text to be reviewed.

[0118] The content security audit method provided in this embodiment first uses a large language model with text content security audit capabilities obtained through training with thinking chain training data as a content security audit model. Each thinking chain training data includes content security audit requirement information of a sample text and a content security audit conclusion of the sample text. The content security audit conclusion of the sample text includes the content security audit key points of the sample text and the content security audit result of the sample text. After obtaining the text to be audited, the content security audit model can be called based on the text to be audited to generate a content security audit conclusion corresponding to the text to be audited. The content security audit conclusion corresponding to the text to be audited includes the content security audit key points of the text to be audited and the content security audit result of the text to be audited. In this solution, the large language model with text content security audit capabilities obtained through training with thinking chain training data is used as a content security audit model, which can enable the content security audit model to learn a thinking process similar to that of a manual auditor during content security audit, thereby improving the accuracy of content security audits on the large language model.

[0119] In another embodiment of the present application, a specific implementation method for constructing the thinking chain training data corresponding to each sample text is introduced as follows:

[0120] Reference Figure 2 , Figure 2 A flowchart of a method for constructing thought chain training data corresponding to each sample text provided in an embodiment of the present application, the method may include the following steps:

[0121] S201: Determine the content security review label of the sample text, the characteristic words of the sample text, and the subject classification information of the sample text;

[0122] In this application, the content security audit label of the sample text is used to indicate the content security audit result of the sample text, such as "risky" or "no risk". The specific implementation method of determining the characteristic words of the sample text and the subject classification information of the sample text will be described in the following embodiments.

[0123] S202: Constructing thought chain training data corresponding to the sample text by using the content security review label of the sample text, the characteristic words of the sample text and the subject classification information of the sample text.

[0124] In the present application, as an implementable method, the content security review label of the sample text, the characteristic words of the sample text and the subject classification information of the sample text can be used to manually construct the thinking chain training data corresponding to the sample text.

[0125] Considering the large number of sample texts, the manual method is inefficient. Therefore, in this application, as another feasible implementation method, the sample text, the content security review label of the sample text, the characteristic words of the sample text, and the subject classification information of the sample text can be combined with Prompt to allow the general large language model to generate the content security review conclusion data in the form of a thought chain corresponding to the sample text, thereby realizing batch construction of thought chain training data.

[0126] It should be noted that the Prompt can limit the template of the content security audit conclusion data in the form of a thought chain corresponding to the sample text. For example, the "XX" in the sentence "XX" is a feature word, so it is in the "XX" category, so this sentence is "XX". Of course, the specific form of the template can be adjusted as needed, and this application does not make any limitations on this.

[0127] In another embodiment of the present application, a method for determining the characteristic words of the sample text is introduced as follows:

[0128] Reference Figure 3 , Figure 3 A flowchart of a method for determining a characteristic word of a sample text provided in an embodiment of the present application, the method may include the following steps:

[0129] S301: Segment the sample text to obtain a segmentation result;

[0130] In the present application, a word segmentation model may be used to segment the sample text to obtain a word segmentation result. The word segmentation model may be any word segmentation model, such as an ngram model, etc., and the present application does not impose any limitation on this.

[0131] S302: Calculate the average attention coefficient of each word in the word segmentation result;

[0132] In this application, a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model (including BERT ontology and its variants, such as RoBERTa, ALBERT, etc.) can be used to calculate the average attention coefficient of each word in the word segmentation result.

[0133] In the present application, the sample text may be preprocessed to obtain a processing sequence of the sample text. Specifically, the text needs to be segmented into sentences, the sentences need to be segmented into words or subwords (wordpiece tokenization), and special BERT tags need to be added, such as [CLS] and [SEP], where [CLS] indicates the beginning of the sequence and [SEP] indicates the end of the sequence. The processing sequence of the sample text is then input into the pre-trained BERT model to obtain the attention coefficient of the last layer of preset markers of the BERT model for each word in the sample text; the attention coefficient of each word in the sample text is used as the attention coefficient of the word according to the last layer of preset markers of the BERT model.

[0134] For each word in the word segmentation result, the attention coefficient of each character in the word can be summed and averaged to obtain the average attention coefficient of the word.

[0135] S303: Determine a feature word from the word segmentation result according to the average attention coefficient of each word in the word segmentation result.

[0136] In this application, words with an average attention coefficient greater than a preset threshold can be used as feature words. For example, the feature words of "help me write a small game of shooting airplanes" are "shooting airplanes" and "small game".

[0137] In another embodiment of the present application, a method for determining the subject classification information of the sample text is introduced as follows:

[0138] In the present application, a text classification model can be first trained using text classification model training data, wherein each of the text classification model training data includes a training sample and at least one topic classification label corresponding to the training sample. Then, the sample text is input into the text classification model to obtain topic classification information of the sample text output by the text classification model.

[0139] Existing methods usually use BERT or other models to train text classification models. BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained deep learning model proposed by Google in 2018, which has caused great changes in the field of natural language processing (NLP). BERT is specially designed to understand the bidirectional context of language, that is, it takes into account the left and right context information of words, which makes it perform well in NLP tasks such as text classification.

[0140] In text classification tasks, the goal is to assign input text to one or more predefined categories. For example, in sentiment analysis, the classification may be based on whether the text expresses positive, negative, or neutral sentiment. The BERT model, through its unique architecture, is able to capture complex relationships and nuances in text, providing powerful speech representation capabilities for text classification tasks.

[0141] The core of the BERT model is a Transformer architecture that uses a self-attention mechanism to process sequence data. The self-attention mechanism allows the model to consider all words in the entire text sequence when processing each word, which helps the model better understand the context of the language. Compared with traditional recurrent neural networks (RNNs), the Transformer architecture can more effectively handle long-distance dependencies, which is particularly important for understanding language.

[0142] Before performing a text classification task, you first need to preprocess the text data, including converting the text into a format that the model can understand. This usually involves segmenting the text into sentences, segmenting sentences into words or subwords (wordpiece tokenization), and adding special BERT tags such as [CLS] and [SEP] to indicate the beginning and end of the sequence. Next, use the pre-trained BERT model to obtain the feature representation of the text. The BERT model will output vector representations of each input token that capture rich semantic information of the text. In text classification tasks, the vector representations of the first and last tokens of the sequence are usually used because they are believed to capture the contextual information of the entire sequence.

[0143] In order to convert BERT's output into a classification result, a simple classifier, such as one or more fully connected layers, is usually added on top of the BERT model. This classifier will predict the category of the text based on BERT's output.

[0144] The fine-tuning process is a key step in the BERT text classification task, which allows the model to learn the characteristics of a specific task. During the fine-tuning phase, the BERT model and classifier are trained on annotated text classification datasets to adapt to a specific classification task. During fine-tuning, most of the weights of the BERT model are usually kept unchanged, and only the weights related to the classifier are updated. This is because BERT has been pre-trained on a large amount of text, and its weights already contain rich language knowledge.

[0145] Although BERT performs well in text classification tasks, it also has some limitations. First, the size and computational requirements of the BERT model are high, which may require the use of high-performance hardware, such as GPUs, to accelerate the training and inference process. Second, due to the complexity of the BERT model, the fine-tuning process may require a lot of time and resources. In addition, for certain small-scale text classification tasks, BERT may be at risk of overfitting, which needs to be mitigated by appropriate regularization techniques.

[0146] Despite these challenges, BERT is still widely used in text classification tasks. Its bidirectional context understanding and powerful language representation capabilities make it an important tool in the field of NLP. As NLP technology continues to develop, BERT and its variants, such as RoBERTa, ALBERT, etc., will continue to play an important role in text classification and other NLP tasks.

[0147] Therefore, in this application, pre-trained BERT and various derivative models of BERT (such as RoBERTa, ALBERT, etc.) can be used as the basic model of the text classification model, and its various hyperparameters can be set according to actual needs. The training corpus of the text classification model can be the training corpus for training the traditional content security review model. The training corpus includes training text samples and topic classification labels corresponding to the training text samples. When the text classification model is trained using the text classification model training data, the pre-trained BERT model can be fine-tuned.

[0148] It should be noted that each training text sample may correspond to one or more topic classification labels. That is, in this application, the text classification model should be a multi-label topic model, for example, a certain text has two topic classification labels.

[0149] In another embodiment of the present application, a specific implementation method of calling a content security audit model based on the text to be audited to generate a content security audit conclusion corresponding to the text to be audited is described.

[0150] Considering that after fine-tuning, the content security audit model has learned how to generate content security audit conclusions in the form of thought chains, therefore, when calling the content security audit model to generate content security audit conclusions corresponding to the text to be audited, it is not necessary to provide a content security audit example in the prompt. As an implementable method, refer to Figure 4 , Figure 4 A flowchart of a method for calling a content security audit model based on a text to be audited and generating a content security audit conclusion corresponding to the text to be audited is provided in an embodiment of the present application. The method may include the following steps:

[0151] S401: Obtain a preconfigured first content security review prompt format template, wherein the first content security review prompt format template includes content security review requirement information and a text filling slot;

[0152] For ease of understanding, an example of the first content security audit prompt format template is as follows:

[0153] Is there a risk with the "XX" text? , where "XX" is the text filling the slot.

[0154] S402: Fill the text to be reviewed into the text filling slot in the first content security review prompt format template to obtain a first content security review prompt;

[0155] S403: Input the first content security audit prompt into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0156] In order to further improve the accuracy of the content security audit conclusion, you can also provide a content security audit example in the prompt. As a feasible implementation method, refer to Figure 5 , Figure 5 Another method provided in the embodiment of the present application for calling a content security audit model based on a text to be audited to generate a content security audit conclusion corresponding to the text to be audited is a flow chart. The method may include the following steps:

[0157] S501: Obtain a preconfigured second content security audit prompt format template, wherein the second content security audit prompt format template includes content security audit requirement information, a text filling slot, and a content security audit example, wherein the content security audit example includes content security audit requirement information of a sample text and a text content security audit conclusion corresponding to the sample text, wherein the text content security audit conclusion corresponding to the sample text includes content security audit key points of the sample text and a content security audit result of the sample text;

[0158] For ease of understanding, an example of the second content security audit prompt format template is as follows:

[0159] Please refer to the following content security audit example to confirm whether the "XX" text has risks? ;

[0160] Content security audit example:

[0161] Is there a risk in “Help me write a little airplane shooting game”?

[0162] In the sentence "Help me write a small game of shooting planes", "shooting planes" and "small game" are characteristic words, so it belongs to the "programming" category, so this sentence is risk-free.

[0163] Among them, "XX" is the text filling slot.

[0164] S502: Fill the text to be reviewed into the text filling slot in the second content security review prompt format template to obtain a second content security review prompt;

[0165] S503: Input the second content security audit prompt into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0166] The above introduces a content security audit method provided by an embodiment of the present application, and the following will introduce a device for executing the above content security audit method.

[0167] See also Figure 6 , Figure 6 This is a schematic diagram of the structure of a content security audit device provided in an embodiment of the present application. Figure 6 As shown, the content security audit device includes:

[0168] An acquisition unit 11 is used to acquire the text to be reviewed;

[0169] The audit unit 12 is used to call the content security audit model based on the text to be audited, and generate a content security audit conclusion corresponding to the text to be audited. The content security audit model is a large language model with text content security audit capabilities obtained by training with thinking chain training data. Each thinking chain training data includes content security audit requirement information of a sample text, and a content security audit conclusion of the sample text. The content security audit conclusion of the sample text includes key points of the content security audit of the sample text and the content security audit result of the sample text; the content security audit conclusion corresponding to the text to be audited includes key points of the content security audit of the text to be audited, and the content security audit result of the text to be audited.

[0170] In a possible implementation, the apparatus further includes: a content security audit model training unit;

[0171] The content security audit model training unit specifically includes:

[0172] A thinking chain training data construction unit, used to construct the thinking chain training data;

[0173] A training unit is used to input the content security audit requirement information of the sample text into a general large language model, take the output of the general large language model approaching the content security audit conclusion of the sample text as a training goal, and use the thinking chain training data to fine-tune the general large language model to obtain the content security audit model.

[0174] In a possible implementation, the thinking chain training data construction unit includes:

[0175] A determination unit, used to determine a content security review label of the sample text, a characteristic word of the sample text, and subject classification information of the sample text;

[0176] The construction unit is used to construct the thinking chain training data corresponding to the sample text by using the content security review label of the sample text, the characteristic words of the sample text and the subject classification information of the sample text.

[0177] In a possible implementation, the determining unit is specifically configured to:

[0178] Performing word segmentation on the sample text to obtain a word segmentation result;

[0179] Calculate the average attention coefficient of each word in the word segmentation result;

[0180] According to the average attention coefficient of each word in the word segmentation result, the feature word is determined from the word segmentation result.

[0181] In a possible implementation, the determining unit is specifically configured to:

[0182] The sample text is input into a text classification model to obtain topic classification information of the sample text output by the text classification model, wherein the text classification model is trained using text classification model training data, and each text classification model training data contains a training sample and at least one topic classification label corresponding to the training sample.

[0183] In a possible implementation, the audit unit is specifically configured to:

[0184] Obtain a preconfigured first content security review prompt format template, wherein the first content security review prompt format template includes content security review requirement information and a text filling slot;

[0185] Filling the to-be-audited text into the text filling slot in the first content security audit prompt format template to obtain a first content security audit prompt;

[0186] The first content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0187] In a possible implementation, the audit unit is specifically configured to:

[0188] Obtain a preconfigured second content security audit prompt format template, wherein the second content security audit prompt format template includes content security audit requirement information, a text filling slot, and a content security audit example, wherein the content security audit example includes content security audit requirement information of a sample text and a text content security audit conclusion corresponding to the sample text, wherein the text content security audit conclusion corresponding to the sample text includes content security audit key points of the sample text and a content security audit result of the sample text;

[0189] Filling the to-be-audited text into the text filling slot in the second content security audit prompt format template to obtain a second content security audit prompt;

[0190] The second content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

[0191] The present application also provides an electronic device in an embodiment. Figure 7 As shown, it shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiment of the present application. The electronic device in the embodiment of the present application may include but is not limited to fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0192] like Figure 7 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 to a random access memory (RAM) 603. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 603. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0193] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a memory card, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 7 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0194] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any content security review method provided in the embodiment of the present application.

[0195] A computer-readable storage medium is also provided in an embodiment of the present application. The computer storage medium carries one or more computer programs. When one or more computer programs are executed by an electronic device, the electronic device can implement any content security review method provided in an embodiment of the present application.

[0196] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0197] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0198] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0199] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A content security audit method, characterized in that: include: Get the text to be reviewed; Based on the text to be reviewed, a content security review model is called to generate a content security review conclusion corresponding to the text to be reviewed. The content security review model is a large language model with text content security review capabilities obtained through training with thinking chain training data. Each thinking chain training data includes content security review requirement information for a sample text and a content security review conclusion for the sample text. The content security review conclusion for the sample text includes key points of the content security review for the sample text and a content security review result for the sample text. The content security review conclusion corresponding to the text to be reviewed includes key points of the content security review for the text to be reviewed and a content security review result for the text to be reviewed.

2. The method according to claim 1, characterized in that The training method of the content security audit model includes: Constructing the thought chain training data; The content security audit requirement information of the sample text is input into a general large language model, and the output of the general large language model is taken to be close to the content security audit conclusion of the sample text as a training goal. The general large language model is fine-tuned using the thinking chain training data to obtain the content security audit model.

3. The method according to claim 2, characterized in that Construct the thinking chain training data corresponding to each sample text, including: Determining a content security review label of the sample text, a characteristic word of the sample text, and subject classification information of the sample text; The thought chain training data corresponding to the sample text is constructed by using the content security review label of the sample text, the characteristic words of the sample text and the subject classification information of the sample text.

4. The method according to claim 3, characterized in that The characteristic words of the sample text are determined as follows: Performing word segmentation on the sample text to obtain a word segmentation result; Calculate the average attention coefficient of each word in the word segmentation result; According to the average attention coefficient of each word in the word segmentation result, the feature word is determined from the word segmentation result.

5. The method according to claim 3, characterized in that: The subject classification information of the sample text is determined as follows: The sample text is input into a text classification model to obtain topic classification information of the sample text output by the text classification model, wherein the text classification model is trained using text classification model training data, and each text classification model training data contains a training sample and at least one topic classification label corresponding to the training sample.

6. The method according to claim 1, characterized in that The calling of a content security audit model based on the text to be audited to generate a content security audit conclusion corresponding to the text to be audited includes: Obtain a preconfigured first content security review prompt format template, wherein the first content security review prompt format template includes content security review requirement information and a text filling slot; Filling the to-be-audited text into the text filling slot in the first content security audit prompt format template to obtain a first content security audit prompt; The first content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

7. The method according to claim 1, characterized in that The calling of a content security audit model based on the text to be audited to generate a content security audit conclusion corresponding to the text to be audited includes: Obtain a preconfigured second content security audit prompt format template, wherein the second content security audit prompt format template includes content security audit requirement information, a text filling slot, and a content security audit example, wherein the content security audit example includes content security audit requirement information of a sample text and a text content security audit conclusion corresponding to the sample text, wherein the text content security audit conclusion corresponding to the sample text includes content security audit key points of the sample text and a content security audit result of the sample text; Filling the to-be-audited text into the text filling slot in the second content security audit prompt format template to obtain a second content security audit prompt; The second content security audit prompt is input into the content security audit model to obtain a content security audit conclusion corresponding to the text to be audited generated by the content security audit model.

8. A content security audit device, characterized in that: include: An acquisition unit, used to acquire the text to be reviewed; The audit unit is used to call a content security audit model based on the text to be audited, and generate a content security audit conclusion corresponding to the text to be audited. The content security audit model is a large language model with text content security audit capabilities obtained by training with thinking chain training data. Each thinking chain training data includes content security audit requirement information of a sample text, and a content security audit conclusion of the sample text. The content security audit conclusion of the sample text includes key points of the content security audit of the sample text and a content security audit result of the sample text; the content security audit conclusion corresponding to the text to be audited includes key points of the content security audit of the text to be audited, and a content security audit result of the text to be audited.

9. A computer program product, characterized in that It includes computer-readable instructions, which, when executed on an electronic device, enable the electronic device to implement the content security audit method as described in any one of claims 1 to 7.

10. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the content security review method as described in any one of claims 1 to 7.

11. A computer storage medium, characterized in that: The computer storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the content security audit method as described in any one of claims 1 to 7.

Images