Large model sensitive content review method and device based on thinking chain

Abstract

The invention relates to a large model sensitive content review method and device based on a thinking chain, and the method comprises the steps: constructing a thinking chain template used for indicating target model review logic; the thinking chain template comprises role setting, behavior description and output specifications for the target model; based on the thinking chain template and a preset content label system, generating a corresponding cue word; and based on the cue word, reviewing the to-be-detected text through the target model to obtain a first output result corresponding to the to-be-detected text. Through the method and the device, the problem of poor generalization caused by the fact that an existing detection method based on training cannot cope with constantly changing sensitive content risks in time is solved, flexible detection of the sensitive content is realized, and the generalization is effectively improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence security technology, and in particular to a method and device for reviewing sensitive content using a large model based on a thought chain. Background Art

[0002] With the continuous development of the Internet and social media, the proliferation of sensitive content has seriously hindered the healthy development of the Internet ecosystem. In order to deal with the threat of sensitive content, a variety of sensitive content detection technologies have emerged, such as keyword detection and semantic similarity detection. However, new sensitive content such as jailbreak prompts used to attack large language models can successfully bypass traditional sensitive content detection tools.

[0003] In order to solve the above problems, existing technologies usually collect relevant sensitive content data sets and use the data sets to train models to detect specific sensitive content. However, the detection capabilities of the above methods are heavily dependent on data sets, and the trained models have poor generalization and cannot respond to the changing risks of sensitive content in a timely manner.

[0004] Currently, no effective solution has been proposed to address the problem that related technologies are unable to respond to the ever-changing risks of sensitive content in a timely manner, resulting in poor generalization. Summary of the invention

[0005] In this embodiment, a large-model sensitive content review method and device based on thought chain are provided to solve the problem that the related technology cannot respond to the changing sensitive content risks in a timely manner, resulting in poor generalization.

[0006] In the first aspect, a large model sensitive content review method based on thought chain is provided in this embodiment, and the method includes:

[0007] Constructing a thought chain template for indicating the review logic of the target model; the thought chain template includes role settings, behavior descriptions and output specifications for the target model;

[0008] Generate corresponding prompt words based on the thought chain template and the preset content label system;

[0009] Based on the prompt word, the text to be detected is examined through the target model to obtain a first output result corresponding to the text to be detected.

[0010] In some of the embodiments, the step of constructing a thought chain template for indicating the target model review logic includes:

[0011] Based on preset role characteristics, reasoning principles and output requirements, construct the thought chain template for indicating the review logic of the target model;

[0012] Among them, the role characteristics are used to guide the role setting for the target model; the reasoning principles are used to guide the behavior description for the target model; and the output requirements are used to guide the output specifications for the target model.

[0013] In some of the embodiments, the preset content label system includes multiple label categories; the output specification indicates that the first output result of the target model includes the label category and the classification reason corresponding to the text to be detected.

[0014] In some embodiments, the generating corresponding prompt words based on the thought chain template and the preset content tag system includes:

[0015] Constructing multiple content detection examples; the content detection examples include sensitive content examples and / or normal content examples of different categories;

[0016] Based on each of the content detection examples, generate a corresponding example set;

[0017] Based on the thought chain template, the preset content tag system and the example collection, the corresponding prompt words are generated.

[0018] In some of the embodiments, after the text to be detected is examined by the target model based on the prompt word to obtain a first output result corresponding to the text to be detected, the method further includes:

[0019] When it is detected that the parsing of the first output result fails, obtaining the vector distribution when the target model generates tokens;

[0020] A vector distribution analysis program is called to analyze and process the vector distribution to obtain a second output result of the text to be detected; the second output result is an output category corresponding to the maximum predicted vector value in the target model.

[0021] In some of the embodiments, after the text to be detected is examined by the target model based on the prompt word to obtain a first output result corresponding to the text to be detected, the method further includes:

[0022] The prompt word is dynamically adjusted according to the first output result.

[0023] In a second aspect, in this embodiment, a large model sensitive content review device based on thought chain is provided, including:

[0024] A construction module, used to construct a thought chain template for indicating the review logic of the target model; the thought chain template includes role settings, behavior descriptions and output specifications for the target model;

[0025] A generation module, used to generate corresponding prompt words based on the thought chain template and a preset content tag system;

[0026] The review module is used to review the text to be detected based on the prompt word through the target model to obtain a first output result corresponding to the text to be detected.

[0027] In the third aspect, a computer device is provided in this embodiment, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the large model sensitive content review method based on the thinking chain described in the first aspect is implemented.

[0028] In a fourth aspect, a storage medium is provided in this embodiment, on which a computer program is stored, and when the program is executed by a processor, the large model sensitive content review method based on the thought chain described in the first aspect above is implemented.

[0029] In a fifth aspect, a computer program product is provided in this embodiment, including a computer program, which, when executed by a processor, implements the large model sensitive content review method based on thought chain described in the first aspect above.

[0030] Compared with the related art, the large-model sensitive content review method and device based on thought chain provided in this embodiment constructs a thought chain template for indicating the review logic of the target model; the thought chain template includes role settings, behavior descriptions and output specifications for the target model; based on the thought chain template and the preset content label system, corresponding prompt words are generated; based on the prompt words, the text to be detected is reviewed through the target model to obtain the first output result corresponding to the text to be detected, which solves the problem of being unable to respond to the ever-changing sensitive content risks in a timely manner, resulting in poor generalization, and realizes flexible detection of sensitive content, effectively improving generalization.

[0031] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0033] Figure 1 This is a hardware structure block diagram of a terminal device of a large model sensitive content review method based on a thought chain provided in an embodiment of the present application;

[0034] Figure 2This is a flow chart of a large model sensitive content review method based on thought chain provided by an embodiment of the present application;

[0035] Figure 3 is a flow chart of a method for constructing prompt words provided in an embodiment of the present application;

[0036] Figure 4 It is a flowchart of a sensitive content review method based on a large model of a thought chain provided in an embodiment of the present application;

[0037] Figure 5 This is a flow chart of a method for reviewing sensitive content based on a large model of a thought chain provided in a preferred embodiment of the present application;

[0038] Figure 6 This is a structural block diagram of a large-model sensitive content review device based on thought chain provided in one embodiment of the present application.

[0039] In the figure: 102, processor; 104, memory; 106, transmission device; 108, input and output device; 10, construction module; 20, generation module; 30, review module. DETAILED DESCRIPTION

[0040] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0041] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not represent quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and system, product or device comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0042] The method embodiment provided in this embodiment can be executed in a terminal, a computer or a similar computing device. For example, running on a terminal, Figure 1 1 is a hardware structure block diagram of a terminal of the method for reviewing sensitive content based on a large model of a thought chain in this embodiment. Figure 1 As shown, the terminal may include one or more ( Figure 1 Only one is shown in the figure) processor 102 and memory 104 for storing data, wherein processor 102 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA. The above terminal may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above terminal. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations shown.

[0043] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the sensitive content review method based on the large model of thinking chain in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, that is, to implement the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories can be connected to the terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0044] The transmission device 106 is used to receive or send data via a network. The above network includes a wireless network provided by the communication provider of the terminal. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, referred to as RF) module, which is used to communicate with the Internet wirelessly.

[0045] In this embodiment, a large model sensitive content review method based on thought chain is provided. Figure 2 is a flowchart of the sensitive content review method of the large model based on the thought chain of this embodiment. Figure 2 As shown, the process includes the following steps:

[0046] Step S210, constructing a thought chain template for indicating the review logic of the target model; the thought chain template includes role settings, behavior descriptions and output specifications for the target model;

[0047] Step S220, generating corresponding prompt words based on the thought chain template and the preset content tag system;

[0048] Step S230: Based on the prompt word, the text to be detected is examined through the target model to obtain a first output result corresponding to the text to be detected.

[0049] Specifically, a thought chain template is pre-built to indicate the review logic of the target model, thereby guiding the target model to identify sensitive and harmful information in the input content through step-by-step reasoning to make efficient decisions. The target model can use a large language model deployed locally, or call a business model application programming interface (Application Programming Interface, API) service, which is not limited here. The thought chain template consists of specific prompt words, including role settings, behavior descriptions, and output specifications for the target model, which are specifically expressed as follows:

[0050] P module =P role +P rule +P out (1)

[0051] In formula (1), P module Represents the thinking chain template; P role Indicates the role setting for the target model; P rule Represents the behavior description of the target model; P out Represents the output specification for the target model. Among them, the role setting for the target model is carried out according to the preset role characteristics, so as to set the role played by the target model; the behavior description is used to describe the specific review method of the input content by the target model, such as stipulating that when the input content contains sensitive words or semantic violations, the input content is judged to be harmful content; and the output specification is used to explain the output of the target model in order to parse the output results of the target model.

[0052] Exemplarily, the target model role is set as a judge, and the model is required to simulate the judge's behavior and review the input content based on principles such as the presumption of innocence. When the input content is judged to be harmful content, the output result is the harmful category of the input content. Otherwise, the output result is that the input content is normal.

[0053] It should be noted that jailbreak prompts used to attack large language models often use role-playing methods to complete tasks by requiring large language models to play specific roles. Based on this, this embodiment sets the role of the target model in the thinking chain template, such as setting the target model as a judge, an expert in a specific field, etc., so that the model can accurately review the input content based on specific review principles, and at the same time can well cope with the working mode of jailbreak prompts.

[0054] Furthermore, based on the actual content detection needs, a corresponding content labeling system is constructed. The content labeling system is mainly composed of a variable formatted content label description. The selected label classification system includes multiple different label classifications. Each label classification can be changed in a targeted manner as the review content and content risk situation change, so as to achieve rapid expansion of the identification of new categories of sensitive content.

[0055] Afterwards, based on the thought chain template and the preset content label system, the corresponding prompt words are generated, as shown below:

[0056] P judge =P module +P label (2)

[0057] In formula (2), P judge Indicates prompt word; P module Represents the thinking chain template; P label Represents a content label system. The constructed prompt word and the text to be detected are input into the target model, and based on the induced prompt provided by the prompt word, the text to be detected is examined by the target model to obtain the first output result corresponding to the text to be detected, which is specifically represented as follows:

[0058] R binary =M(T,P judge ) (3)

[0059] In formula (3), R binary represents the standard output of the target model; M represents the target model; T represents the input text to be detected; P judge Indicates prompt words. Based on this, by organically combining the thinking chain template and the preset content label system, an inductive prompt for sensitive content identification is constructed, thereby effectively stimulating the content review ability of the target model, improving the accuracy of content review, and realizing flexible detection of sensitive content, improving generalization and extensibility.

[0060] In the prior art, relevant sensitive content data sets are usually collected and used to train models to detect specific sensitive content. However, the detection capabilities of the above methods are heavily dependent on data sets, and the trained models have poor generalization and cannot respond to the changing risks of sensitive content in a timely manner.

[0061] Compared with the prior art, the present application constructs a thinking chain template for indicating the review logic of the target model; the thinking chain template includes role settings, behavior descriptions and output specifications for the target model; based on the thinking chain template and the preset content label system, the corresponding prompt words are generated; based on the prompt words, the text to be detected is reviewed through the target model to obtain the first output result corresponding to the text to be detected. Based on this, the role played by the model is set by the thinking chain template, which effectively responds to the working mode of the jailbreak prompt. At the same time, by organically combining the thinking chain template and the preset content label system, an induced prompt for sensitive content identification is constructed, so that the target model can accurately review the input content, thereby solving the problem of being unable to respond to the changing risks of sensitive content in a timely manner, resulting in poor generalization, and realizing flexible detection of sensitive content, effectively improving generalization.

[0062] Moreover, this application has flexible requirements on the accuracy of model loading. The target model can use a locally loaded low-precision large language model. Compared with other large language model downstream tasks, such as content continuation, which requires high-precision parameters, the target model only needs to be deployed at a minimum of INT4 accuracy and still has strong detection capabilities. That is, while ensuring strong detection capabilities, it can effectively save computing memory consumption and facilitate low-precision deployment.

[0063] In some embodiments, the step S210 of constructing a thought chain template for indicating the target model review logic includes the following steps:

[0064] Based on the preset role characteristics, reasoning principles and output requirements, a thinking chain template is constructed to indicate the review logic of the target model;

[0065] Among them, role characteristics are used to guide the role setting for the target model; reasoning principles are used to guide the behavior description for the target model; and output requirements are used to guide the output specifications for the target model.

[0066] Specifically, a thought chain template is pre-built, and the thought chain template is used to indicate the review logic of the target model to guide the target model to identify sensitive and harmful information in the input content by step-by-step reasoning. In this embodiment, the thought chain template is composed of specific prompt words, specifically including role settings, behavior descriptions, and output specifications for the target model.

[0067] Among them, the role setting for the target model is carried out according to the preset role characteristics, so as to specify the role characteristics of the target model; the behavior description is used to describe the specific review method of the target model on the input content, such as stipulating that when the input content contains sensitive words or semantic violations, the input content is judged to be harmful content; the output specification is used to describe the output of the target model in order to parse the output results of the target model.

[0068] It should be noted that by setting the role of the target model in the thinking chain template, such as setting the target model as a judge, an expert in a specific field, etc., the model can accurately review the input content based on specific review principles. For example, the role of the target model is set as a judge, requiring the model to simulate the behavior of the judge and review the input content based on principles such as the presumption of innocence; or, the role of the target model is set as an expert in a specific field, taking the field of computer science as an example, the input content is analyzed based on professional computer terminology to detect whether the content is harmful (such as whether it contains malicious code, etc.), so as to be able to deeply analyze the potential risks of the content and provide accurate judgments. This embodiment does not specifically limit the role setting, and can be flexibly set according to actual application needs.

[0069] Through this embodiment, based on preset role characteristics, reasoning principles and output requirements, a thinking chain template for indicating the review logic of the target model is constructed. The role characteristics are used to guide the role setting for the target model, the reasoning principles are used to guide the behavior description for the target model, and the output requirements are used to guide the output specifications for the target model. In this way, the target model is guided to identify sensitive and harmful information in the content by using a step-by-step reasoning method, which helps to achieve efficient decision-making.

[0070] In some of the embodiments, the preset content label system includes multiple label categories; the output specification indicates that the first output result of the target model includes the label category and the reason for the category corresponding to the text to be detected.

[0071] Specifically, since the large language model identifies sensitive content by analyzing the correlation between the semantics of the input content and the content tag, this embodiment pre-builds a content tag system, which is mainly composed of a variable formatted content tag description. The content tag description format is as follows:

[0072] P label ={S1,S2,S3,……,S n} (4)

[0073] In formula (4), P label Indicates the selected label classification system; S n represents the nth label classification. Accordingly, the canonical output R binary It will contain the classification result of the input content and optional classification reason, as shown below:

[0074] R binary =S out +S reason (option) (5)

[0075] In formula (5), R binary represents the canonical output of the target model; S outIndicates the classification result of the input content; S reason (option) indicates an optional classification reason. For example, if the input content is classified as privacy violation, the corresponding classification reason is that words involving personal privacy are detected in the input content. It should be noted that the above-mentioned label classifications can be changed in a targeted manner as the review content and content risk situation change, so as to achieve rapid expansion of the identification of new categories of sensitive content.

[0076] Through this embodiment, a content label system including multiple label classifications is preset, so that through the output specification, it is indicated that the first output result of the target model includes the label classification and classification reason corresponding to the text to be detected, so as to improve the transparency of model reasoning and facilitate targeted adjustment of prompt words and other model-related parameters when the model output results are erroneous or biased.

[0077] In some of the embodiments, Figure 3 As shown, the step S220 generates corresponding prompt words based on the thought chain template and the preset content tag system, including the following steps:

[0078] Step S221, constructing multiple content detection examples; the content detection examples include sensitive content examples and / or normal content examples of different categories;

[0079] Step S222, generating a corresponding example set based on each content detection example;

[0080] Step S223, generating corresponding prompt words based on the thought chain template, the preset content tag system and the example collection.

[0081] Specifically, multiple content detection examples are constructed, including different categories of sensitive content examples and / or normal content examples. For example, normal content examples include "how to achieve data protection" and "how to achieve data visualization", and sensitive content examples include "how to hack into other people's computers", etc., which are not limited here.

[0082] Furthermore, based on each content detection example, a corresponding example set is generated, which is specifically expressed as follows:

[0083] E sum ={E1, E2, E3, ..., E n} (6)

[0084] In formula (6), E sum represents the example set; E n Indicates that it is the nth example. Based on the thought chain template, the preset content tag system and the example collection, the corresponding prompt words are generated, which are specifically expressed as follows:

[0085] P judge =Pmodule +P label +E sum (7)

[0086] In formula (7), P judge Indicates prompt word; P module Represents the thinking chain template; P label Indicates the content label system; E sum Represents a collection of examples.

[0087] It should be noted that the target model has the ability of in-context learning (ICL), which enables the model to significantly improve the generation quality by learning a small number of examples. Based on this model feature, by adding different categories of sensitive content examples and normal content examples to the prompt words, the model's ability to understand semantics is used to improve the model's detection ability in real time, thereby improving the accuracy of model review.

[0088] Through this embodiment, multiple content detection examples are constructed, and the content detection examples include sensitive content examples and / or normal content examples of different categories. An example set is generated based on each content detection example, and corresponding prompt words are generated based on the thinking chain template, the preset content label system and the example collection. In this way, by organically combining the thinking chain template and the preset content label system, an induced prompt for sensitive content identification is constructed, which helps to stimulate the content review capability of the target model and improve the accuracy of content review.

[0089] In some of the embodiments, after the text to be detected is examined by the target model based on the prompt word to obtain the first output result corresponding to the text to be detected, the following steps are also included:

[0090] When it is detected that the parsing of the first output result fails, obtaining the vector distribution when the target model generates tokens;

[0091] The vector distribution analysis program is called to analyze and process the vector distribution to obtain a second output result of the text to be detected; the second output result is the output category corresponding to the maximum predicted vector value in the target model.

[0092] Specifically, the number of tokens output by the preset model is used to control the length of the model output result, so as to speed up the model review process, and after obtaining the first output result of the target model, it is detected whether the first output result is parseable.

[0093] Among them, if there is an abnormality in the first output result, that is, the target model does not output according to the pre-built prompt word requirements, and the parsing output program cannot parse the first output result, then obtain the vector distribution when the target model generates tokens, call the vector distribution analysis program to analyze and process the vector distribution, and obtain the second output result corresponding to the text to be detected. The second output result is the output category corresponding to the maximum predicted vector value in the target model. In this way, the category with the largest predicted vector value in the output category is assigned as the new result, so as to obtain a format-compliant review result from the model output vector, and avoid the output deviating from the format requirements in the prompt word or outputting meaningless content. The specific representation of the above analysis process is as follows:

[0094] R binary , Vector out =M(T,P judge ) (8)

[0095] if Proc(R binary )==False:R binary =Ex(Vector out ) (9)

[0096] In formula (8) and formula (9), R binary represents the standard output of the target model; M represents the target model; T represents the input text to be detected; P judge Indicates prompt word; Vector out is a vector set of output restriction tokens; Proc is the parsing output program; Ex is the vector distribution analysis program.

[0097] Through this embodiment, when it is detected that the parsing failure of the first output result is detected, the vector distribution when the target model generates tokens is obtained, and the vector distribution analysis program is called to analyze and process the vector distribution to obtain the second output result of the text to be detected. The second output result is the output category corresponding to the maximum predicted vector value in the target model. Therefore, through prompt word induction and output vector analysis, it is ensured that for any sensitive content input, a valid and parsable result can be output, thereby improving the output stability of the target model and facilitating the realization of automated operation.

[0098] In some of the embodiments, after the text to be detected is examined by the target model based on the prompt word to obtain the first output result corresponding to the text to be detected, the following steps are also included:

[0099] According to the first output result, the prompt word is dynamically adjusted.

[0100] Specifically, after obtaining the first output result of the model, if it is detected that the output result is not output according to the pre-built prompt word requirements, or the classification result of the text to be detected is incorrect, etc., the reason for the abnormal situation is analyzed and fed back, and the prompt word is dynamically adjusted according to the feedback reason.

[0101] For example, if the reason for the abnormal situation in the analysis feedback is that there are loopholes in the reasoning logic of the target model, the prompt word content in the thinking chain template is adjusted in a targeted manner, such as adding conditional judgments in each reasoning step, introducing multiple reasoning paths, etc., to refine the reasoning steps in the thinking chain template, and add new content detection examples for the error information in the output results, including sensitive content examples and normal content examples, to provide richer reasoning basis.

[0102] Through this embodiment, after obtaining the first output result corresponding to the text to be detected, the prompt word is dynamically adjusted according to the first output result to achieve optimization of the prompt word, which helps to improve the review accuracy of the target model.

[0103] The present embodiment is described and illustrated by means of specific examples below.

[0104] Figure 4 : is a flowchart of the sensitive content review method of the large model based on the thought chain of this embodiment. Figure 4 As shown, the sensitive content review method based on the big model of thought chain specifically includes the following steps:

[0105] The text to be detected is input into the sensitive content review system, which includes a prompt word construction module, an inference decision module, and an output vector correction module. In the prompt word construction module, a thought chain template for indicating the target model review logic is constructed, and the thought chain template includes role settings, behavior descriptions, and output specifications for the target model, and a content label system is selected. The content label system consists of a variable formatted content label description, and multiple content detection examples are constructed. The content detection examples include sensitive content examples and / or normal content examples of different categories, and a corresponding example set is generated based on each content detection example.

[0106] Furthermore, corresponding prompt words are generated based on the thought chain template, content label system and example collection. The reasoning decision module is mainly composed of a variable large language model. The text to be detected and the prompt words are input into the reasoning decision module, and the text to be detected is subject to content review to determine whether the text to be detected is harmful content or normal content, and the first output result of the model is obtained. Afterwards, when it is detected that the first output result fails to be parsed, the output vector correction module obtains the formatted review result from the model output vector. Specifically, the vector distribution when the target model generates tokens is obtained, and the vector distribution analysis program is called to analyze and process the vector distribution, and the category with the largest predicted vector value in the output category is assigned as the new result.

[0107] The present embodiment is described and illustrated below through preferred embodiments.

[0108] Figure 5 Flowchart of the sensitive content review method based on the big model of thought chain in this preferred embodiment. Figure 5 As shown, the sensitive content review method based on the big model of thought chain includes the following steps:

[0109] Step S510, based on preset role characteristics, reasoning principles and output requirements, construct a thinking chain template for indicating the review logic of the target model; the role characteristics are used to guide the role setting for the target model; the reasoning principles are used to guide the behavior description for the target model; the output requirements are used to guide the output specifications for the target model;

[0110] Step S520, constructing multiple content detection examples, the content detection examples include different categories of sensitive content examples and / or normal content examples, generating corresponding example sets based on each content detection example, and generating corresponding prompt words based on the thought chain template, the preset content label system and the example collection;

[0111] Step S530, based on the prompt word, the text to be detected is examined by the target model to obtain a first output result corresponding to the text to be detected, and whether the first output result is parseable;

[0112] Step S540, when it is detected that the parsing failure of the first output result is detected, the vector distribution when the target model generates tokens is obtained, and the vector distribution analysis program is called to analyze and process the vector distribution to obtain the second output result of the text to be detected; the second output result is the output category corresponding to the maximum predicted vector value in the target model.

[0113] Through this embodiment, a thinking chain template for indicating the review logic of the target model is constructed based on preset role characteristics, reasoning principles and output requirements. Among them, role characteristics are used to guide the role setting for the target model; reasoning principles are used to guide the behavior description for the target model; output requirements are used to guide the output specifications for the target model, so that the model uses a step-by-step reasoning method to identify the input content. Construct multiple content detection examples, the content detection examples include different categories of sensitive content examples and / or normal content examples, and generate corresponding example sets based on each content detection example, and generate corresponding prompt words based on the thinking chain template, the preset content label system and the example collection, so as to organically combine the thinking chain template, the content label system and the example collection to construct an inductive prompt for sensitive content identification, thereby effectively stimulating the content review capability of the target model.

[0114] Furthermore, based on the prompt words, the text to be detected is reviewed through the target model to obtain the first output result corresponding to the text to be detected, which solves the problem of being unable to respond to the ever-changing risks of sensitive content in a timely manner, resulting in poor generalization, and realizes flexible detection of sensitive content, effectively improving generalization.

[0115] Afterwards, check whether the first output result is parseable. When it is detected that the parsing of the first output result fails, obtain the vector distribution when the target model generates tokens, and call the vector distribution analysis program to analyze and process the vector distribution to obtain the second output result of the text to be detected. The second output result is the output category corresponding to the maximum predicted vector value in the target model, so as to obtain a format-compliant review result from the model output vector, avoid the output deviating from the format requirements in the induced prompt or outputting meaningless content, and effectively improve the output stability of the target model.

[0116] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0117] In this embodiment, a large model sensitive content review device based on a thought chain is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and will not be repeated hereafter. The terms "module", "unit", "sub-unit", etc. used below can implement a combination of software and / or hardware for predetermined functions. Although the devices described in the following embodiments are preferably implemented in software, the implementation in hardware, or a combination of software and hardware, is also possible and conceivable.

[0118] Figure 6 is a structural block diagram of the large model sensitive content review device based on the thought chain of this embodiment, such as Figure 6 As shown, the device comprises:

[0119] A construction module 10 is used to construct a thought chain template for indicating the review logic of the target model; the thought chain template includes role settings, behavior descriptions and output specifications for the target model;

[0120] A generating module 20, for generating corresponding prompt words based on the thought chain template and the preset content tag system;

[0121] The review module 30 is used to review the text to be detected through the target model based on the prompt word to obtain a first output result corresponding to the text to be detected.

[0122] Through the device provided in this embodiment, a thinking chain template for indicating the review logic of the target model is constructed; the thinking chain template includes role settings, behavior descriptions and output specifications for the target model; based on the thinking chain template and the preset content label system, corresponding prompt words are generated; based on the prompt words, the text to be detected is reviewed through the target model to obtain the first output result corresponding to the text to be detected, which solves the problem of being unable to respond to the constantly changing risks of sensitive content in a timely manner, resulting in poor generalization, realizes flexible detection of sensitive content, and effectively improves generalization.

[0123] In some of the embodiments, the construction module 10 is also used to construct a thinking chain template for indicating the review logic of the target model based on preset role characteristics, reasoning principles and output requirements; wherein the role characteristics are used to guide the role setting for the target model; the reasoning principles are used to guide the behavior description for the target model; and the output requirements are used to guide the output specifications for the target model.

[0124] In some of the embodiments, the generation module 20 is also used to construct multiple content detection examples; the content detection examples include sensitive content examples and / or normal content examples of different categories; based on each content detection example, a corresponding example set is generated; based on the thought chain template, the preset content label system and the example collection, a corresponding prompt word is generated.

[0125] In some of these embodiments, Figure 6 On the basis of, the device also includes a correction module, which is used to obtain the vector distribution when the target model generates tokens when it is detected that the parsing of the first output result fails; call the vector distribution analysis program to analyze and process the vector distribution to obtain the second output result of the text to be detected; the second output result is the output category corresponding to the maximum predicted vector value in the target model.

[0126] In some of these embodiments, Figure 6 On the basis of, the device also includes an adjustment module, which is used to dynamically adjust the prompt word according to the first output result.

[0127] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0128] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0129] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0130] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0131] S1, construct a thinking chain template for indicating the review logic of the target model; the thinking chain template includes role setting, behavior description and output specification for the target model;

[0132] S2, based on the thought chain template and the preset content label system, generates corresponding prompt words;

[0133] S3, based on the prompt word, the text to be detected is examined through the target model to obtain a first output result corresponding to the text to be detected.

[0134] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0135] In addition, in combination with the large model sensitive content review method based on thought chain provided in the above embodiment, a storage medium can also be provided in this embodiment to implement. The storage medium stores a computer program; when the computer program is executed by the processor, any of the large model sensitive content review methods based on thought chain in the above embodiment is implemented.

[0136] In this embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements any one of the large model sensitive content review methods based on thought chain in the above embodiments.

[0137] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0138] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0139] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0140] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A large model sensitive content review method based on thought chain, characterized in that: The method comprises: Constructing a thought chain template for indicating the review logic of the target model; the thought chain template includes role settings, behavior descriptions and output specifications for the target model; Generate corresponding prompt words based on the thought chain template and the preset content label system; Based on the prompt word, the text to be detected is examined through the target model to obtain a first output result corresponding to the text to be detected.

2. The method for reviewing sensitive content based on a large model of thought chain according to claim 1 is characterized in that: The thought chain template for constructing the target model review logic includes: Based on preset role characteristics, reasoning principles and output requirements, construct the thought chain template for indicating the review logic of the target model; Among them, the role characteristics are used to guide the role setting for the target model; the reasoning principles are used to guide the behavior description for the target model; and the output requirements are used to guide the output specifications for the target model.

3. The sensitive content review method based on the big model of thought chain according to claim 1 is characterized in that: The preset content label system includes multiple label categories; the output specification indicates that the first output result of the target model includes the label category and the classification reason corresponding to the text to be detected.

4. The method for reviewing sensitive content based on a large model of thought chain according to claim 1 is characterized in that: The generating corresponding prompt words based on the thought chain template and the preset content tag system includes: Constructing multiple content detection examples; the content detection examples include sensitive content examples and / or normal content examples of different categories; Based on each of the content detection examples, generate a corresponding example set; Based on the thought chain template, the preset content tag system and the example collection, the corresponding prompt words are generated.

5. The method for reviewing sensitive content based on a large model of thought chain according to claim 1 is characterized in that: After the text to be detected is examined based on the prompt word by the target model to obtain a first output result corresponding to the text to be detected, the method further includes: When it is detected that the parsing of the first output result fails, obtaining the vector distribution when the target model generates tokens; A vector distribution analysis program is called to analyze and process the vector distribution to obtain a second output result of the text to be detected; the second output result is an output category corresponding to the maximum predicted vector value in the target model.

6. The method for reviewing sensitive content based on a large model of thought chain according to claim 1 is characterized in that: After the text to be detected is examined based on the prompt word by the target model to obtain a first output result corresponding to the text to be detected, the method further includes: The prompt word is dynamically adjusted according to the first output result.

7. A large model sensitive content review device based on thought chain, characterized in that: include: A building module for building a thought chain template for indicating the review logic of the target model; The thought chain template includes role settings, behavior descriptions and output specifications for the target model; A generation module, used to generate corresponding prompt words based on the thought chain template and a preset content tag system; The review module is used to review the text to be detected based on the prompt word through the target model to obtain a first output result corresponding to the text to be detected.

8. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps of the large model sensitive content review method based on thought chain as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the large model sensitive content review method based on thought chain described in any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the large model sensitive content review method based on thought chain described in any one of claims 1 to 6 are implemented.

Images