Text auditing method and related device

Abstract

The invention discloses a text auditing method and a related device, and relates to the technical field of natural language processing, the text auditing method comprises the following steps: obtaining a first target text, the first target text being an input text of a large model; determining a risk category of the first target text; if the risk category of the first target text is harmful, outputting a preset bottom reply; if the risk category of the first target text is harmless, inputting the first target text into a large model; under the condition that the risk category of the first target text is harmless, a text replied by the large model for the first target text can be obtained as a second target text; determining a risk category of the second target text; if the risk category of the second target text is harmful, outputting a preset bottom reply; and if the risk category of the second target text is harmless, outputting the second target text. The text auditing method disclosed by the invention can ensure that the large model outputs harmless texts.

Description

Technical Field

[0001] The present application relates to the field of natural language processing technology, and in particular to a text review method and related devices. Background Art

[0002] In recent years, breakthroughs in the field of deep learning have driven the rapid development of pre-trained large models, which have mastered rich language structure and semantic knowledge through unsupervised learning on massive data. Taking natural language processing (NLP) as an example, models such as GPT, Spark Large Model, Wenxin Yiyan, etc. can not only generate coherent and highly information-dense text, but also perform a variety of complex tasks from question-answering to translation, from summarization to reasoning. With the surge in the number of model parameters, its ability to understand and generate natural language has been significantly improved, bringing revolutionary changes to many industries, including but not limited to education, entertainment, finance and healthcare.

[0003] However, the powerful capabilities of large models are also accompanied by a series of challenges, among which the most critical is the issue of content security. Since the sources of training data are extensive and complex, large models may inadvertently absorb and amplify bias, discrimination, false information, and other information in the data. As a result, large models may output harmful information (such as offensive speech, false information, inappropriate suggestions, etc.). In order to make large models output harmless information, there is an urgent need for a text review solution for large model scenarios. Summary of the invention

[0004] In view of this, the present application provides a text review method and related devices to enable the large model to output harmless information. The technical solution is as follows:

[0005] The first aspect of the present application provides a text review method, comprising:

[0006] Acquire a first target text, where the first target text is a text to be input into a target macro model;

[0007] Based on the first risk category prediction model obtained by pre-training, predict the risk category of the first target text to obtain the first risk category of the first target text, wherein the first risk category prediction model is trained using a first training text marked with the first risk category, a first training text marked with the second risk category, and a first training text marked with the third risk category, and the first risk category, the second risk category, and the third risk category are harmful, suspected harmful, and harmless, respectively;

[0008] Determining a final risk category of the first target text according to the first risk category of the first target text, wherein the final risk category of the first target text is one of harmful and harmless;

[0009] If the final risk category of the first target text is harmful, a preset catch-all response is output; if the final risk category of the first target text is harmless, the first target text is input into the target macro model.

[0010] In a possible implementation, the text review method further includes:

[0011] When the final risk category of the first target text is harmless, obtaining the text in which the target large model responds to the first target text as the second target text;

[0012] Based on the pre-trained second risk category prediction model, predict the risk category of the second target text to obtain the first risk category of the second target text, wherein the second risk category prediction model is trained using the second training text annotated with the first risk category and the second training text annotated with the third risk category;

[0013] Determining a final risk category of the second target text according to the first risk category of the second target text, wherein the final risk category of the second target text is one of harmful and harmless;

[0014] If the final risk category of the second target text is harmful, a preset fallback response is output; if the final risk category of the second target text is harmless, the second target text is output.

[0015] In a possible implementation, the text review method further includes:

[0016] Determining the risk category of the first target text based on whether the first target text contains keywords in the first keyword set, and obtaining a second risk category of the first target text, wherein the first keyword set includes must-kill words and sensitive words, the must-kill words are harmful, independent words that are not allowed to appear, and the sensitive words are words composed of sensitive points and negative words;

[0017] The determining, according to the first risk category of the first target text, a final risk category of the first target text includes:

[0018] A final risk category of the first target text is determined according to the first risk category of the first target text in combination with the second risk category of the first target text.

[0019] In a possible implementation, determining the final risk category of the first target text according to the first risk category of the first target text and in combination with the second risk category of the first target text includes:

[0020] If the first risk category of the first target text is harmful or harmless, the first risk category of the first target text is determined as the first target risk category of the first target text; if the first risk category of the first target text is suspected harmful, the first risk category of the first target text is adjusted to one of harmful and harmless according to a preset risk category adjustment strategy, and the adjusted risk category is used as the first target risk category of the first target text;

[0021] If at least one of the first target risk category of the first target text and the second risk category of the first target text is harmful, determining that the final risk category of the first target text is harmful, and the second risk category of the first target text is one of harmful and harmless;

[0022] If the first target risk category of the first target text and the second risk category of the first target text are both harmless, then the final risk category of the first target text is determined to be harmless.

[0023] In a possible implementation, the process of acquiring the first training text annotated with the first risk category and the first training text annotated with the second risk category includes:

[0024] Acquire multiple text samples containing keywords in a first keyword set, wherein the first keyword set includes must-kill words and sensitive words, the must-kill words are harmful, independent words that are not allowed to appear, and the sensitive words are words composed of sensitive points and negative words;

[0025] For each text sample obtained:

[0026] Input the text sample into the big model to obtain the reply text of the big model for the text sample;

[0027] If the reply text of the text sample is harmful text, then determining the risk category of the text sample is harmful, so as to obtain a first training text marked with a first risk category;

[0028] If the reply text of the text sample is harmless text, the risk category of the text sample is determined to be suspected harmful, so as to obtain a first training text marked with a second risk category.

[0029] In a possible implementation, the text review method further includes:

[0030] After the target large model is upgraded, the first keyword set is updated according to the recovery capability of the upgraded target large model;

[0031] And / or, according to the recovery capability of the upgraded target large model, the risk category annotated by the first training text of the first risk category prediction model is adjusted, and the first risk category prediction model is adjusted using the first training text after the risk category is adjusted.

[0032] In a possible implementation, the predicting the risk category of the second target text based on the pre-trained second risk category prediction model to obtain the first risk category of the second target text includes:

[0033] Predicting the probability that the risk category of the second target text is harmful based on the pre-trained second risk category prediction model;

[0034] If the probability that the risk category of the second target text is harmful is greater than or equal to a preset first probability threshold, determining that the first risk category of the second target text is harmful;

[0035] If the probability that the risk category of the second target text is harmful is greater than a preset second probability threshold and less than a preset first probability threshold, determining that the first risk category of the second target text is suspected harmful, wherein the second probability threshold is less than the first probability threshold;

[0036] If the probability that the risk category of the second target text is harmful is less than or equal to a preset second probability threshold, it is determined that the first risk category of the second target text is harmless.

[0037] In a possible implementation, the text review method further includes:

[0038] Determining the risk category of the second target text based on whether the second target text contains keywords in the second keyword set, thereby obtaining a second risk category of the second target text, wherein the second keyword set includes sensitive words and negative words, and the sensitive words are words composed of sensitive points and negative words;

[0039] The determining, according to the first risk category of the second target text, a final risk category of the second target text comprises:

[0040] A final risk category of the second target text is determined according to the first risk category of the second target text and in combination with the second risk category of the second target text.

[0041] In a possible implementation, the training process of the second risk category prediction model includes:

[0042] Obtaining a training triplet, wherein the training triplet includes an anchor sample, a positive sample corresponding to the anchor sample, and a negative sample corresponding to the anchor sample, wherein the anchor sample and the positive sample corresponding to the anchor sample are both the second training text annotated with the third risk category, and the negative sample corresponding to the anchor sample is the second training text annotated with the first risk category;

[0043] Based on the second risk category prediction model, respectively, the anchor sample, the positive sample corresponding to the anchor sample, and the negative sample corresponding to the anchor sample are text-encoded to obtain the text features of the anchor sample, the text features of the positive sample corresponding to the anchor sample, and the text features of the negative sample corresponding to the anchor sample;

[0044] Based on the second risk category prediction model, and taking the text features of the anchor sample as a basis, predicting the risk category of the anchor sample;

[0045] Calculating the distance between the anchor sample and the positive sample corresponding to the anchor sample according to the text features of the anchor sample and the text features of the positive sample corresponding to the anchor sample, and calculating the distance between the anchor sample and the negative sample corresponding to the anchor sample according to the text features of the anchor sample and the text features of the negative sample corresponding to the anchor sample;

[0046] The parameters of the second risk category prediction model are updated with the goal of making the predicted risk category consistent with the risk category marked by the anchor sample and making the distance between the anchor sample and the positive sample corresponding to the anchor sample smaller than the distance between the anchor sample and the negative sample corresponding to the anchor sample.

[0047] A second aspect of the present application provides a text review device, comprising: a large model input text acquisition module, a first input risk category determination module, a target input risk category determination module, and an input processing module;

[0048] The large model input text acquisition module is used to acquire a first target text, where the first target text is the text to be input into the target large model;

[0049] The first input risk category determination module is used to predict the risk category of the first target text based on a pre-trained first risk category prediction model to obtain a first risk category of the first target text, wherein the first risk category prediction model is trained using a first training text annotated with a first risk category, a first training text annotated with a second risk category, and a first training text annotated with a third risk category, and the first risk category, the second risk category, and the third risk category are harmful, suspected harmful, and harmless, respectively;

[0050] The target input risk category determination module is used to determine the final risk category of the first target text according to the first risk category of the first target text, wherein the final risk category of the first target text is one of harmful and harmless;

[0051] The input processing module is used to output a preset fallback response when the final risk category of the first target text is harmful, and to input the first target text into the target large model when the final risk category of the first target text is harmless.

[0052] In a possible implementation, the text audit device further includes: a large model output text acquisition module, a first output risk category determination module, a target output risk category determination module, and an output processing module;

[0053] The large model output text acquisition module is used to acquire the text of the target large model's reply to the first target text as the second target text when the final risk category of the first target text is harmless;

[0054] The first output risk category determination module is used to predict the risk category of the second target text based on a pre-trained second risk category prediction model to obtain a first risk category of the second target text, wherein the second risk category prediction model is trained using a second training text annotated with the first risk category and a second training text annotated with the third risk category;

[0055] The target output risk category determination module is used to determine the final risk category of the second target text according to the first risk category of the second target text, wherein the final risk category of the second target text is one of harmful and harmless;

[0056] The output processing module is used to output a preset fallback response when the final risk category of the second target text is harmful, and to output the second target text when the final risk category of the second target text is harmless.

[0057] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0058] The memory is used to store computer programs;

[0059] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the above-mentioned text review methods.

[0060] A fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the above-mentioned text review methods.

[0061] A fifth aspect of the present application provides a computer program product, comprising computer-readable instructions, which, when executed on an electronic device, enables the electronic device to implement the steps of any one of the above-described text review methods.

[0062] By means of the above technical scheme, the text review method provided by the present application first obtains the text to be input into the target large model, that is, the first target text, and then based on the first risk category prediction model obtained by pre-training, the final risk category of the first target text is determined according to the first risk category of the first target text. If the final risk category of the first target text is harmful, a preset fallback response is output; if the final risk category of the first target text is harmless, the first target text is input into the target large model. The text review method provided by the present application can review the input text of the target large model, and reviewing the input text of the target large model can identify and intercept harmful input text, so that only harmless input text is input into the target large model, so that the target large model outputs harmless response text. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0064] Figure 1 A schematic diagram of a system architecture involved in this application;

[0065] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0066] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0067] Figure 4 A flowchart of a text review method provided in an embodiment of the present application;

[0068] Figure 5 A flowchart of another text review method provided in an embodiment of the present application;

[0069] Figure 6A schematic diagram of the structure of a text review device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0071] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0072] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0073] In one possible implementation, Figure 1 As shown, the system architecture involved in the present application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The terminal 101 can obtain the text to be reviewed for the large model scene (such as the input text of the large model), and transmit the text to be reviewed to the server 102 through the network, and the server 102 performs text review.

[0074] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal can obtain the text to be reviewed for the large model scene, and then perform text review on the text to be reviewed.

[0075] Next, the product form of the above-mentioned terminal is described.

[0076] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0077] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0078] refer to Figure 2 As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0079] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0080] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0081] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0082] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0083] Among them, the memory 220 can be used to store software codes related to the text review method, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to implement corresponding functions.

[0084] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0085] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0086] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0087] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0088] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0089] Next, the product form of the above server is described.

[0090] Figure 3 A structural diagram of the above server is provided, such as Figure 3As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0091] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0092] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0093] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0094] The memory 304 may be used to store software codes related to the text review method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0095] The processors in the above-mentioned terminals and servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0096] Next, the text review method provided by this application is introduced through the following embodiments.

[0097] See also Figure 4 , shows a flow chart of a text review method provided by an embodiment of the present application, the method may include:

[0098] Step S401: Acquire the first target text.

[0099] The first target text is the text to be input into the target macro model, that is, the input text of the target macro model.

[0100] Step S402: Based on the pre-trained first risk category prediction model, predict the risk category of the first target text to obtain the first risk category of the first target text.

[0101] Among them, the first risk category prediction model is trained using a first training text labeled with a first risk category, a first training text labeled with a second risk category, and a first training text labeled with a third risk category. The first risk category is harmful, the second risk category is suspected harmful, and the third risk category is harmless. The first risk category prediction model is essentially a three-classification model, which can classify the input text as harmful / suspected harmful / harmless.

[0102] Since the first risk category prediction model is a three-classification model, the first risk category obtained by predicting the risk category of the first target text based on the first risk category prediction model is one of the three risk categories of harmful, suspected harmful, and harmless.

[0103] Step S403: Determine a final risk category of the first target text according to the first risk category of the first target text.

[0104] The final risk category of the first target text is one of harmful and harmless.

[0105] In a possible implementation manner, the final risk category of the first target text may be determined only based on the first risk category of the first target text.

[0106] Specifically, the process of determining the final risk category of the first target text based only on the first risk category of the first target text may include: if the first risk category of the first target text is harmful or harmless, determining the first risk category of the first target text as the final risk category of the first target text; if the first risk category of the first target text is suspected harmful, adjusting the first risk category of the first target text to one of harmful and harmless according to a preset risk category adjustment strategy, and determining the adjusted risk category as the final risk category of the first target text.

[0107] It should be noted that the risk category adjustment strategy can be set according to the specific application scenario. In some more stringent scenarios, the risk category adjustment strategy can be set to adjust suspected harmful to harmful. In some more relaxed scenarios, the risk category adjustment strategy can be set to adjust suspected harmful to harmless.

[0108] In another possible implementation, the risk category of the first target text can be determined based on whether the first target text contains keywords in the first keyword set, and the second risk category of the first target text is obtained. Then, the final risk category of the first target text is determined based on the first risk category of the first target text and the second risk category of the first target text. That is, two different risk category determination strategies (a risk category determination strategy based on the first risk category prediction model and a risk category determination strategy based on keywords) are used to determine the risk category of the first target text, and the final risk category of the first target text is determined based on the risk categories determined by the two different risk category determination strategies.

[0109] Among them, the first keyword concentration includes must-kill words and sensitive words. Must-kill words are harmful words that are not allowed to appear. They are independent words that are extremely harmful and have a greater negative impact on society. Sensitive words are words composed of sensitive points and negative words (words that convey negative and negative meanings). Compared with sensitive words, must-kill words have higher risks.

[0110] The process of determining the risk category of the first target text based on whether the first target text contains keywords in the first keyword set may include: if the first target text contains keywords in the first keyword set, determining the risk category of the first target text as harmful; if the first target text does not contain keywords in the first keyword set, determining the risk category of the first target text as harmless.

[0111] After obtaining the first risk category of the first target text and the second risk category of the first target text, the final risk category of the first target text can be determined according to the first risk category of the first target text and in combination with the second risk category of the first target text, specifically including: if the first risk category of the first target text is harmful or harmless, the first risk category of the first target text is determined as the first target risk category of the first target text; if the first risk category of the first target text is suspected harmful, according to a preset risk category adjustment strategy, the first risk category of the first target text is adjusted to one of harmful and harmless, and the adjusted risk category is used as the first target risk category of the first target text; if at least one of the first target risk category of the first target text and the second risk category of the first target text is harmful, the final risk category of the first target text is determined to be harmful; if the first target risk category of the first target text and the second risk category of the first target text are both harmless, the final risk category of the first target text is determined to be harmless.

[0112] Step S404a: If the final risk category of the first target text is harmful, a preset fallback response is output.

[0113] If the first target text is harmful text, the first target text is intercepted, that is, the first target text is no longer input into the target large model, and a preset fallback response is output.

[0114] Step S404b: If the final risk category of the first target text is harmless, the first target text is input into the target macro model.

[0115] This embodiment audits the input text of the target large model. Since the content of the input text is relatively random and is a spontaneous behavior of the user, it often contains highly attacking or misleading content. Auditing the input text can intercept harmful text generated by the target large model induced by the input text.

[0116] Taking into account the situation that the target big model is upgraded in actual applications, after the target big model is upgraded, the response ability of the target big model will be improved. For example, for some harmful or suspected harmful input texts, it can output harmless response texts. In view of this, after the target big model is upgraded, the above-mentioned first keyword set can be updated (for example, some keywords are deleted) according to the response ability of the upgraded target big model to make the first keyword set adapt to the upgraded target big model, and / or, according to the response ability of the upgraded target big model, the risk category marked by some first training texts of the first risk category prediction model is adjusted (for example, the risk category marked by some first training texts marked with the first risk category is adjusted to the second risk category), and the first risk category prediction model is adjusted using the first training text after the risk category is adjusted, so that the adjusted first risk category prediction model is adapted to the upgraded target big model.

[0117] The text review method provided in the embodiment of the present application first obtains the text to be input into the target large model, that is, the first target text, and then based on the pre-trained first risk category prediction model, the final risk category of the first target text is determined according to the first risk category of the first target text. If the final risk category of the first target text is harmful, a preset fallback response is output; if the final risk category of the first target text is harmless, the first target text is input into the target large model. The text review method provided in the embodiment of the present application can review the input text of the target large model, and reviewing the input text of the target large model can identify and intercept harmful input text, so that only harmless input text is input into the target large model, so that the target large model outputs harmless response text.

[0118] The above embodiment mentioned that the risk category of the first target text can be predicted based on the pre-trained first risk category prediction model. Next, the training process of the first risk category prediction model is introduced.

[0119] Before introducing the training process of the first risk category prediction model, the training data set of the first risk category prediction model is first introduced.

[0120] The first risk category prediction model is trained using the training data in the first training data set, and the first training data set includes three types of training data, namely, the first training text annotated with the first risk category (harmful), the first training text annotated with the second risk category (suspected to be harmful), and the first training text annotated with the third risk category (harmless). In order to train a first risk category prediction model with better results, the ratio of the first training text annotated with the first risk category (harmful), the first training text annotated with the second risk category (suspected to be harmful), and the first training text annotated with the third risk category (harmless) can be 1:2:4. This ratio is mainly based on the comprehensive consideration of the actual usage ratio and the reduction of risk omissions. It should be noted that the above 1:2:4 is only an example. This embodiment does not limit the ratio of the three types of first training texts to 1:2:4. The ratio of the three types of first training texts can be determined according to the specific application scenario.

[0121] Among them, the first training text annotated with the third risk category (harmless) is easier to obtain, while the first training text annotated with the first risk category (harmful) and the first training text annotated with the second risk category (suspected harmful) are not easy to obtain. This embodiment focuses on introducing the process of obtaining the first training text annotated with the first risk category (harmful) and the first training text annotated with the second risk category (suspected harmful).

[0122] The process of obtaining the first training text annotated with the first risk category (harmful) and the first training text annotated with the second risk category (suspected to be harmful) may include:

[0123] Step a1: Obtain multiple text samples containing keywords in the first keyword set.

[0124] The first keyword set can be constructed in the following manner: first, a large number of text samples are collected from various channels, and then the large number of collected texts are roughly classified into harmful text samples and harmless text samples to obtain harmful text samples, and finally keywords are extracted from the obtained harmful text samples.

[0125] Step a2: For each acquired text sample, execute:

[0126] Step a21: input the text sample into the big model to obtain the reply text of the big model for the text sample.

[0127] The text sample is input into the big model, and the big model will output the response text for the text sample.

[0128] Step a22-a: If the reply text of the text sample is harmful text, determine the risk category of the text sample as harmful, so as to obtain a first training text marked with a first risk category (harmful).

[0129] Step a22-b: If the reply text of the text sample is harmless text, determine the risk category of the text sample as suspected harmful, so as to obtain a training text marked with a second risk category (suspected harmful).

[0130] This embodiment uses the text reply capability of the large model to determine whether a text sample is harmful text or suspected harmful text.

[0131] Through step a1 and step a2, a plurality of first training texts marked with the first risk category (harmful) and a plurality of first training texts marked with the second risk category (suspected to be harmful) can be obtained.

[0132] Taking into account that the number of harmful texts and suspected harmful texts is usually small, that is, the number of first training texts annotated with the first risk category (harmful) and the number of first training texts annotated with the second risk category (suspected harmful) obtained in the end is small, a text with a different expression but the same semantics as each first training text annotated with the first risk category (harmful) can be generated. Similarly, a text with a different expression but the same semantics as each first training text annotated with the second risk category (suspected harmful) can be generated to expand the number of first training texts of the first risk category and the number of first training texts of the second risk category.

[0133] Next, the training process of the first risk category prediction model is introduced.

[0134] The training process of the first risk category prediction model includes:

[0135] Step b1: Obtain a first training text labeled with a risk category from a first training data set.

[0136] Step b2: input the acquired first training text into the first risk category prediction model to perform risk category prediction to obtain a risk category prediction result of the first training text.

[0137] The acquired first training text is input into the first risk category prediction model, and the first risk category prediction model predicts the probability that the risk category of the first training text is harmful, suspected harmful, or harmless.

[0138] Step b3: determining the risk category prediction loss according to the risk category prediction result of the first training text and the risk category annotated by the first training text.

[0139] The risk category prediction loss is determined according to the probability that the risk category of the first training text is harmful, suspected harmful, or harmless and the risk category annotated by the first training text. The risk category prediction loss may adopt a cross entropy loss, and the calculation method of the cross entropy loss is an existing technology, which will not be described in detail in this embodiment.

[0140] Step b4: predict the loss according to the risk category and update the parameters of the first risk category prediction model.

[0141] The first risk category prediction model in this embodiment can be an Albert model. The Albert model has relatively fewer parameters and a relatively fast inference speed. Of course, this embodiment is not limited to this. For example, the first risk category prediction model can also be a simple FastText model.

[0142] The first risk category prediction model is iteratively trained multiple times according to the process of steps b1 to b4 above until the training end condition is met (such as reaching a preset number of training iterations, model convergence, etc.). The first risk category prediction model obtained through training can be used to predict the risk category of the input text of the target large model.

[0143] Considering that sometimes, after the text that has passed the review is input into the target large model, the target large model may still output harmful reply text. In view of this, the embodiment of the present application provides another text review method, such as Figure 5 As shown, the text review method includes not only the above steps S401 to S404 (input review process), but also the following output review process:

[0144] Step S405: When the final risk category of the first target text is harmless, the text in which the target large model responds to the first target text is obtained as the second target text.

[0145] The text that the target large model responds to the first target text is the output text of the target large model.

[0146] Step S406: Based on the pre-trained second risk category prediction model, predict the risk category of the second target text to obtain the first risk category of the second target text.

[0147] The second risk category prediction model is trained using a second training text labeled with the first risk category (harmful) and a second training text labeled with the third risk category (harmless).

[0148] Specifically, based on the pre-trained second risk category prediction model, the risk category prediction of the second target text is performed to obtain the first risk category of the second target text, and the process may include:

[0149] Step S4061: Based on the pre-trained second risk category prediction model, predict the probability that the risk category of the second target text is harmful.

[0150] The second risk category prediction model in this embodiment is a binary classification model. By inputting the second target text into the pre-trained second risk category prediction model, the probability that the risk category of the second target text is harmful and the probability that the risk category of the second target text is harmless can be obtained. This embodiment uses the probability that the risk category of the second target text is harmful to determine the first risk category of the second target text.

[0151] Step S4062-a: If the probability that the risk category of the second target text is harmful is greater than or equal to a preset first probability threshold, determine that the first risk category of the second target text is harmful.

[0152] Exemplarily, the first probability threshold is 0.6. If the probability that the risk category of the second target text is harmful is greater than or equal to 0.6, then the first risk category of the second target text is determined to be harmful. That is, if the probability that the risk category of the second target text is harmful is in the interval [0.6, 1], then the first risk category of the second target text is determined to be harmful.

[0153] Step S4062-b: If the probability that the risk category of the second target text is harmful is greater than the preset second probability threshold and less than the preset first probability threshold, determine that the first risk category of the second target text is suspected harmful.

[0154] The second probability threshold is smaller than the first probability threshold.

[0155] Exemplarily, the first probability threshold is 0.6, and the second probability threshold is 0.3. If the probability that the risk category of the second target text is harmful is greater than 0.3 and less than 0.6, then the first risk category of the second target text is determined to be suspected harmful, that is, the probability that the risk category of the second target text is harmful is within the interval (0.3, 0.6), then the first risk category of the second target text is determined to be suspected harmful.

[0156] Step S4062-c: if the probability that the risk category of the second target text is harmful is less than or equal to a preset second probability threshold, determine that the first risk category of the second target text is harmless.

[0157] Exemplarily, the second probability threshold is 0.3. If the probability that the risk category of the second target text is harmful is less than or equal to 0.3, the first risk category of the second target text is determined to be harmless. That is, if the probability that the risk category of the second target text is harmful is in the interval [0, 0.3], the first risk category of the second target text is determined to be harmless.

[0158] It should be noted that the first probability threshold of 0.6 and the second probability threshold of 0.3 are only examples. This embodiment does not limit the first probability threshold to 0.6 and the second probability threshold to 0.3. The specific values ​​of the two probability thresholds can be determined according to the actual scenario.

[0159] Through the above process, it can be known that the first risk category of the second target text is one of harmful, suspected harmful, and harmless.

[0160] Step S407: Determine a final risk category of the second target text according to the first risk category of the second target text.

[0161] The final risk category of the second target text is one of harmful and harmless.

[0162] In a possible implementation manner, the final risk category of the second target text may be determined only according to the first risk category of the second target text.

[0163] Specifically, the process of determining the final risk category of the second target text based only on the first risk category of the second target text may include: if the first risk category of the second target text is harmful or harmless, then determining the first risk category of the second target text as the final risk category of the second target text; if the first risk category of the second target text is suspected harmful, then adjusting the first risk category of the second target text to one of harmful and harmless according to a preset risk category adjustment strategy, and determining the adjusted risk category as the final risk category of the second target text. Among them, the risk category adjustment strategy can be set according to the specific application scenario. In some stricter scenarios, the risk category adjustment strategy can be set to adjust suspected harmful to harmful, and in some looser scenarios, the risk category adjustment strategy can be set to adjust suspected harmful to harmless.

[0164] In another possible implementation, the risk category of the second target text can be determined based on whether the second target text contains keywords in the second keyword set, and the second risk category of the second target text is obtained (if the second target text contains keywords in the second keyword set, the risk category of the second target text is determined to be harmful; if the second target text does not contain keywords in the second keyword set, the risk category of the second target text is determined to be harmless), and then, according to the first risk category of the second target text, combined with the second risk category of the second target text, the final risk category of the second target text is determined. Among them, the second keyword set includes sensitive words and negative words.

[0165] The process of determining the final risk category of the second target text according to the first risk category of the second target text and in combination with the second risk category of the second target text may include: if the first risk category of the second target text is harmful or harmless, determining the first risk category of the second target text as the first target risk category of the second target text; if the first risk category of the second target text is suspected harmful, adjusting the first risk category of the second target text to one of harmful and harmless according to a preset risk category adjustment strategy, and using the adjusted risk category as the first target risk category of the second target text; if at least one of the first target risk category of the second target text and the second risk category of the second target text is harmful, determining the final risk category of the second target text to be harmful; if both the first target risk category of the second target text and the second risk category of the second target text are harmless, determining the final risk category of the second target text to be harmless.

[0166] Step S408a: If the final risk category of the second target text is harmful, a preset fallback response is output.

[0167] If the final risk category of the second target text is harmful, the second target text is not output, but a preset fallback response is output.

[0168] Step S408b: If the final risk category of the second target text is harmless, the second target text is output.

[0169] The text review method provided in the embodiment of the present application first obtains the input text of the target large model, that is, the first target text, and then predicts the risk category of the first target text based on the first risk category prediction model obtained by pre-training to obtain the first risk category of the first target text, and then determines the final risk category of the first target text according to the first risk category of the first target text (which can be combined with the risk category determined based on keywords at the same time), if the final risk category of the first target text is harmful, then outputs a preset catch-all response, if the final risk category of the first target text is harmless, then inputs the first target text into the target large model, and when the final risk category of the first target text is harmless, obtains the text of the target large model's response to the first target text, that is, the second target text, and predicts the risk category of the second target text based on the second risk category prediction model obtained by pre-training to obtain the first risk category of the second target text, and determines the final risk category of the second target text according to the first risk category of the second target text (which can be combined with the risk category determined based on keywords at the same time), if the final risk category of the second target text is harmful, then outputs a preset catch-all response, and if the final risk category of the second target text is harmless, then outputs the second target text. The text audit method provided in the embodiment of the present application can audit the input text of the target large model. Auditing the input text can identify and intercept harmful input, so that only harmless text is input into the target large model, so that the target large model outputs harmless reply text. After the harmless text is input into the target large model, the text that the target large model replies to the input text, that is, the output text, can also be audited. Auditing the output text can ensure that the large model outputs harmless reply text. The text audit method provided in the embodiment of the present application ensures that the large model outputs harmless text through input audit and output audit.

[0170] The above embodiment mentioned that the risk category of the second target text can be predicted based on the pre-trained second risk category prediction model. Next, the training process of the second risk category prediction model is introduced.

[0171] Before introducing the training process of the second risk category prediction model, the training data set of the second risk category prediction model is first introduced.

[0172] The second risk category prediction model is trained using the training data in the second training data set. The second training data set includes multiple training triplets, each training triplet includes an anchor sample, a positive sample corresponding to the anchor sample, and a negative sample corresponding to the anchor sample. The anchor sample and the positive sample corresponding to the anchor sample are both the second training text labeled with the third risk category (harmless), and the negative sample corresponding to the anchor sample is the second training text labeled with the first risk category (harmful).

[0173] The training triplets can be obtained as follows:

[0174] Step c1: Obtain harmful or suspected harmful text samples.

[0175] Step c2-a: input the harmful or suspected harmful text sample into the conventional large model to obtain a plurality of harmless reply texts generated by the conventional large model for the harmful or suspected harmful text sample.

[0176] Based on the conventional large model, a plurality of second training texts annotated with the third risk category (harmless) may be obtained.

[0177] Step c2-b: input harmful or suspected harmful text samples into a large model that is prone to generating harmful texts, and obtain harmful reply texts generated by the large model that is prone to generating harmful texts for the input texts.

[0178] A large model that is prone to generate harmful text can be pre-trained, and then harmful or suspected harmful text samples are input into the large model to obtain harmful reply text output by the large model. Based on the large model that is prone to generate harmful text, a second training text marked with the first risk category (harmful) can be obtained.

[0179] Step c3, obtain two harmless reply texts from the multiple harmless reply texts obtained in step c2-a, use one of the harmless reply texts as an anchor sample, use the other harmless reply text as a positive sample corresponding to the anchor sample, and use the harmful reply text obtained in step c2-b as a negative sample corresponding to the anchor sample, thereby obtaining a training triplet.

[0180] In the above manner, a plurality of training triplets can be obtained, and the second training data set is composed of the obtained plurality of training triplets.

[0181] Next, the training process of the second risk category prediction model is introduced.

[0182] The training process of the second risk category prediction model may include:

[0183] Step d1, obtaining training triples from the second training data set.

[0184] The triplet includes an anchor sample, a positive sample corresponding to the anchor sample, and a negative sample corresponding to the anchor sample.

[0185] Step d2: Based on the second risk category prediction model, text encoding is performed on the anchor sample, the positive sample corresponding to the anchor sample, and the negative sample corresponding to the anchor sample to obtain the text features of the anchor sample, the text features of the positive sample corresponding to the anchor sample, and the text features of the negative sample corresponding to the anchor sample.

[0186] The second risk category prediction model includes a text encoder, which inputs the anchor sample, the positive sample corresponding to the anchor sample, and the negative sample corresponding to the anchor sample into the text encoder for encoding to obtain the text features of the anchor sample, the text features of the positive sample corresponding to the anchor sample, and the text features of the negative sample corresponding to the anchor sample.

[0187] Step d3-a: Based on the second risk category prediction model and the text features of the anchor sample, predict the risk category of the anchor sample.

[0188] The second risk category prediction model also includes a risk category prediction module, into which the text features of the anchor sample are input, and the risk category prediction module outputs the probability that the category of the anchor sample is harmful and the probability that it is harmless.

[0189] Step d3-b: Calculate the distance between the anchor sample and the positive sample corresponding to the anchor sample based on the text features of the anchor sample and the text features of the positive sample corresponding to the anchor sample, and calculate the distance between the anchor sample and the negative sample corresponding to the anchor sample based on the text features of the anchor sample and the text features of the negative sample corresponding to the anchor sample.

[0190] Calculate the similarity between the text features of the anchor sample and the text features of the positive sample corresponding to the anchor sample as the distance between the anchor sample and the positive sample corresponding to the anchor sample. Similarly, calculate the similarity between the text features of the anchor sample and the text features of the negative sample corresponding to the anchor sample as the distance between the anchor sample and the negative sample corresponding to the anchor sample.

[0191] Step d4, update the parameters of the second risk category prediction model with the goal of making the predicted risk category consistent with the risk category marked by the anchor sample and making the distance between the positive samples corresponding to the anchor samples smaller than the distance between the negative samples corresponding to the anchor samples.

[0192] Specifically, the risk category prediction loss L1 is determined according to the risk category prediction probability of the anchor sample (i.e., the probability that the anchor sample is harmful and the probability that it is harmless) and the actual risk category of the anchor sample (the labeled risk category). The triplet loss is determined according to the distance between the positive samples corresponding to the anchor samples and the distance between the negative samples corresponding to the anchor samples. The risk category prediction loss is fused with the triplet loss, and the parameters of the second risk category prediction model are updated according to the fused loss.

[0193] The risk category predicted loss L1 can be calculated as follows:

[0194] L1=-(y1log(p1)+ y2log(p2)) (1)

[0195] Among them, p1 is the probability that the risk category of the anchor sample is harmful, and p2 is the probability that the risk category of the anchor sample is harmless. If the true risk category of the anchor sample is harmful, then y1=1, y2=0; if the true risk category of the anchor sample is harmless, then y1=0, y2=1.

[0196] The triplet loss L2 can be calculated as follows:

[0197] L2=max(d(a,p)- d(a,n)+margin, 0) (2)

[0198] Among them, a represents the anchor sample, p represents the positive sample corresponding to the anchor sample, n represents the negative sample corresponding to the anchor sample, d(a,p) represents the distance between the anchor sample and the positive sample corresponding to the anchor sample, d(a,n) represents the distance between the anchor sample and the negative sample corresponding to the anchor sample, margin is a hyperparameter, and the triplet loss is used to shorten the distance between a and p and increase the distance between a and n.

[0199] After obtaining the risk category prediction loss L1 and the triplet loss L2, the risk category prediction loss L1 and the triplet loss L2 can be fused in the following fusion method to obtain the final loss L all :

[0200] L all =w1L1+ w2L2(3)

[0201] Among them, w1 represents the weight corresponding to L1, w2 represents the weight corresponding to L2, and the specific values ​​of w1 and w2 can be determined according to the actual application scenario.

[0202] The second risk category prediction model is iteratively trained multiple times according to the process of the above steps d1 to d4 until the training end condition is met (such as reaching a preset number of training iterations, model convergence, etc.). The main framework of the second risk category prediction model can be, but is not limited to, a 12-layer architecture using the encoder part of the transformer.

[0203] The above introduces the text review method provided in the embodiment of the present application, and the device corresponding to the above text review method will be introduced below.

[0204] See also Figure 6 , Figure 6 A structural schematic diagram of a text review device provided in an embodiment of the present application, the text review device may include: a large model input text acquisition module 601, a first input risk category determination module 602a, a target input risk category determination module 603 and an input processing module 604.

[0205] The large model input text acquisition module 601 is used to acquire a first target text, where the first target text is the text to be input into the target large model.

[0206] The first input risk category determination module 602a is used to predict the risk category of the first target text based on the first risk category prediction model obtained by pre-training, so as to obtain the first risk category of the first target text.

[0207] The first risk category prediction model is trained using a first training text labeled with the first risk category, a first training text labeled with the second risk category, and a first training text labeled with the third risk category. The first risk category, the second risk category, and the third risk category are harmful, suspected harmful, and harmless, respectively.

[0208] The target input risk category determination module 603 is used to determine the final risk category of the first target text according to the first risk category of the first target text.

[0209] The final risk category of the first target text is one of harmful and harmless.

[0210] The input processing module 604 is used to output a preset fallback response when the final risk category of the first target text is harmful, and to input the first target text into the target large model when the final risk category of the first target text is harmless.

[0211] In a possible implementation, the text review device may further include: a large model output text acquisition module 605 , a first output risk category determination module 606 a , a target output risk category determination module 607 , and an output processing module 608 .

[0212] The large model output text acquisition module 605 is used to obtain the text of the target large model's reply to the first target text as the second target text when the final risk category of the first target text is harmless.

[0213] The first output risk category determination module 606a is used to predict the risk category of the second target text based on the pre-trained second risk category prediction model to obtain the first risk category of the second target text.

[0214] The second risk category prediction model is trained by using the second training text annotated with the first risk category and the second training text annotated with the third risk category.

[0215] The target output risk category determination module 607 is used to determine the final risk category of the second target text according to the first risk category of the second target text, wherein the final risk category of the second target text is one of harmful and harmless.

[0216] The output processing module 608 is used to output a preset fallback response when the final risk category of the second target text is harmful, and to output the second target text when the final risk category of the second target text is harmless.

[0217] In a possible implementation, the text review apparatus may further include: a second input risk category determination module 602b.

[0218] The second input risk category determination module 602b is used to determine the risk category of the first target text based on whether the first target text contains keywords in the first keyword set, so as to obtain a second risk category of the first target text.

[0219] Among them, the first keyword set includes must-kill words and sensitive words. Must-kill words are harmful, independent words that are not allowed to appear, and sensitive words are words composed of sensitive points and negative words.

[0220] When determining the final risk category of the first target text according to the first risk category of the first target text, the target input risk category determination module 603 is specifically used to:

[0221] A final risk category of the first target text is determined according to the first risk category of the first target text in combination with the second risk category of the first target text.

[0222] In a possible implementation, the target input risk category determination module 603 determines the final risk category of the first target text according to the first risk category of the first target text and in combination with the second risk category of the first target text, including:

[0223] If the first risk category of the first target text is harmful or harmless, the first risk category of the first target text is determined as the first target risk category of the first target text; if the first risk category of the first target text is suspected harmful, the first risk category of the first target text is adjusted to one of harmful and harmless according to the preset risk category adjustment strategy, and the adjusted risk category is used as the first target risk category of the first target text;

[0224] If at least one of the first target risk category of the first target text and the second risk category of the first target text is harmful, determining that the final risk category of the first target text is harmful, and the second risk category of the first target text is one of harmful and harmless;

[0225] If the first target risk category of the first target text and the second risk category of the first target text are both harmless, then the final risk category of the first target text is determined to be harmless.

[0226] In a possible implementation, the text review device may further include: a training data acquisition module. When acquiring the first training text annotated with the first risk category and the first training text annotated with the second risk category, the training data acquisition module is specifically configured to:

[0227] Obtaining multiple text samples containing keywords in the first keyword set;

[0228] For each text sample obtained:

[0229] Input the text sample into the big model to obtain the reply text of the big model for the text sample;

[0230] If the reply text of the text sample is harmful text, then determining the risk category of the text sample is harmful, so as to obtain a first training text marked with a first risk category;

[0231] If the reply text of the text sample is harmless text, the risk category of the text sample is determined to be suspected harmful, so as to obtain a first training text marked with a second risk category.

[0232] In a possible implementation, the text review device may further include: a keyword updating module and / or a model adjusting module.

[0233] The keyword updating module is used to update the first keyword set according to the recovery capability of the upgraded target large model after the target large model is upgraded.

[0234] The model adjustment module is used to adjust the risk category annotated by the first training text of the first risk category prediction model according to the recovery capability of the upgraded target large model, and adjust the first risk category prediction model using the first training text after the risk category is adjusted.

[0235] In a possible implementation, when the first output risk category determination module 606a predicts the risk category of the second target text based on the pre-trained second risk category prediction model to obtain the first risk category of the second target text, it is specifically used to:

[0236] Predicting the probability that the risk category of the second target text is harmful based on the pre-trained second risk category prediction model;

[0237] If the probability that the risk category of the second target text is harmful is greater than or equal to a preset first probability threshold, determining that the first risk category of the second target text is harmful;

[0238] If the probability that the risk category of the second target text is harmful is greater than a preset second probability threshold and less than a preset first probability threshold, then determining that the first risk category of the second target text is suspected harmful, wherein the second probability threshold is less than the first probability threshold;

[0239] If the probability that the risk category of the second target text is harmful is less than or equal to a preset second probability threshold, the first risk category of the second target text is determined to be harmless.

[0240] In a possible implementation, the text review apparatus may further include: a second output risk category determination module 606b.

[0241] The second output risk category determination module 606b is used to determine the risk category of the second target text based on whether the second target text contains keywords in the second keyword set, so as to obtain a second risk category of the second target text.

[0242] Among them, the second keyword set includes sensitive words and negative words, and the sensitive words are words composed of sensitive points and negative words.

[0243] When determining the final risk category of the second target text according to the first risk category of the second target text, the target output risk category determination module 607 is specifically used to:

[0244] A final risk category of the second target text is determined based on the first risk category of the second target text and in combination with the second risk category of the second target text.

[0245] In a possible implementation, the text review device may further include: a model training module.

[0246] Model training module, used to:

[0247] Obtaining a training triplet, the training triplet includes an anchor sample, a positive sample corresponding to the anchor sample, and a negative sample corresponding to the anchor sample, the anchor sample and the positive sample corresponding to the anchor sample are both the second training text annotated with the third risk category, and the negative sample corresponding to the anchor sample is the second training text annotated with the first risk category;

[0248] Based on the second risk category prediction model, text encoding is performed on the anchor sample, the positive sample corresponding to the anchor sample, and the negative sample corresponding to the anchor sample to obtain the text features of the anchor sample, the text features of the positive sample corresponding to the anchor sample, and the text features of the negative sample corresponding to the anchor sample;

[0249] Based on the second risk category prediction model, the risk category of the anchor sample is predicted based on the text features of the anchor sample;

[0250] According to the text features of the anchor sample and the text features of the positive sample corresponding to the anchor sample, the distance between the anchor sample and the positive sample corresponding to the anchor sample is calculated, and according to the text features of the anchor sample and the text features of the negative sample corresponding to the anchor sample, the distance between the anchor sample and the negative sample corresponding to the anchor sample is calculated;

[0251] The parameters of the second risk category prediction model are updated with the goal of making the predicted risk category consistent with the risk category marked by the anchor sample and making the distance between the positive samples corresponding to the anchor samples smaller than the distance between the negative samples corresponding to the anchor samples.

[0252] The text auditing device provided in the embodiment of the present application can audit the input text of the target large model. Auditing the input text can identify and intercept harmful input, so that only harmless text is input into the target large model, so that the target large model outputs harmless reply text. After the harmless text is input into the target large model, the text auditing device provided in the embodiment of the present application can also audit the text that the target large model replies to the input text, that is, the output text. Auditing the output text can ensure that the large model outputs harmless reply text. The text auditing device provided in the embodiment of the present application ensures that the large model outputs harmless text through input auditing and output auditing.

[0253] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0254] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0255] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0256] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0257] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the text review method provided in the above embodiment.

[0258] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the text review method provided in the above embodiment.

[0259] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the text review method provided in the above embodiment.

[0260] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0261] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0262] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0263] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A text review method, characterized in that: include: Acquire a first target text, where the first target text is a text to be input into a target macro model; Based on the first risk category prediction model obtained by pre-training, the risk category of the first target text is predicted to obtain a first risk category of the first target text, wherein the first risk category prediction model is trained using a first training text annotated with a first risk category, a first training text annotated with a second risk category, and a first training text annotated with a third risk category, and the first risk category, the second risk category, and the third risk category are harmful, suspected harmful, and harmless, respectively; Determining a final risk category of the first target text according to the first risk category of the first target text, wherein the final risk category of the first target text is one of harmful and harmless; If the final risk category of the first target text is harmful, a preset catch-all response is output; if the final risk category of the first target text is harmless, the first target text is input into the target macro model.

2. The text review method according to claim 1, characterized in that: Also includes: When the final risk category of the first target text is harmless, obtaining the text in which the target large model responds to the first target text as the second target text; Based on the pre-trained second risk category prediction model, predict the risk category of the second target text to obtain the first risk category of the second target text, wherein the second risk category prediction model is trained using the second training text annotated with the first risk category and the second training text annotated with the third risk category; Determining a final risk category of the second target text according to the first risk category of the second target text, wherein the final risk category of the second target text is one of harmful and harmless; If the final risk category of the second target text is harmful, a preset fallback response is output; if the final risk category of the second target text is harmless, the second target text is output.

3. The text review method according to claim 1, characterized in that: Also includes: Determining the risk category of the first target text based on whether the first target text contains keywords in the first keyword set, and obtaining a second risk category of the first target text, wherein the first keyword set includes must-kill words and sensitive words, the must-kill words are harmful, independent words that are not allowed to appear, and the sensitive words are words composed of sensitive points and negative words; The determining, according to the first risk category of the first target text, a final risk category of the first target text includes: A final risk category of the first target text is determined according to the first risk category of the first target text in combination with the second risk category of the first target text.

4. The text review method according to claim 3, characterized in that: The determining the final risk category of the first target text according to the first risk category of the first target text and in combination with the second risk category of the first target text comprises: If the first risk category of the first target text is harmful or harmless, the first risk category of the first target text is determined as the first target risk category of the first target text; if the first risk category of the first target text is suspected harmful, the first risk category of the first target text is adjusted to one of harmful and harmless according to a preset risk category adjustment strategy, and the adjusted risk category is used as the first target risk category of the first target text; If at least one of the first target risk category of the first target text and the second risk category of the first target text is harmful, determining that the final risk category of the first target text is harmful, and the second risk category of the first target text is one of harmful and harmless; If the first target risk category of the first target text and the second risk category of the first target text are both harmless, then the final risk category of the first target text is determined to be harmless.

5. The text review method according to claim 1, characterized in that: The process of acquiring the first training text annotated with the first risk category and the first training text annotated with the second risk category includes: Acquire multiple text samples containing keywords in a first keyword set, wherein the first keyword set includes must-kill words and sensitive words, the must-kill words are harmful, independent words that are not allowed to appear, and the sensitive words are words composed of sensitive points and negative words; For each text sample obtained: Input the text sample into the big model to obtain the reply text of the big model for the text sample; If the reply text of the text sample is harmful text, then determining the risk category of the text sample is harmful, so as to obtain a first training text marked with a first risk category; If the reply text of the text sample is harmless text, the risk category of the text sample is determined to be suspected harmful, so as to obtain a first training text marked with a second risk category.

6. The text review method according to claim 3, characterized in that: Also includes: After the target large model is upgraded, the first keyword set is updated according to the recovery capability of the upgraded target large model; And / or, according to the recovery capability of the upgraded target large model, the risk category annotated by the first training text of the first risk category prediction model is adjusted, and the first risk category prediction model is adjusted using the first training text after the risk category is adjusted.

7. The text review method according to claim 2, characterized in that: The step of predicting the risk category of the second target text based on the pre-trained second risk category prediction model to obtain the first risk category of the second target text includes: Predicting the probability that the risk category of the second target text is harmful based on the pre-trained second risk category prediction model; If the probability that the risk category of the second target text is harmful is greater than or equal to a preset first probability threshold, determining that the first risk category of the second target text is harmful; If the probability that the risk category of the second target text is harmful is greater than a preset second probability threshold and less than a preset first probability threshold, determining that the first risk category of the second target text is suspected harmful, wherein the second probability threshold is less than the first probability threshold; If the probability that the risk category of the second target text is harmful is less than or equal to a preset second probability threshold, it is determined that the first risk category of the second target text is harmless.

8. The text review method according to claim 2, characterized in that: Also includes: Determining the risk category of the second target text based on whether the second target text contains keywords in the second keyword set, thereby obtaining a second risk category of the second target text, wherein the second keyword set includes sensitive words and negative words, and the sensitive words are words composed of sensitive points and negative words; The determining, according to the first risk category of the second target text, a final risk category of the second target text comprises: A final risk category of the second target text is determined according to the first risk category of the second target text and in combination with the second risk category of the second target text.

9. The text review method according to claim 2, characterized in that: The training process of the second risk category prediction model includes: Obtaining a training triplet, wherein the training triplet includes an anchor sample, a positive sample corresponding to the anchor sample, and a negative sample corresponding to the anchor sample, wherein the anchor sample and the positive sample corresponding to the anchor sample are both the second training text annotated with the third risk category, and the negative sample corresponding to the anchor sample is the second training text annotated with the first risk category; Based on the second risk category prediction model, respectively, the anchor sample, the positive sample corresponding to the anchor sample, and the negative sample corresponding to the anchor sample are text-encoded to obtain the text features of the anchor sample, the text features of the positive sample corresponding to the anchor sample, and the text features of the negative sample corresponding to the anchor sample; Based on the second risk category prediction model, and taking the text features of the anchor sample as a basis, predicting the risk category of the anchor sample; Calculating the distance between the anchor sample and the positive sample corresponding to the anchor sample according to the text features of the anchor sample and the text features of the positive sample corresponding to the anchor sample, and calculating the distance between the anchor sample and the negative sample corresponding to the anchor sample according to the text features of the anchor sample and the text features of the negative sample corresponding to the anchor sample; The parameters of the second risk category prediction model are updated with the goal of making the predicted risk category consistent with the risk category marked by the anchor sample and making the distance between the anchor sample and the positive sample corresponding to the anchor sample smaller than the distance between the anchor sample and the negative sample corresponding to the anchor sample.

10. A text review device, characterized in that: include: A large model input text acquisition module, a first input risk category determination module, a target input risk category determination module and an input processing module; The large model input text acquisition module is used to acquire a first target text, where the first target text is the text to be input into the target large model; The first input risk category determination module is used to predict the risk category of the first target text based on a pre-trained first risk category prediction model to obtain a first risk category of the first target text, wherein the first risk category prediction model is trained using a first training text annotated with a first risk category, a first training text annotated with a second risk category, and a first training text annotated with a third risk category, and the first risk category, the second risk category, and the third risk category are harmful, suspected harmful, and harmless, respectively; The target input risk category determination module is used to determine the final risk category of the first target text according to the first risk category of the first target text, wherein the final risk category of the first target text is one of harmful and harmless; The input processing module is used to output a preset fallback response when the final risk category of the first target text is harmful, and to input the first target text into the target large model when the final risk category of the first target text is harmless.

11. The text review device according to claim 10, characterized in that: Also includes: A large model output text acquisition module, a first output risk category determination module, a target output risk category determination module and an output processing module; The large model output text acquisition module is used to acquire the text of the target large model's reply to the first target text as the second target text when the final risk category of the first target text is harmless; The first output risk category determination module is used to predict the risk category of the second target text based on a pre-trained second risk category prediction model to obtain a first risk category of the second target text, wherein the second risk category prediction model is trained using a second training text annotated with the first risk category and a second training text annotated with the third risk category; The target output risk category determination module is used to determine the final risk category of the second target text according to the first risk category of the second target text, wherein the final risk category of the second target text is one of harmful and harmless; The output processing module is used to output a preset fallback response when the final risk category of the second target text is harmful, and to output the second target text when the final risk category of the second target text is harmless.

12. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the text review method as described in any one of claims 1 to 9.

13. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the text review method as described in any one of claims 1 to 9.

14. A computer program product, characterized in that It includes computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the text review method as described in any one of claims 1 to 9.

Images