Method for detecting text toxicity by using large language model output difference

Abstract

The invention relates to the technical field of text analysis, in particular to a method for detecting text toxicity by using large language model output difference, which comprises the following steps: extracting logits from text content input by a user; constructing a deep learning model, wherein the deep model is used for extracting useful information from the extracted logits; performing model training on the constructed deep learning model; and performing toxicity detection on the new prompt by using the trained deep learning model. According to the invention, the precision and efficiency of toxicity detection can be improved, and tiny toxicity characteristics in LLM output can be accurately captured; the toxicity detection cost is reduced, and the operation cost investment is reduced; the flexibility and the applicability of toxicity detection are enhanced, and the application requirements of different LLMs or changes are met.

Description

Technical Field

[0001] The present invention relates to the technical field of text analysis, and in particular to a method for detecting text toxicity by utilizing output differences of a large language model. Background Art

[0002] With the rapid development of artificial intelligence technology, large language models (LLMs) have been widely used in many fields such as text generation and dialogue systems. However, as the amount of data used to train LLMs has increased significantly, the amount of bad content, bias, and toxic information contained in them has also increased. Although these problems are prevalent in training data, they have not yet been fully identified and resolved. As a result, when LLMs generate text, they may generate biased, toxic, or privacy-sensitive content, which not only seriously affects the user experience, but also poses a major threat to the security of the platform.

[0003] Existing technologies mainly use specific toxicity detectors, such as the OpenAI Moderation API, which uses the highest score in the toxicity subcategory to analyze the text content generated by the LLM to determine whether it contains toxic or inappropriate information. The OpenAI Moderation API can accurately identify toxic or inappropriate content in the text and give corresponding evaluation results. However, it mainly relies on the analysis of text content and cannot accurately capture the subtle toxicity features in the LLM output; it is also expensive and is usually optimized for specific LLMs or application scenarios, making it difficult to adapt to different LLMs or changing application requirements.

[0004] Existing technologies also try to improve the security of LLM output by improving its training process. For example, by introducing alignment technology to enhance the alignment performance of LLM, it can make it more compliant with ethical and regulatory requirements when generating text. However, it requires sufficient data that reflects human values ​​and expectations, and requires a lot of resources and time, which increases the difficulty and cost of the alignment process, and has low stability.

[0005] In summary, there is an urgent need for an efficient and accurate text toxicity detection method to ensure the security and compliance of the content output by LLM. Summary of the invention

[0006] To solve the above problems, the present invention provides a method for detecting text toxicity by using the output difference of a large language model, which is used to improve the accuracy and efficiency of toxicity detection and accurately capture the subtle toxicity characteristics in the LLM output; reduce the cost of toxicity detection and reduce operating cost investment; enhance the flexibility and applicability of toxicity detection and adapt to different LLMs or changing application requirements.

[0007] In order to achieve the above object, the technical solution of the present invention is as follows: A method for detecting text toxicity using a large language model output difference comprises the following steps:

[0008] Extract logits based on the text content entered by the user;

[0009] Build a deep learning model, which is used to extract useful information from the extracted logits;

[0010] Conduct model training on the constructed deep learning model;

[0011] Use the trained deep learning model to detect toxicity of new prompts.

[0012] Further, extracting logits includes:

[0013] When LLM receives the prompt text entered by the user;

[0014] Locate the start position of the prompt text response;

[0015] Extract n labeled logits from the starting position.

[0016] Furthermore, logits are represented as a vector l(x)∈R n , where n represents the number of tags, and each element of the vector corresponds to the logits value of a tag;

[0017] Apply additional processing function f to the logits: R n →R n ;

[0018] Define the additional function f as follows:

[0019] Perform Min-Max normalization on logits to scale the value of logits to the range of [0, 1];

[0020] Applying the ReLU activation function introduces nonlinear characteristics, and the mathematical expression is:

[0021]

[0022] Among them, l is the logits vector of the current batch or the entire training set, and f i (l i ) is the value of the i-th element after processing.

[0023] Further, the deep learning model is any one of an SLR model, a deep neural network, a deep neural network, and a recurrent neural network;

[0024] Or it can be an integration of any number of SLR models, deep neural networks, deep neural networks, and recurrent neural networks.

[0025] Furthermore, the deep learning model is an SLR model. The SLR model is constructed:

[0026] The weights and biases of the SLR model are represented by w∈R n and b∈R, the output formula is:

[0027] SLR(x)=w·f(l(x))+b

[0028] Among them, f(l(x)) is the logits vector after being processed by the additional function f.

[0029] Further, SLR model training:

[0030] The weights and biases of the SLR model are optimized to minimize the following loss function:

[0031]

[0032] Where x, y∈X represents the samples in the training set, each sample consists of a prompt x and a corresponding toxicity label y∈{0,1}; BCE() represents the binary cross entropy loss function, which is used to measure the difference between the predicted value and the true value; represents the L2 regularization term, that is, the sum of the squares of each element of the weight vector w, which is used to prevent the model from overfitting; λ is the regularization coefficient, which is used to control the strength of the regularization.

[0033] Furthermore, the SLR model performs toxicity detection on new cues:

[0034] When a new prompt text is received, the logits vector l′ of all n tags at the starting position of the LLM response is first extracted, and then input into the trained SLR model; the SLR model outputs a probability value, namely Sigmoid(SLR(l′)), indicating the possibility that the prompt is toxic; based on this probability value and a preset threshold, it is determined whether the prompt is toxic and the detection result is output.

[0035] The above scheme has the following beneficial effects:

[0036] 1. The present invention uses the introspection function of LLM to extract key information from the logits output by the model, and further analyzes it with the sparse logistic regression (SLR) model to significantly improve the accuracy of toxicity detection

[0037] 2. The present invention avoids the use of additional toxicity detectors and reduces operating costs

[0038] 3. The technical solution of the present invention is flexible in design and is not only applicable to a variety of LLM models, but also can adapt to different application scenarios and needs. By adjusting the model parameters and training data sets, customized toxicity detection for specific fields or scenarios can be easily achieved, which has strong flexibility and applicability.

[0039] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a general flow chart of an embodiment of a method for detecting text toxicity by using a large language model output difference according to the present invention;

[0041] Figure 2 The present invention is a flowchart of toxicity detection according to an embodiment of a method for detecting text toxicity using large language model output differences. DETAILED DESCRIPTION

[0042] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0043] The following is further described in detail through specific implementation methods:

[0044] Implementation example Figure 1 As shown in the figure, the method of detecting text toxicity by using the output difference of a large language model mainly includes the following steps:

[0045] S1. Extract logits based on the text content input by the user;

[0046] When LLM receives the prompt text input by the user, this embodiment first locates the starting position of the response and extracts the logits of all n tokens from that position. These logits represent the model's original prediction scores for each possible category of each token, which is key information for subsequent toxicity detection.

[0047] In this embodiment, logits are represented as a vector l(x)∈R n (where n represents the number of tokens), each element of the vector corresponds to the logits value of a token. Before passing the logits to the deep learning model, an additional processing function f is applied to them: R n →R n .

[0048] The additional function f is defined as follows: first, the logits are Min-Max normalized to scale their values ​​to the range of [0, 1], and then the ReLU activation function is applied to introduce nonlinear characteristics. The mathematical expression is:

[0049]

[0050] Where l is the logits vector of the current batch or the entire training set (note: the normalization here is based on the logits of the entire batch or training set, not the logits of a single sample. In actual operation, a sliding window is used to approximate it), and f i (l i ) is the value of the i-th element after processing.

[0051] S2. Build a deep learning model, which is used to extract useful information from the extracted logits to detect toxic cues:

[0052] This embodiment extracts useful information from logits by using a sparse logistic regression (SLR) model. In some embodiments, in addition to the sparse logistic regression (SLR) model, other deep learning models may be used to process the logits of LLM, such as a deep neural network (DNN), a deep neural network (CNN), or a recurrent neural network (RNN). These models have powerful feature extraction and pattern recognition capabilities, and may be able to extract more useful information from logits, further improving the detection accuracy of toxicity cues.

[0053] In addition, it is possible to consider integrating multiple different models (such as SLR, DNN, SVM, etc.) to improve the detection performance of toxicity cues. Ensemble learning can reduce the bias and variance of a single model by combining the prediction results of multiple models, thereby improving the overall generalization ability.

[0054] The SLR model combines logistic regression and sparsity constraints, which can capture the key features in logits and reduce the complexity of the model. In this embodiment, the weights and biases of the SLR model are represented by w∈R n and b∈R, the output formula is:

[0055] SLR(x)=w·f(l(x))+b

[0056] Among them, f(l(x)) is the logits vector after being processed by the additional function f.

[0057] S3. Perform model training on the constructed deep learning model:

[0058] During the training process, for the Ridge regularized SLR problem, the formula is:

[0059] SLR(x)=w·f(l(x))+b

[0060] The optimization goal is to minimize the following loss function:

[0061]

[0062] Where: x, y∈X represents the samples in the training set, and each sample consists of a prompt x and a corresponding toxicity label y∈{0, 1}.

[0063] BCE() represents the Binary Cross-Entropy loss function, which is used to measure the difference between the predicted value and the true value.

[0064] Represents the L2 regularization term, which is the sum of the squares of each element of the weight vector w, and is used to prevent the model from overfitting.

[0065] λ is the regularization coefficient, which is used to control the strength of regularization.

[0066] The training dataset contains a series of prompt texts with toxicity labels, which are used to train the model so that it can learn the features that distinguish toxic prompts from benign prompts.

[0067] S4. Detect toxicity of new prompts using the trained deep learning model:

[0068] As attached Figure 2 As shown, after the SLR model training of this embodiment is completed, this embodiment uses the trained SLR model to detect toxicity of new prompts. Specifically, when a new prompt text is received, the logits vector l′ of all n tags at the starting position of the LLM response is first extracted, and then input into the trained SLR model. The SLR model outputs a probability value (i.e., Sigmoid(SLR(l′))), indicating the possibility that the prompt is toxic. Based on this probability value and a preset threshold (such as: 0.5), this embodiment can determine whether the prompt is toxic and output the detection result.

[0069] Using the logits of the first token (or word) in the LLM (Large Language Model) response, we found that tokens such as "sorry" and "can't", which usually trigger rejection, have much higher logits under toxic prompts than under benign prompts. The probability of a response starting with a rejection token (such as "sorry", "can't" or "will...", where "will..." as the beginning of a response usually also means rejection) is calculated, and defined as PBR (Probability of Beginning Reject), which refers to the probability of a response starting with a specific rejection token (such as "sorry", "can't" or "will...", which usually indicates rejection or inability to meet a request) when using a large language model (LLM) for toxic content detection. Although the probability of these rejection tokens starting in a response may not be high, there is still a significant difference in this probability between toxic and benign prompts. The PBR-based detector performs particularly well in application scenarios with low false positive rate (FPR), avoiding the performance degradation due to sampling errors at low FPRs, and thus can detect poisonous cues more accurately.

[0070] Obviously, the above embodiments are merely examples for the purpose of clear explanation, and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived therefrom are still within the scope of protection of the invention.

Claims

1. A method for detecting text toxicity using output differences of a large language model, characterized in that: The steps include: Extract logits based on the text content entered by the user; Build a deep learning model, which is used to extract useful information from the extracted logits; Conduct model training on the constructed deep learning model; Use the trained deep learning model to detect toxicity of new prompts.

2. The method for detecting text toxicity using large language model output differences according to claim 1, characterized in that: Extracting logits includes: When LLM receives the prompt text entered by the user; Locate the start position of the prompt text response; Extract n labeled logits from the starting position.

3. The method for detecting text toxicity using large language model output differences according to claim 2, characterized in that: Logits are represented as a vector l(x)∈R n , where n represents the number of tags, and each element of the vector corresponds to the logits value of a tag; Apply additional processing function f:R to logits n →R n ; Define the additional function f as follows: Perform Min-Max normalization on logits to scale the value of logits to the range of [0,1]; Applying the ReLU activation function introduces nonlinear characteristics, and the mathematical expression is: Among them, l is the logits vector of the current batch or the entire training set, and f i (l i ) is the value of the i-th element after processing.

4. The method for detecting text toxicity using large language model output differences according to claim 1, characterized in that: The deep learning model is any one of an SLR model, a deep neural network, a deep neural network, and a recurrent neural network; Or it can be an integration of any number of SLR models, deep neural networks, deep neural networks, and recurrent neural networks.

5. The method for detecting text toxicity using large language model output differences according to claim 3, characterized in that: The deep learning model is an SLR model. Build an SLR model: The weights and biases of the SLR model are represented by w∈R n and b∈R, the output formula is: SLR(x)=w·f(l(x))+b Among them, f(l(x)) is the logits vector after being processed by the additional function f.

6. The method for detecting text toxicity using large language model output differences according to claim 5, characterized in that: SLR model training: The weights and biases of the SLR model are optimized to minimize the following loss function: Where x,y∈X represents the samples in the training set, each sample consists of a prompt x and a corresponding toxicity label y∈{0,1}; BCE() represents the binary cross entropy loss function, which is used to measure the difference between the predicted value and the true value; represents the L2 regularization term, that is, the sum of the squares of each element of the weight vector w, which is used to prevent the model from overfitting; λ is the regularization coefficient, which is used to control the strength of the regularization.

7. The method for detecting text toxicity using large language model output differences according to claim 6, characterized in that: SLR model performs toxicity detection on novel cues: When a new prompt text is received, the logits vector l′ of all n tags at the starting position of the LLM response is first extracted, and then input into the trained SLR model; the SLR model outputs a probability value, namely Sigmoid(SLR(l′)), indicating the possibility that the prompt is toxic; based on this probability value and a preset threshold, it is determined whether the prompt is toxic and the detection result is output.

Images