Large model violation content detection method

Abstract

The invention relates to a large model illegal content detection method, which comprises the following steps: S1, inputting a content generation request of a user into a Transform model, respectively outputting a potential feature sequence by each layer of the Transform model, and inputting the current layer into the potential feature sequence output by the previous layer; s2, analyzing the violation probability of the potential feature sequence output by each layer by using the optimized space-time sequence discrimination model; and S3, inputting the violation probability of the potential feature sequence output by each layer into a trained secondary learner, and outputting a final violation probability. According to the method, the deep learning technology is utilized, through large model potential feature extraction and violation probability estimation and violation content detection based on ensemble learning, accurate detection of violation content is achieved, and the actual application requirement of a large model generative model is met.

Description

Technical Field

[0001] The present application relates to the technical field of illegal content detection, and in particular to a large-model illegal content detection method. Background Art

[0002] At present, with the continuous expansion of the scale of generative large language models and the diversification of application scenarios, the problems that arise are becoming increasingly prominent. For example, when generating content, large language models may generate text that is illegal in nature; such illegal content not only poses a threat to user experience and platform reputation, but may also lead to legal and regulatory risks. How to effectively detect violations of content generated by large language models has become a challenge in the current technical field.

[0003] However, existing illegal content detection methods still have many shortcomings; traditional illegal content detection methods mostly rely on keyword filtering, rule matching and simple classifiers. Although these methods can play a certain role in dealing with certain types of illegal content, they are obviously ineffective when faced with complex and diverse language expressions, implicit illegal intentions and contextual dependencies of model generation. Therefore, a more accurate and efficient illegal content detection technology is needed to cope with complex text generation tasks. Summary of the invention

[0004] Based on this, it is necessary to provide a large model illegal content detection method, which includes: S1: Input the user's content generation request into the Transformer model. Each layer of the Transformer model outputs a potential feature sequence, where the input of the current layer is the potential feature sequence output by the previous layer. S2: Use the optimized spatiotemporal sequence discrimination model to analyze the violation probability of the potential feature sequence output by each layer; S3: Input the violation probability of the potential feature sequence output by each layer into the trained secondary learner and output the final violation probability.

[0005] Beneficial effects: This method uses deep learning technology to extract potential features of a large model and estimate the probability of violation, and detects illegal content based on ensemble learning, thereby achieving accurate detection of illegal content and meeting the actual application needs of large model generative models. BRIEF DESCRIPTION OF THE DRAWINGS

[0006] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0007] Figure 1 This is a flow chart of the large model illegal content detection method in an embodiment of the present application. DETAILED DESCRIPTION

[0008] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present application, so the present application is not limited by the specific embodiments disclosed below.

[0009] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0010] like Figure 1 As shown, this embodiment provides a large model illegal content detection method, the method comprising: S1: Input the user's content generation request into the Transformer model. Each layer of the Transformer model outputs a potential feature sequence, where the input of the current layer is the potential feature sequence output by the previous layer.

[0011] In this embodiment, when performing generative large language model reasoning of the Transformer architecture, the user's content generation request is first converted into an input sequence, and the input sequence is passed through the Transformer model. N Each layer is processed as a TransformerLayer Function operation, the input of the current layer is the potential feature sequence output by the previous layer.

[0012] For each input sequence, each layer of the model outputs a latent feature sequence that captures the fine-grained variation from input to output; the dimension of the latent feature sequence is D , the length of the latent feature sequence matches the sequence length of the user's content generation request as L , to ensure that each generated label has a corresponding feature representation.

[0013] Transformer model n The layers are processed as follows: Firstn- The potential feature sequence output by layer 1 is n The input of the layer is processed by the TransformerLayer function and the output is n The potential feature sequence of the layer is expressed as: ; in, Indicates n The potential feature sequence of the layer output; express TransformerLayer function; Indicates n- Latent feature sequence of layer 1 output.

[0014] For all 1≤ n ≤ N After completing all N After the processing of the layer, the set of potential feature sequences output by the Transformer model is , where each potential feature sequence is the feature sequence output by the corresponding layer, which contains comprehensive information about the processing and transformation of the input data by the layer.

[0015] The user's content generation request will be converted into an input sequence and input into the Transformer model. The input sequence is processed through multiple layers, and each layer outputs a latent feature sequence to capture the fine-grained transformation from input to output. The model ultimately outputs multiple latent feature sequences to provide a data basis for illegal content detection, and the certainty of the length and dimension of the latent feature sequence output by each layer helps with subsequent standardized processing and analysis.

[0016] Before analyzing the probability of violation, it also includes: The second potential feature sequences output by each layer of the Transformer model in the actual scenario are collected, and each second potential feature sequence is labeled separately. The labeling value includes violation / non-violation. The spatiotemporal sequence discrimination model is optimized based on the labeled second potential feature sequences.

[0017] Specifically, the optimization of the spatiotemporal sequence discrimination model based on the labeled second potential feature sequences includes: Step 1: each of the annotated second potential feature sequences is passed through the spatiotemporal sequence discrimination model to obtain the violation probability estimate of each second potential feature sequence; Step 2: Based on the violation probability estimate of each second potential feature sequence and the corresponding annotation value, the binary cross entropy is used to calculate the loss function of the second potential feature sequence output by each layer. The loss function calculation formula is: ; in, Indicatesn The loss function of the layer; y Indicates the label value, including 0 or 1, 0 indicates no violation, 1 indicates violation; Presentation Layer n The violation probability estimate of the second latent feature sequence output by the layer; Step 3: Calculate the total loss based on the loss function of each layer. The calculation formula is: ; in, represents the total loss; N Indicates the total number of layers of the Transformer model; Step 4: Minimize the total loss, and perform backpropagation on the spatiotemporal sequence discrimination model based on the total loss to optimize the parameters of the spatiotemporal sequence discrimination model, wherein the parameters include: the weights of the multi-head attention module in the spatiotemporal sequence discrimination model, the parameters of the convolutional network and the fully connected layer.

[0018] The second potential feature sequence generated by the Transformer model is collected through a wide range of data sets, and manually labeled as violations or not; the model outputs the violation probability of each layer, and optimizes it through the binary cross entropy loss function to learn the violation discrimination ability layer by layer; the back propagation algorithm is used to optimize the parameters of the multi-head attention module, convolutional network and fully connected layer; the spatiotemporal sequence discrimination model is trained with a large amount of labeled data to effectively capture semantic features from low-level to high-level, and can effectively capture semantic features and improve the accuracy of illegal content detection.

[0019] S2: Use the optimized spatiotemporal sequence discrimination model to analyze the violation probability of the potential feature sequence output by each layer.

[0020] Specifically, the step includes: The optimized spatiotemporal sequence discriminant model was used to analyze the n The process of calculating the violation probability of the latent feature sequence output by the layer includes: For n The latent feature sequence output by the layer calculates the corresponding multi-head attention in the spatial dimension and multi-head attention in the temporal dimension respectively; In the optimized spatiotemporal sequence discrimination model, the spatial (dimension D of the latent feature sequence) and temporal (sequence length of the user’s content generation request / length L of the latent feature sequence) dimensions are processed separately through a multi-head attention mechanism; Calculate the n The multi-head attention of the spatial dimensions corresponding to the potential feature sequence of the layer output includes: For n The potential feature sequence output by the layer is transposed so that all features corresponding to each time point are processed in parallel; Based on the transposed n The potential feature sequence output by the layer calculates the multi-head attention of the spatial dimension, and the calculation formula is: ; ; ; ; in, Indicates n Multi-head attention on the spatial dimensions corresponding to the latent feature sequence output by the layer; represents the spatial query vector; represents the spatial key vector; represents a space-valued vector; , D represents the dimension of the latent feature sequence, L Indicates the sequence length of the user's content generation request; represents the softmax function; Represents the dimension of the spatial key vector; represents the spatial query weight matrix; represents the spatial key weight matrix; represents the spatial value weight matrix; ; Indicates the first n The potential feature sequence of the layer output; T represents transpose; Formula for multi-head attention in spatial dimensions, focusing on the interactions between different features.

[0021] For the time dimension, the focus is on capturing the temporal correlation of the sequence, which requires the attention mechanism to be applied along the time axis (the sequence length of the user's content generation request / the length L of the latent feature sequence); Calculate the n The multi-head attention in the time dimension corresponding to the potential feature sequence output by the layer is calculated as: ; ; ; ; in, Indicates n Multi-head attention in the time dimension corresponding to the potential feature sequence output by the layer; represents the time query vector; represents the time key vector; represents a time value vector; , D represents the dimension of the latent feature sequence, L Indicates the sequence length of the user's content generation request; represents the softmax function; Represents the dimension of the time key vector; represents the time query weight matrix; represents the time key weight matrix; represents the time value weight matrix; ; Indicates n The potential feature sequence of the layer output; Represents the multi-head attention calculation formula in the time dimension, which focuses on the dependencies between different time points in the sequence.

[0022] The feature map is obtained based on the multi-head attention of the corresponding spatial dimension and the multi-head attention of the temporal dimension; Performing global average pooling on the feature map to obtain a compressed feature representation; Furthermore, the process of obtaining the compressed feature representation includes: Based on the multi-head attention of the corresponding spatial dimension and the multi-head attention of the time dimension, the feature map is obtained, and the calculation formula is: ; in, Indicates n Feature map corresponding to the potential feature sequence output by the layer; represents a convolutional network; Indicates n Multi-head attention on the spatial dimensions corresponding to the latent feature sequence output by the layer; Indicates n Multi-head attention in the time dimension corresponding to the potential feature sequence output by the layer; Represents the feature merging operation; Perform global average pooling on the feature map to obtain a compressed feature representation. The calculation formula is: ; in, Indicates n The compressed feature representation corresponding to the potential feature sequence output by the layer; Represents a global average pooling operation.

[0023] The compressed feature representation is passed through the fully connected layer to obtain n The violation probability of the potential feature sequence output by the layer is calculated as: ; in, Indicatesn The violation probability of the latent feature sequence output by the layer; Represents the sigmoid activation function; represents the weight of the fully connected layer, represents the bias of the fully connected layer.

[0024] The optimized spatiotemporal sequence discrimination model uses a multi-head attention mechanism to process the spatial and temporal dimensions respectively, capturing the interactions and temporal dependencies between features. The outputs of the spatial and temporal dimensions are merged and downsampled by the convolutional network to optimize the feature representation, and then the features are compressed by global average pooling. Finally, the violation probability is calculated through a fully connected layer with a sigmoid activation function. This series of processing provides a probability assessment of illegal content based on deep spatiotemporal analysis for each potential feature sequence, providing effective illegal content detection capabilities and realizing illegal content detection based on deep spatiotemporal analysis.

[0025] S3: Input the violation probability of the potential feature sequence output by each layer into the trained secondary learner and output the final violation probability.

[0026] Among them, the training process of the secondary learner includes: Step 1: Optimize the labeled second potential feature sequences separately N A spatiotemporal sequence discrimination model, N Indicates the total number of layers of the Transformer model; Step 2: Use N Correspondence analysis of an optimized spatiotemporal sequence discriminant model N The violation probability of the second potential feature sequence; Step 3: Weighted combination N The violation probability of the second potential feature sequence is calculated to obtain the total violation probability; Step 4: Based on the total violation probability and its corresponding true label, calculate the cross entropy loss; back-propagate the secondary learner based on the cross entropy loss to optimize the weight and bias of the secondary learner.

[0027] In this embodiment, in order to avoid overfitting, the second potential feature sequence combinations output by the Transformer model in multiple actual scenarios are collected, and a data set is constructed based on each second potential feature sequence and its corresponding annotation value. An independent validation set is separated from the data set, and the remaining part is used as a training set. In this way, the secondary learner is trained on an independent validation set, and the base model (space-time sequence discrimination model) is trained on the training set, which improves the accuracy and robustness of the large model in detecting illegal content. The Stacking ensemble learning method utilizes the discriminative power of each space-time sequence discrimination model to significantly improve the accuracy of illegal content detection.

[0028] In this embodiment, the secondary learner is a logistic regression model, which can also be replaced by other models according to the actual scenario needs; the secondary learner is responsible for learning how to best combine the prediction results of each base model to provide overall prediction accuracy.

[0029] Furthermore, the prediction process of the trained secondary learner includes: ; ; in, represents the final violation probability; Represents the sigmoid activation function; W represents the weight of the secondary learner; T represents transpose; b represents the bias of the secondary learner; z represents a vector of violation probabilities of the latent feature sequence output by each layer; represents the violation probability of the latent feature sequence output by layer 1; represents the violation probability of the latent feature sequence output by layer 2; Indicates N The violation probability of the latent feature sequence output by the layer.

[0030] The Stacking ensemble learning method is adopted to integrate the violation probabilities of each layer of the Transformer model analyzed by multiple spatiotemporal sequence discrimination models, and these violation probabilities are optimized and fused through the secondary learner to obtain the final violation probability.

[0031] This method uses deep learning technology to extract potential features and estimate violation probabilities from large models, and detects illegal content based on ensemble learning. Compared with traditional detection methods, this method can capture semantic features from low-level to high-level, and achieve accurate detection of illegal content, meeting the actual application needs of large model generative models.

[0032] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0033] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be construed as limiting the scope of the patent application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent application shall be subject to the attached claims.

Claims

1. A large-model illegal content detection method, characterized in that: include: S1: Input the user's content generation request into the Transformer model. Each layer of the Transformer model outputs a potential feature sequence, where the input of the current layer is the potential feature sequence output by the previous layer. S2: Use the optimized spatiotemporal sequence discrimination model to analyze the violation probability of the potential feature sequence output by each layer; S3: Input the violation probability of the potential feature sequence output by each layer into the trained secondary learner and output the final violation probability.

2. The large-model illegal content detection method according to claim 1 is characterized in that: Transformer model n The layers are processed as follows: First n- The potential feature sequence output by layer 1 is n The input of the layer is processed by the TransformerLayer function and the output is n The potential feature sequence of the layer is expressed as: ； in, Indicates n The potential feature sequence of the layer output; express TransformerLayer function; Indicates n- Latent feature sequence of layer 1 output.

3. The large-model illegal content detection method according to claim 1 is characterized in that: Also includes: The second potential feature sequences output by each layer of the Transformer model in the actual scenario are collected, and each second potential feature sequence is labeled separately. The labeling value includes violation / non-violation. The spatiotemporal sequence discrimination model is optimized based on the labeled second potential feature sequences.

4. The large-model illegal content detection method according to claim 3 is characterized in that: The optimization of the spatiotemporal sequence discrimination model based on each labeled second potential feature sequence includes: Step 1: each of the annotated second potential feature sequences is passed through the spatiotemporal sequence discrimination model to obtain the violation probability estimate of each second potential feature sequence; Step 2: Based on the violation probability estimate of each second potential feature sequence and the corresponding annotation value, the binary cross entropy is used to calculate the loss function of the second potential feature sequence output by each layer. The loss function calculation formula is: ； in, Indicates n The loss function of the layer; y Indicates the label value, including 0 or 1, 0 indicates no violation, 1 indicates violation; Presentation Layer n The violation probability estimate of the second latent feature sequence output by the layer; Step 3: Calculate the total loss based on the loss function of each layer. The calculation formula is: ； in, represents the total loss; N Indicates the total number of layers of the Transformer model; Step 4: Minimize the total loss, and perform backpropagation on the spatiotemporal sequence discrimination model based on the total loss to optimize the parameters of the spatiotemporal sequence discrimination model, wherein the parameters include: the weights of the multi-head attention module in the spatiotemporal sequence discrimination model, the parameters of the convolutional network and the fully connected layer.

5. The large-model illegal content detection method according to claim 1 is characterized in that: The optimized spatiotemporal sequence discriminant model was used to analyze the n The process of calculating the violation probability of the latent feature sequence output by the layer includes: For n The potential feature sequence output by the layer calculates the corresponding multi-head attention in the spatial dimension and the multi-head attention in the temporal dimension respectively, and obtains the feature map based on the corresponding multi-head attention in the spatial dimension and the multi-head attention in the temporal dimension; Performing global average pooling on the feature map to obtain a compressed feature representation; The compressed feature representation is passed through the fully connected layer to obtain n The violation probability of the latent feature sequence output by the layer.

6. The large-model illegal content detection method according to claim 5 is characterized in that: Calculate the n The multi-head attention of the spatial dimensions corresponding to the potential feature sequence of the layer output includes: For n Transpose the potential feature sequence output by the layer; Based on the transposed n The potential feature sequence output by the layer calculates the multi-head attention of the spatial dimension, and the calculation formula is: ； ； ； ； in, Indicates n Multi-head attention on the spatial dimensions corresponding to the latent feature sequence output by the layer; represents the spatial query vector; represents the spatial key vector; represents a space-valued vector; , D represents the dimension of the latent feature sequence, L Indicates the sequence length of the user's content generation request; represents the softmax function; Represents the dimension of the spatial key vector; represents the spatial query weight matrix; represents the spatial key weight matrix; represents the spatial value weight matrix; ; Indicates the first n The potential feature sequence of the layer output; T Indicates transpose.

7. The large-model illegal content detection method according to claim 6 is characterized in that: Calculate the n The multi-head attention in the time dimension corresponding to the potential feature sequence output by the layer is calculated as: ； ； ； ； in, Indicates n Multi-head attention in the time dimension corresponding to the potential feature sequence output by the layer; represents the time query vector; represents the time key vector; represents a time value vector; , D represents the dimension of the latent feature sequence, L Indicates the sequence length of the user's content generation request; represents the softmax function; Represents the dimension of the time key vector; represents the time query weight matrix; represents the time key weight matrix; represents the time value weight matrix; ; Indicates n The latent feature sequence output by the layer.

8. The large-model illegal content detection method according to claim 7 is characterized in that: The process of obtaining the compressed feature representation includes: Based on the multi-head attention of the corresponding spatial dimension and the multi-head attention of the time dimension, the feature map is obtained, and the calculation formula is: ； in, Indicates n Feature map corresponding to the potential feature sequence output by the layer; represents a convolutional network; Indicates n Multi-head attention on the spatial dimensions corresponding to the latent feature sequence output by the layer; Indicates n Multi-head attention in the time dimension corresponding to the potential feature sequence output by the layer; Represents the feature merging operation; Perform global average pooling on the feature map to obtain a compressed feature representation. The calculation formula is: ； in, Indicates n The compressed feature representation corresponding to the potential feature sequence output by the layer; Represents a global average pooling operation.

9. The large-model illegal content detection method according to claim 4 is characterized in that: The training process of the secondary learner includes: Step 1: Optimize the labeled second potential feature sequences separately N A spatiotemporal sequence discrimination model, N Indicates the total number of layers of the Transformer model; Step 2: Use N Correspondence analysis of an optimized spatiotemporal sequence discriminant model N The violation probability of the second potential feature sequence; Step 3: Weighted combination N The violation probability of the second potential feature sequence is calculated to obtain the total violation probability; Step 4: Based on the total violation probability and its corresponding true label, calculate the cross entropy loss; back-propagate the secondary learner based on the cross entropy loss to optimize the weight and bias of the secondary learner.

10. The large-model illegal content detection method according to claim 1, characterized in that: The prediction process of the trained secondary learner includes: ； ； in, represents the final violation probability; Represents the sigmoid activation function; W represents the weight of the secondary learner; T represents transpose; b represents the bias of the secondary learner; z represents a vector of violation probabilities of the latent feature sequence output by each layer; represents the violation probability of the latent feature sequence output by layer 1; represents the violation probability of the latent feature sequence output by layer 2; Indicates N The violation probability of the latent feature sequence output by the layer.

Images