A system and method for identifying and filtering inappropriate content based on a large language model

Abstract

The present application discloses an inappropriate content identification and filtering system based on a large language model, including a text collection and preprocessing module, a feature extraction and analysis module, an inappropriate content identification module, a filtering decision module, and a user interaction module. The present application also discloses an inappropriate content identification and filtering method based on a large language model, the steps of which are as follows: automatically collect text and perform preprocessing, analyze and extract features from the preprocessed text, and obtain a spliced ​​and integrated comprehensive feature vector; based on the comprehensive feature vector, calculate the probability of inappropriate content and the sentiment intensity score, calculate the inappropriate content identification weight score, compare it with the optimal threshold calculated by the F1 score, and mark the inappropriate content text; then introduce dynamically adjusted warning thresholds and deletion thresholds, formulate a hierarchical processing strategy, and take corresponding measures for users and inappropriate content texts. The present application can greatly improve the accuracy of semantic recognition.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a system and method for identifying and filtering inappropriate content based on a large language model. Background Art

[0002] With the rapid development of the Internet and the deepening of informatization, the widespread dissemination of inappropriate content in social media, online forums and various comment areas can easily cause negative social impacts. Effectively identifying and filtering inappropriate content has become an urgent problem to be solved. The inappropriate content identification and filtering system and method based on the large language model have demonstrated unique technical advantages. It is specifically aimed at the identification and filtering of inappropriate content. By comprehensively utilizing the deep learning and natural language processing capabilities of the large language model, it can not only identify obvious inappropriate terms, but also accurately capture implicit negative intentions and subtle language differences. Compared with traditional technologies, it improves the accuracy and efficiency of recognition and can adapt to the changing network environment and the ever-changing content strategy requirements.

[0003] Patent application number: CN202311146313.3, publication date: 2023.10.13, a text sentiment analysis method and system based on natural language processing, collects sentiment text sentences, and segment each text sentence to obtain each phrase; obtains the word vector of each phrase; for each text sentence, takes the sequence composed of the word vectors of each phrase as the word vector sequence; obtains the word meaning opposition index of the sentence according to the relationship between each word vector in the word vector sequence; presets a degree adverb table; obtains the abnormality rating of the sentence according to the degree adverb position score in the degree adverb table and the position of each degree adverb in the sentence; obtains the potential irony index of the sentence according to the word meaning opposition index and the abnormality rating of the sentence; inserts the potential irony index of the sentence into the sentiment analysis model for sentiment recognition; realizes text sentiment analysis, improves the accuracy of sentiment analysis in text sentences, and has high text sentiment detection accuracy.

[0004] The above technology has at least the following technical problems: word vector representation has limitations. Using word vectors to represent text sentences cannot capture all the semantic information of the text, especially for some professional terms and new words, the word vector representation is not accurate enough; the setting of the self-similarity distance threshold lacks universality and can easily lead to inaccurate judgments; when calculating the word meaning opposition index, only the position distance of the phrase and the elements of the semantic opposition matrix are considered, which cannot fully reflect the semantic relationship of the phrase. Summary of the invention

[0005] The present invention provides an inappropriate content identification and filtering system and method based on a large language model to solve the technical problem that the existing word vector representation has limitations and cannot capture all the semantic information of the text, especially for some professional terms and new words, the word vector representation is not accurate enough, which easily leads to inaccurate judgment, and when calculating the word meaning opposition index, it cannot fully reflect the semantic relationship of the phrase.

[0006] The present invention discloses an inappropriate content identification and filtering system based on a large language model in an embodiment, including:

[0007] The text collection and preprocessing module is used to automatically collect texts from social media platforms and online forums, and preprocess the collected texts to obtain preprocessed texts;

[0008] The feature extraction and analysis module is used to extract features from the preprocessed text to obtain word frequency-inverse document frequency feature vectors, word embedding vectors, and syntactic feature vectors, and integrate them into a comprehensive feature vector through vector splicing; the preprocessed text is analyzed through natural language processing to obtain the total number of sentiment words, the basic weight of sentiment words, sentiment polarity, and the total number of modifiers of sentiment words;

[0009] The inappropriate content identification module is used to construct an inappropriate content probability model based on the comprehensive feature vector to calculate the probability of inappropriate content; based on the total number of sentiment words, the basic weight of sentiment words, the basic sentiment polarity of sentiment words, and the total number of modifiers of sentiment words output by the feature extraction and analysis module, a sentiment analysis model is constructed to calculate the sentiment intensity score, the probability of inappropriate content and the sentiment intensity score are comprehensively calculated to obtain the inappropriate content identification weight score, the inappropriate content identification weight score is compared with the optimal threshold obtained by calculating the F1 score, and the inappropriate content text is marked;

[0010] A filtering decision module is used to set a warning threshold and a deletion threshold based on dynamic adjustment, formulate a hierarchical processing strategy based on the inappropriate content identification weight score from the inappropriate content identification module, take processing measures according to the hierarchical processing strategy, and output the processing measures;

[0011] The user interaction module is used to receive the processing measures and automatically send a notification to the user, explaining the inappropriate content in the text and providing modification suggestions, or notifying the user that the text has been deleted and explaining the reasons.

[0012] The present application embodiment discloses a method for identifying and filtering inappropriate content based on a large language model, comprising the following steps:

[0013] S1. Automatically collect texts, preprocess the collected texts to obtain preprocessed texts, analyze and extract features from the preprocessed texts using natural language processing technology, and perform vector concatenation on the extracted word frequency-inverse document frequency feature vectors, word embedding vectors, and syntactic feature vectors to integrate them into a comprehensive feature vector;

[0014] S2. Based on the comprehensive feature vector, an inappropriate content probability model is constructed to calculate the probability of inappropriate content, and a sentiment analysis model is constructed to calculate the sentiment intensity score. The probability of inappropriate content and the sentiment intensity score are calculated comprehensively to obtain an inappropriate content identification weight score. The inappropriate content identification weight score is compared with the optimal threshold calculated by the F1 score to mark the inappropriate content text;

[0015] S3. Introduce dynamically adjusted warning thresholds and deletion thresholds, combine them with the weight scores for identifying inappropriate content, formulate a graded processing strategy, and take warning, isolation, and deletion measures against users and inappropriate content texts based on the graded processing strategy.

[0016] As a specific implementation, in step S2, the inappropriate content probability model includes a basic classification component, a semantic enhancement component, and an anomaly detection component;

[0017] Based on the comprehensive feature vector, in the basic classification component, the categories of inappropriate content are defined as pornographic, violent, and abusive, and the basic classification probability value is obtained through logistic regression calculation;

[0018] Based on the comprehensive feature vector, in the semantic enhancement component, after processing using a multi-layer perceptron, the deep semantic score of the text is output;

[0019] Based on the comprehensive feature vector, in the anomaly detection component, the improved lonely forest algorithm is used to calculate the anomaly score;

[0020] The outputs of the basic classification component, semantic enhancement component, and anomaly detection component are integrated into the inappropriate content probability model, the weight parameters are initialized to equal values, and the probability of each type of inappropriate content is calculated.

[0021] As a specific implementation, in step S2, "constructing a sentiment analysis model to calculate a sentiment intensity score" specifically includes:

[0022] The sentiment analysis model is based on the total number of sentiment words, the total number of modifiers of sentiment words, the basic weights of sentiment words, and sentiment polarity obtained through natural language processing technology. It first calculates the cumulative result of the modifier adjustment factor of each sentiment word, then accumulates the weighted sentiment values ​​of all sentiment words, and finally divides the accumulated sentiment value by the total number of sentiment words to obtain the sentiment intensity score.

[0023] As a specific implementation, in step S2, "comprehensively calculating the probability of inappropriate content and the sentiment intensity score to obtain an inappropriate content identification weight score" specifically includes:

[0024] For each inappropriate content category, the adjusted sentiment intensity contribution is calculated, and then a nonlinear transformation is performed using the hyperbolic tangent function. The contributions of all categories are then accumulated, and exponential decay is applied to adjust the influence of inappropriate content to calculate the final inappropriate content identification weight score.

[0025] As a specific implementation, the specific formula of the inappropriate content identification weight score is:

[0026]

[0027] Where S is the weight score of inappropriate content identification, ranging from [0, 1]; ω is the normalization coefficient; g is the attenuation factor; is the summation symbol; p i is the probability that the predicted text belongs to the i-th category of inappropriate content; v i is the sentiment intensity score of the i-th category of content; l is an adjustment factor; h is an offset; tanh represents the hyperbolic tangent function; exp represents the exponential function.

[0028] As a specific implementation, the sentiment intensity score v i The specific calculation formula is as follows:

[0029]

[0030] Among them, v i represents the sentiment intensity score of the i-th category of content; M represents the total number of sentiment words identified in the text; w j The basic weight of the jth sentiment word; s j The sentiment polarity of the jth sentiment word, positive is +1, negative is -1; r jk K represents the regulatory factor affecting the kth modifier of the jth sentiment word; j Represents the total number of modifiers acting on the j-th sentiment word; Indicates the accumulation of weighted sentiment values ​​of all sentiment words; It represents the cumulative result of calculating the modifier adjustment factor of each sentiment word, that is, calculating the final adjusted sentiment value of the sentiment word.

[0031] As a specific implementation, in step S2, "comparing the inappropriate content identification weight score with the optimal threshold obtained by calculating the F1 score, and marking the inappropriate content text" specifically includes:

[0032] According to the range of the inappropriate content identification weight score being [0, 1], the candidate thresholds are set at intervals of 0.1, and the candidate threshold that maximizes the F1 score is selected as the optimal threshold. The inappropriate content is marked by comparing the inappropriate content identification weight score with the optimal threshold; if the inappropriate content identification weight score is higher than or equal to the optimal threshold, the marked text is inappropriate content text.

[0033] As a specific implementation, in step S3, “introducing dynamically adjusted warning threshold and deletion threshold”, the warning threshold and deletion threshold are dynamically adjusted, and the specific formula is:

[0034] θ1(t)=μ1+δ1·sin(ω1t+φ1)+ξ1·log(1+η·E(t))

[0035] θ2(t)=μ2+δ2·cos(ω2t+φ2)+ξ2·log(1+η·E(t))

[0036] Among them, θ1(t) is the warning threshold that varies with time, θ2(t) is the deletion threshold that varies with time; μ1 and μ2 are baseline thresholds; δ1 and δ2 are parameters that control the amplitude of the warning threshold and the deletion threshold, respectively; ω1 and ω2 are the frequency of change; φ1 and φ2 are the phases; ξ1 and ξ2 are the coefficients that enhance the influence of environmental factors; η is the scaling factor of environmental influence; E(t) is the external environmental variable; log(1+η·E(t)) indicates that the environmental factor is processed using a logarithmic function.

[0037] As a specific implementation, in step S3, "introducing dynamically adjusted warning thresholds and deletion thresholds, combining inappropriate content identification weight scores, and formulating a graded processing strategy" is as follows:

[0038]

[0039] Among them, C(S) is the treatment measure determined by the weight score of inappropriate content identification, including warning, isolation, and deletion; S is the weight score of inappropriate content; θ1 and θ2 are the warning threshold and deletion threshold respectively;

[0040] When it is warn, a warning is issued to the user, indicating the specific part of the inappropriate content in the inappropriate content text and suggesting modification or deletion;

[0041] When it is isolated, the inappropriate content text will be automatically moved from the public visible area to the isolation area, and the user will be notified to modify it before publishing;

[0042] When the inappropriate content is deleted, the inappropriate content text will be automatically deleted from the platform, and the user will be notified that the inappropriate content text has been deleted and the reason for the deletion will be provided.

[0043] The technical solution of the embodiment of the present invention has at least the following beneficial effects:

[0044] 1. By vectorizing the extracted word frequency-inverse document frequency features, word embedding feature extraction, and syntactic dependencies, and integrating them into a comprehensive feature vector through vector splicing, the text content can be captured and analyzed more accurately, and the accuracy of subsequent content analysis and classification can be improved, especially in processing complex text data, such as obscure and inappropriate content, providing more comprehensive technical support.

[0045] 2. By integrating the three components of basic classification, semantic enhancement and anomaly detection, each takes its own strengths to form a powerful inappropriate content probability model, which can effectively classify and identify inappropriate content, especially in semantic enhancement and abnormal pattern recognition, improving the recognition accuracy and sensitivity of inappropriate content, effectively reducing misjudgments and missed judgments, and enhancing the reliability of inappropriate content identification.

[0046] 3. Combining the sentiment dictionary and dependency syntax analysis in natural language processing technology, we can conduct in-depth analysis of the sentiment expression in the text. This can carefully evaluate the sentiment intensity in the text, and even the subtle differences in sentiment expression. It can more accurately judge the sentiment tendency of the text, especially when dealing with content involving intense emotions. It can effectively distinguish between positive and negative emotions, which plays an important role in improving user experience and maintaining community atmosphere.

[0047] 4. Combining the probability and sentiment intensity of inappropriate content, the influence of the two is balanced through a mathematical model, and the exponential and logarithmic functions are used to reconcile them to obtain the inappropriate content identification weight score, which accurately reflects the probability and sentiment intensity of inappropriate content in the text.

[0048] 5. The warning threshold and deletion threshold can be adaptively adjusted according to the real-time environment, such as changes in community feedback or updates to legal policies, providing a hierarchical processing strategy that can flexibly respond to different situations. While protecting user rights, it can also comply with the latest social and legal requirements, reducing the lag and inadaptability of decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A structural diagram of an inappropriate content identification and filtering system based on a large language model according to an embodiment of the present invention;

[0050] Figure 2 This is a flow chart of a method for identifying and filtering inappropriate content based on a large language model according to another embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0052] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0053] The specific scheme of the inappropriate content identification and filtering system and method based on a large language model provided by the embodiments of the present invention is described in detail below with reference to the accompanying drawings.

[0054] See attached Figure 1 The inappropriate content identification and filtering system based on a large language model disclosed in an embodiment of the present invention includes a text collection and preprocessing module, a feature extraction and analysis module, an inappropriate content identification module, a filtering decision module, and a user interaction module.

[0055] The text collection and preprocessing module is used to automatically collect text from social media platforms and online forums through web crawler technology, preprocess the collected text, obtain the preprocessed text and input it into the feature extraction and analysis module.

[0056] The feature extraction and analysis module is used to extract features from the preprocessed text to obtain word frequency-inverse document frequency feature vectors, word embedding vectors, and syntactic feature vectors. Through vector splicing, they are integrated into a comprehensive feature vector. The preprocessed text is analyzed through natural language processing to obtain the total number of sentiment words, the basic weights of sentiment words, the basic sentiment polarity of sentiment words, and the total number of modifiers of sentiment words.

[0057] The inappropriate content identification module is used to construct an inappropriate content probability model to calculate the probability of inappropriate content based on the comprehensive feature vector output by the feature extraction and analysis module. Based on the total number of sentiment words, the basic weight of sentiment words, the basic sentiment polarity of sentiment words, and the total number of modifiers of sentiment words output by the feature extraction and analysis module, a sentiment analysis model is constructed to calculate the sentiment intensity score. The probability of inappropriate content and the sentiment intensity score are comprehensively calculated to obtain the inappropriate content identification weight score. The inappropriate content identification weight score is compared with the optimal threshold obtained by calculating the F1 score to mark the inappropriate content text.

[0058] The filtering decision module is used to set the warning threshold and deletion threshold based on dynamic adjustment, and formulate a hierarchical processing strategy in combination with the inappropriate content identification weight score from the inappropriate content identification module. The processing measures are taken according to the hierarchical processing strategy, and the processing measures include warning, isolation, and deletion, and the processing measures are input into the user interaction module.

[0059] The user interaction module is used to receive the processing measures from the filtering decision module, automatically notify the user, explain the inappropriate content in the text and provide modification suggestions, or notify the user that the text has been deleted and explain the reasons.

[0060] Refer to the attached Figure 2 Another embodiment of the present invention provides a method for identifying and filtering inappropriate content based on a large language model, the method comprising the following steps:

[0061] S1. Automatically collect texts, preprocess the collected texts to obtain preprocessed texts, analyze and extract features from the preprocessed texts through natural language processing technology, perform vector concatenation on the extracted word frequency-inverse document frequency feature vectors, word embedding vectors, and syntactic feature vectors, and integrate them into a comprehensive feature vector.

[0062] Texts are automatically collected from social media platforms and online forums through web crawler technology, including public social media posts and forum comments. The collected texts are preprocessed, including removing useless format tags, non-text content and standardizing text formats to obtain preprocessed texts. The web crawler technology and preprocessing methods are technical means well known to those skilled in the art and will not be elaborated here.

[0063] Through natural language processing technology, the text preprocessing library (spaCy library) is used to segment the preprocessed text, and the continuous text string is decomposed into words that are easy to analyze. SpaCy is used for part-of-speech tagging to automatically identify the part of speech of each word, such as noun, verb, and adjective. The part-of-speech tagged text is combined with a sentiment dictionary, such as AFINN, to perform sentiment scoring on the words, and the total number of sentiment words, the total number of modifiers of sentiment words, the basic weight of sentiment words, and the sentiment polarity are obtained. The annotated text is subjected to word frequency-inverse document frequency feature extraction, word embedding feature extraction, and syntactic dependency extraction to obtain word frequency-inverse document frequency feature vectors, word embedding vectors, and syntactic feature vectors.

[0064] The obtained word frequency-inverse document frequency feature vector, word embedding vector, and syntactic feature vector are integrated into a comprehensive feature vector X through vector splicing. This setting can more accurately capture and analyze text content, improve the accuracy of content analysis and classification, and provide more comprehensive technical support, especially in processing complex text data, such as obscure and inappropriate content.

[0065] S2. Based on the comprehensive feature vector, an inappropriate content probability model is constructed to calculate the probability of inappropriate content, and a sentiment analysis model is constructed to calculate the sentiment intensity score. The probability of inappropriate content and the sentiment intensity score are calculated comprehensively to obtain an inappropriate content identification weight score. The inappropriate content identification weight score is compared with the optimal threshold calculated by the F1 score to mark the inappropriate content text.

[0066] This embodiment calculates the inappropriate content identification weight score by comprehensively considering the probability and sentiment intensity of inappropriate content in the text, and can accurately identify whether the text contains inappropriate content. The specific steps are as follows:

[0067] The first step is to calculate the probability of inappropriate content;

[0068] The probability of inappropriate content is calculated by constructing an inappropriate content probability model. Specifically, the inappropriate content probability model includes a basic classification component, a semantic enhancement component, and anomaly detection components. It improves the detection accuracy and sensitivity of inappropriate content in the text, can capture complex semantics and abnormal patterns, effectively reduces misjudgments and missed judgments, and improves the reliability of inappropriate content identification.

[0069] Based on the comprehensive feature vector, in the basic classification component, the categories of inappropriate content are defined as pornographic, violent, and abusive. The basic classification probability value is obtained through logistic regression calculation. The specific formula is:

[0070]

[0071] Among them, f b (X) is the output function of the basic classification component, that is, the basic classification probability value, which is used to represent the probability of inappropriate content classification given the comprehensive feature vector X; X is a comprehensive feature vector used to reflect the key information and context of the text; e is the base of the natural logarithm, which is used to define the exponential function and is a standard mathematical element used in logistic regression; w b The weight vector of logistic regression, where each element corresponds to a feature in the feature vector, is used to determine the importance of each feature in the classification decision; Represents the weight vector w b Transpose from column vector to row vector and perform inner product operation with comprehensive eigenvector X; It is the exponential part of the logistic function, which means taking the negative of the dot product of the weight vector and the feature vector, and then calculating the exponent to help the basic classification output a value in the range of (0, 1); is the core formula of logistic regression, which maps real numbers to the interval (0, 1) so that the output can be interpreted as probability. This ensures that the output is between 0 and 1.

[0072] Based on the comprehensive feature vector, in the semantic enhancement component, after processing using a multi-layer perceptron, the deep semantic score of the text is output. The specific formula is:

[0073] f s (X)=σ3(W3σ2(W2σ1(W1X+b1)+b2)+b3)

[0074] Among them, f s (X) is the output function of the semantic enhancement component, i.e., the deep semantic score of the text; X is a comprehensive feature vector, which contains the basic semantic information and structural features of the text; W1, W2, and W3 are weight matrices of the neural network, W1 is the weight matrix of the first layer, which is used to initially transform the input feature vector, W2 is the weight matrix of the second layer, which is used to further process the output of the first layer, and W3 is the newly added weight matrix of the third layer, which is used for the final high-level feature extraction and enhances the learning and expression capabilities of the semantic enhancement component; b1, b2, and b3 are bias vectors, b1 is the bias of the first layer, which adds nonlinearity to prevent the semantic enhancement component from relying solely on the linear relationship of the input, b2 is the bias of the second layer, and b3 is the newly added bias of the third layer, which is used to adjust the activation threshold of the final output layer; σ1 and σ2 are ReLU functions, which help solve the gradient vanishing problem and accelerate the convergence of the neural network; σ3 is the Sigmoid function, which is used in the output layer to limit the output between 0 and 1.

[0075] Based on the comprehensive feature vector, in the anomaly detection component, the improved lonely forest algorithm is used to calculate the anomaly score. The improved lonely forest algorithm is based on the existing lonely forest algorithm and adds two new parameters λ and μ to adjust the sensitivity and bias of the anomaly score. The specific formula is:

[0076] f a (X) = 1-e -λs(X)-μ

[0077] Among them, f a (X) is the output of the anomaly detection component, that is, the anomaly score; X is the comprehensive feature vector; s(X) is the standardized isolation score; λ is an adjustment factor used to adjust the influence of the isolation score on the final anomaly score. By changing the value of λ, the sensitivity of the anomaly detection component to anomalies can be enhanced or weakened to adapt to different text distributions and anomaly detection requirements; μ is a bias term used to adjust the baseline of the anomaly score.

[0078] The outputs of the basic classification component, semantic enhancement component, and anomaly detection component are integrated into the inappropriate content probability model, and the weight parameters are initialized to equal values, such as Calculate the probability p of each type of inappropriate content i , the specific formula is as follows:

[0079] pi =σ3(α·f b (X)+β·f s (X)+γ·f a (X)

[0080] Among them, p i represents the probability of the i-th category of inappropriate content, which is a value in the interval [0, 1], where 0 represents no inappropriate content at all and 1 represents completely inappropriate content; X represents a comprehensive feature vector, which reflects the content of the text from different perspectives and provides comprehensive information about the importance of vocabulary, semantic depth, and text structure; f b (X) represents the output of the basic classification component, that is, the basic classification probability value; f s (X) represents the output of the semantic enhancement component, i.e., the deep semantic score of the text; f a (X) represents the output of the anomaly detection component, i.e., the anomaly score; α, β, and γ are weight parameters used to adjust the output of each component in p i The contribution ratio in , an adaptive learning rate adjustment technology, such as Adam optimizer, is used to dynamically adjust the weight parameters. The specific adaptive learning rate adjustment technology is a technical means well known to those skilled in the art and will not be described in detail here; σ3 is a Sigmoid function, which is used to map the result of the linear combination to a probability range of [0, 1].

[0081] The second step is to calculate the emotion intensity score;

[0082] In order to calculate the sentiment intensity score v of the i-th category content i , construct a sentiment analysis model. The sentiment analysis model is calculated based on the total number of sentiment words, the total number of modifiers of sentiment words, the basic weight of sentiment words, and sentiment polarity obtained by natural language processing technology, fully considering the complexity and multi-dimensionality of sentiment expression.

[0083] Based on the total number of sentiment words, the total number of modifiers of sentiment words, the basic weight of sentiment words, and sentiment polarity obtained through natural language processing technology, we first calculate the cumulative result of the modifier adjustment factor of each sentiment word, then add up the weighted sentiment values ​​of all sentiment words, and finally divide the accumulated sentiment value by the total number of sentiment words to get the sentiment intensity score.

[0084] The specific formula for calculating the sentiment intensity score using the sentiment analysis model is:

[0085]

[0086] Among them, v i represents the sentiment intensity score of the i-th category of content; M represents the total number of sentiment words identified in the text, which is used to normalize the sentiment intensity score to ensure that the sentiment intensity will not be biased due to the different number of sentiment words; wj The base weight of the jth sentiment word, based on the preset strength in the sentiment dictionary; s j The sentiment polarity of the jth sentiment word, +1 for positive and -1 for negative, defines the type of emotion conveyed by the sentiment word, whether it is positive or negative; r jk K represents the adjustment factor of the kth modifier word that affects the jth sentiment word, which is used to adjust the sentiment intensity of the sentiment word, which can be to enhance or weaken the influence of the sentiment word; j Represents the total number of modifiers acting on the j-th sentiment word; Indicates the accumulation of weighted sentiment values ​​of all sentiment words; It represents the cumulative result of calculating the modifier adjustment factor of each sentiment word, that is, calculating the final adjusted sentiment value of the sentiment word.

[0087] The third step is to calculate the weight score of inappropriate content identification;

[0088] In order to integrate the results of the probability and sentiment intensity of inappropriate content, an inappropriate content identification weight score S of a text is calculated to determine whether the text contains inappropriate content, and the sentiment intensity of the content is taken into account, thereby improving the accuracy and sensitivity of inappropriate content identification.

[0089] For each inappropriate content category i, through v i 1+h is used to calculate the adjusted sentiment intensity contribution, and then the hyperbolic tangent function tanh is used for nonlinear transformation. Then the contribution of all categories is accumulated, and exponential decay is applied to adjust the influence of inappropriate content to calculate the final inappropriate content identification weight score.

[0090] The specific formula for the inappropriate content identification weight score is:

[0091]

[0092] Among them, S is the weight score of inappropriate content identification, which is used to evaluate whether the text contains inappropriate content. The higher the score, the greater the possibility that the text data contains inappropriate content, and the range is [0, 1]. ω is the normalization coefficient, which is used to adjust the scale of the entire score so that the value of S is within [0, 1], which is convenient for comparison and threshold setting. g is the attenuation factor, which is used to adjust the impact of the cumulative term. The influence of the inappropriate content category in the total score can be controlled by changing the value of g. is a summation symbol, indicating the accumulation of contributions to all inappropriate content categories, and each category contributes to the inappropriate content identification weight score S; p i is the probability that the predicted text belongs to the i-th category of inappropriate content; v i is the sentiment intensity score of the i-th category of content; l is a regulation factor used to adjust the sentiment intensity score v iThe influence degree of the whole formula, by changing the value of l, the weight of emotional intensity in the weight score of inappropriate content identification can be controlled, thus reflecting the importance of emotional factors in inappropriate content identification; h is the offset, which is used to adjust the center position of the tanh function. By changing the value of h, the adjustment effect of emotional intensity can be optimized; tanh represents the hyperbolic tangent function, which is used to convert v i The result of l+h is converted into a value between -1 and 1, which helps to smooth the nonlinear growth of sentiment intensity and limit the impact of extreme values; exp represents the exponential function, which is used to convert the summation term into a decaying exponential form, which helps to further smooth and regulate the contribution of different inappropriate content categories to the total score.

[0093] According to the range of inappropriate content identification weight score is [0, 1], the candidate threshold is set at an interval of 0.1, that is, the candidate threshold is {0, 0.1, 0.2, ..., 0.9, 1}, and the F1 score of the candidate threshold is calculated, where the F1 score is the harmonic mean of precision and recall. The candidate threshold that maximizes the F1 score is selected as the optimal threshold S F1 , which is used to mark inappropriate content. The specific calculation methods of F1 score, precision and recall rate are well known to those skilled in the art and will not be elaborated here.

[0094] Rules for flagging inappropriate content:

[0095] If S ≥ S F1 , then the text is marked as inappropriate content text;

[0096] If S F1 , the text is not marked as inappropriate.

[0097] S3. Introduce dynamically adjusted warning thresholds and deletion thresholds, combine them with the weight scores for identifying inappropriate content, formulate a graded processing strategy, and take warning, isolation, and deletion measures against users and inappropriate content texts based on the graded processing strategy.

[0098] This embodiment formulates a hierarchical processing strategy based on the weight score of inappropriate content identification. In order to ensure that the social platform can flexibly respond to the changing environment and comply with legal provisions when handling inappropriate content, the warning threshold and the deletion threshold are dynamically adjusted. The specific formula is:

[0099] θ1(t)=μ1+δ1·sin(ω1t+φ1)+ξ1·log(1+η·E(t))

[0100] θ2(t)=μ2+δ2·cos(ω2t+φ2)+ξ2·log(1+η·E(t))

[0101] ​Among them, θ1(t) and θ2(t) are the warning threshold and deletion threshold that change over time, respectively; μ1 and μ2 are baseline thresholds, which represent the default processing level when there is no additional environmental factor influence; δ1 and δ2 represent the parameters that control the amplitude of the warning threshold and deletion threshold, respectively, and define the maximum fluctuation range of the warning threshold and deletion threshold; ω1 and ω2 represent the change frequency, which determine the periodic speed of the adjustment of the warning threshold and deletion threshold; φ1 and φ2 represent the phase, which control the starting point of the periodic change, and ensure that the adjustment of the warning threshold and deletion threshold starts at the appropriate time to synchronize with external events or internal policy updates; ξ1 and ξ2 represent the coefficients that enhance the influence of environmental factors, adjust the weight of external environmental variables, and ensure that external conditions are properly considered; η represents the scaling factor of environmental influence, which is used to adjust the influence of the external environmental variable E(t); E(t) represents the external environmental variable, which can be the intensity of community feedback or the urgency of legal changes; log(1+η·E(t) represents the use of logarithmic functions to process environmental factors, avoid the influence of extreme values, and ensure that external environmental variables gradually affect the warning threshold and deletion threshold in a linear manner.

[0102] In this embodiment, the dynamic adjustment mechanism can update the threshold in real time according to the current environment of the social platform and changes in legal policies, thereby ensuring the adaptability and legality of the processing strategy. Among them, the use of logarithmic functions to process environmental factors can smooth the environmental impact and avoid sudden changes in thresholds caused by extreme external environmental variables, so as to maintain the continuity and rationality of threshold adjustment.

[0103] Dynamically adjusted warning thresholds and deletion thresholds are introduced, combined with the weight scores for identifying inappropriate content, to formulate a graded processing strategy as follows:

[0104]

[0105] Among them, C(S) is the treatment measure determined according to the weight score of inappropriate content identification, including warning, isolation, and deletion; S is the weight score of inappropriate content; θ1 and θ2 are the warning threshold and deletion threshold, respectively.

[0106] The following are the specific treatment methods for each treatment measure:

[0107] Warning (warn): Issue a warning to the user, indicating the specific part of the inappropriate content in the inappropriate content text, and recommending modification or deletion to comply with platform regulations and legal requirements;

[0108] Isolation: Automatically move inappropriate content from the public visible area to the isolation area, and notify the user to modify it before publishing;

[0109] Delete: Automatically delete the inappropriate content from the platform, notify the user that the inappropriate content has been deleted, and provide the reason for the deletion.

[0110] In this embodiment, dynamically adjusted warning thresholds and deletion thresholds are introduced and adjusted according to time changes and external environmental factors, thereby improving the flexibility of the processing strategy. Warning, isolation, and deletion measures are completed through automated processes, thereby improving the efficiency and consistency of handling inappropriate content.

[0111] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0112] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0113] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A method for identifying and filtering inappropriate content based on a large language model, characterized in that: The following steps are involved: S1. Automatically collect texts, preprocess the collected texts to obtain preprocessed texts, analyze and extract features from the preprocessed texts using natural language processing technology, and perform vector concatenation on the extracted word frequency-inverse document frequency feature vectors, word embedding vectors, and syntactic feature vectors to integrate them into a comprehensive feature vector; S2. constructing an inappropriate content probability model based on the comprehensive feature vector; the inappropriate content probability model includes a basic classification component, a semantic enhancement component, and an anomaly detection component; Based on the comprehensive feature vector, in the basic classification component, the categories of inappropriate content are defined as pornographic, violent, and abusive, and the basic classification probability value f is obtained through logistic regression calculation. b (X), the specific formula is: Among them, X is the comprehensive feature vector; w b is the weight vector for logistic regression; Represents the weight vector w b Transpose from column vector to row vector; is the exponential part of the logistic function; Based on the comprehensive feature vector, in the semantic enhancement component, after processing using a multi-layer perceptron, the text deep semantic score f is output. s (X), the specific formula is: f s (X)=σ3(W3σ2(W2σ1(W1X+b1)+b2)+b3) Among them, W1, W2, W3 are the weight matrices of the first, second, and third layers of the neural network respectively; b1, b2, b3 are the bias vectors of the first, second, and third layers respectively; σ1, σ2 are ReLU functions; σ3 is the Sigmoid function; Based on the comprehensive feature vector, in the anomaly detection component, the improved lonely forest algorithm is used to calculate the anomaly score f a (X), the specific formula is: f a (X)=1-e -λs(X)-μ Among them, λ is the adjustment factor for adjusting the impact of the isolated score on the final anomaly score; s(X) is the standardized isolated score; μ is the bias term; The outputs of the basic classification component, semantic enhancement component, and anomaly detection component are integrated into the inappropriate content probability model, the weight parameters are initialized to equal values, and the probability p of each type of inappropriate content is calculated. i , the specific formula is as follows: p i =σ3(α·f b (X)+β·f s (X)+γ·f a (X)) Among them, α, β, γ are weight parameters; σ3 is the Sigmoid function; A sentiment analysis model is constructed. Based on the total number of sentiment words, the total number of modifiers of sentiment words, the basic weight of sentiment words, and sentiment polarity obtained by natural language processing technology, the sentiment intensity score is calculated. The specific calculation formula of the sentiment intensity score is as follows: Among them, v i represents the sentiment intensity score of the i-th category of inappropriate content; M represents the total number of sentiment words identified in the text; w j The basic weight of the jth sentiment word; s j The sentiment polarity of the jth sentiment word, positive is +1, negative is -1; r jk K represents the regulatory factor affecting the kth modifier of the jth sentiment word; j Represents the total number of modifiers acting on the jth sentiment word; Indicates the accumulation of weighted sentiment values ​​of all sentiment words; It represents the cumulative result of calculating the modifier adjustment factor of each sentiment word, that is, calculating the final adjusted sentiment value of the sentiment word; The probability of inappropriate content and the sentiment intensity score are calculated comprehensively to obtain the inappropriate content identification weight score. The specific calculation formula is: Where S is the weight score of inappropriate content identification, ranging from [0, 1]; ω is the normalization coefficient; g is the attenuation factor; is a summation symbol; N is the total number of inappropriate content categories; p i is the probability that the predicted text belongs to the i-th category of inappropriate content; v i is the emotional intensity score of the i-th category of content; l is an adjustment factor; h is an offset; tanh represents the hyperbolic tangent function; exp represents the exponential function; Compare the inappropriate content identification weight score with the optimal threshold calculated by the F1 score to mark the inappropriate content text; the F1 score is the harmonic average of precision and recall; S3. Introduce a dynamically adjusted warning threshold and deletion threshold, and dynamically adjust the warning threshold and deletion threshold. The specific formula is: θ1(t)=μ1+δ1·sin(ω1t+φ1)+ξ1·log(1+η·E(t)) θ2(t)=μ2+δ2·cos(ω2t+φ2)+ξ2·log(1+η·E(t)) Among them, θ1(t) is the warning threshold that changes with time, θ2(t) is the deletion threshold that changes with time; μ1 and μ2 are baseline thresholds; δ1 and δ2 are parameters that control the amplitude of the warning threshold and the deletion threshold, respectively; ω1 and ω2 are the frequency of change; φ1 and φ2 are the phases; ξ1 and ξ2 are the coefficients that enhance the influence of environmental factors; η is the scaling factor of environmental influence; E(t) is the external environmental variable; log(1+η·E(t)) indicates that the environmental factor is processed using a logarithmic function; Based on the weight scores for identifying inappropriate content, a graded processing strategy is formulated, and warning, isolation, and deletion measures are taken against users and inappropriate content texts based on the graded processing strategy.

2. The method for identifying and filtering inappropriate content based on a large language model as claimed in claim 1, characterized in that: In step S2, "comparing the inappropriate content identification weight score with the optimal threshold calculated by the F1 score, and marking the inappropriate content text" specifically includes: According to the range of the inappropriate content identification weight score being [0, 1], the candidate thresholds are set at intervals of 0.1, and the candidate threshold that maximizes the F1 score is selected as the optimal threshold. The inappropriate content is marked by comparing the inappropriate content identification weight score with the optimal threshold; if the inappropriate content identification weight score is higher than or equal to the optimal threshold, the marked text is inappropriate content text.

3. The method for identifying and filtering inappropriate content based on a large language model as claimed in claim 1, characterized in that: In step S3, "introducing dynamically adjusted warning thresholds and deletion thresholds, combining inappropriate content identification weight scores, and formulating a graded processing strategy" is as follows: Among them, C(S) is the treatment measure determined according to the weight score of inappropriate content identification; S is the weight score of inappropriate content; θ1 and θ2 are the warning threshold and deletion threshold respectively; When the handling action is warn, a warning is issued to the user, indicating the specific part of the inappropriate content in the inappropriate content text and suggesting modification or deletion; When the handling action is isolated, the inappropriate content text is automatically moved from the public visible area to the isolation area, and the user is notified to modify it before publishing; When the handling action is delete, the inappropriate content text is automatically deleted from the platform, the user is notified that the inappropriate content text has been deleted, and the reason for the deletion is provided.

4. A system for identifying and filtering inappropriate content based on a large language model, applied to the method for identifying and filtering inappropriate content based on a large language model as claimed in claim 1, characterized in that: include: The text collection and preprocessing module is used to automatically collect texts from social media platforms and online forums, and preprocess the collected texts to obtain preprocessed texts; The feature extraction and analysis module is used to extract features from the preprocessed text to obtain word frequency-inverse document frequency feature vectors, word embedding vectors, and syntactic feature vectors, and integrate them into a comprehensive feature vector through vector splicing; the preprocessed text is analyzed through natural language processing to obtain the total number of sentiment words, the basic weight of sentiment words, sentiment polarity, and the total number of modifiers of sentiment words; An inappropriate content identification module is used to construct an inappropriate content probability model based on the comprehensive feature vector to calculate the probability of inappropriate content; construct a sentiment analysis model to calculate the sentiment intensity score based on the total number of sentiment words, the basic weight of sentiment words, the basic sentiment polarity of sentiment words, and the total number of modifiers of sentiment words output by the feature extraction and analysis module, comprehensively calculate the probability of inappropriate content and the sentiment intensity score to obtain an inappropriate content identification weight score, compare the inappropriate content identification weight score with the optimal threshold obtained by calculating the F1 score, and mark the inappropriate content text; A filtering decision module, which is used to set a warning threshold and a deletion threshold based on dynamic adjustment, formulate a hierarchical processing strategy in combination with the inappropriate content identification weight score from the inappropriate content identification module, take processing measures according to the hierarchical processing strategy, and output the processing measures; The user interaction module is used to receive the processing measures and automatically send a notification to the user, explaining the inappropriate content in the text and providing modification suggestions, or notifying the user that the text has been deleted and explaining the reasons.

Images