NLP-based illegal speech processing method and device, and computer equipment

Abstract

The application discloses a method and device for processing illegal speech based on NLP and a computer device. The method comprises the following steps: obtaining a speech text and preprocessing the speech text to obtain a preprocessed speech text; extracting multi-dimensional features from the preprocessed speech text by using a pre-trained language model and fusing the multi-dimensional features to obtain a comprehensive illegal risk representation; classifying the comprehensive illegal risk representation into a main illegal type, identifying a fine-grained label and calculating a confidence score by using a multi-task learning model to obtain a prediction result; wherein the prediction result comprises a confidence score and a severity level; wherein the confidence score is used to represent the degree of correctness and reliability of the multi-task learning model. By implementing the method of the application, accurate identification and intelligent grading disposal of various complex illegal types can be realized, the adaptability of the system to new illegal forms and low-resource languages is enhanced, and the cost and pressure of manual review are reduced.

Description

Technical Field

[0001] This invention relates to the field of internet security technology, and more specifically to a method, apparatus, and computer equipment for handling inappropriate speech based on NLP. Background Technology

[0002] With the widespread adoption of internet social platforms, forums, comment sections, and instant messaging applications, user-generated content has experienced explosive growth. However, amidst this massive amount of information, there is no shortage of content that violates laws and regulations, contravenes platform rules, or undermines public order and morals. This includes, but is not limited to, insults, attacks, discrimination, hatred, violence, terrorism, pornography, vulgarity, false rumors, harassing advertising, and privacy violations. Such illegal content not only severely pollutes the online environment and threatens users' physical and mental health but may also exacerbate social conflicts and even cross legal boundaries.

[0003] Faced with this challenge, the limitations of existing technologies are becoming increasingly apparent. Keyword filtering methods identify violations by matching pre-defined keywords or sensitive word lists, but their accuracy and recall are low, they are prone to misjudging normal expressions, and they struggle to identify violations manipulated through variations, homophones, or other techniques. Rule engines rely on complex, manually written rules for filtering, which improves accuracy to some extent, but their maintenance costs are high, their coverage is limited, and they are poorly adaptable to newly emerging forms of violations. Traditional machine learning models such as SVM and Naive Bayes analyze based on bag-of-words models or TF-IDF features, but their ability to understand deep semantics, context, sentiment, and intent is limited. Manual review is costly and inefficient, making it difficult to meet the needs of large-scale content review, and the subjectivity of review standards and the psychological burden on reviewers are also significant issues. Even current advanced NLP (Natural Language Processing) solutions, despite the application of deep learning models such as CNN and RNN for text classification, still have shortcomings in multi-dimensional recognition capabilities, contextual understanding, refined processing strategies, and adaptability to low-resource languages ​​or specific domains. Therefore, how to effectively solve the above problems and improve the security and quality of online content remains an important issue that urgently needs to be addressed.

[0004] Therefore, it is necessary to design a new method to achieve accurate identification and intelligent graded handling of various complex types of violations, while enhancing the system's adaptability to new forms of violations and low-resource language, and significantly reducing the cost and pressure of manual review, so as to solve the problem of low accuracy and efficiency of existing technologies in handling illegal online speech. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus, and computer equipment for handling illegal speech based on NLP.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: a method for handling inappropriate speech based on NLP, comprising:

[0007] The text of the speech is obtained and preprocessed to obtain the preprocessed text of the speech.

[0008] A pre-trained language model is used to extract semantic, sentiment, intent, and risk entity multi-dimensional features from the pre-processed speech text, and the multi-dimensional features are fused to obtain a comprehensive violation risk representation.

[0009] The comprehensive violation risk representation is classified into main violation types and identified with fine-grained labels using a multi-task learning model. A confidence score reflecting the reliability of the prediction is then calculated to obtain the prediction result. The prediction result includes a confidence score and a severity level. The confidence score is used to characterize the degree to which the multi-task learning model is confident in the correctness of the current result and to measure its reliability.

[0010] The further technical solution is as follows: after classifying the main violation type and identifying fine-grained labels of the comprehensive violation risk representation through a multi-task learning model, and calculating the confidence score reflecting the prediction reliability to obtain the prediction result, it also includes:

[0011] Based on the treatment strategy, the prediction results are mapped to different treatment actions, and the treatment actions are dynamically adjusted.

[0012] The further technical solution is as follows: after mapping the prediction results to different handling actions and dynamically adjusting the handling actions, it also includes:

[0013] Collect the processing results obtained after performing the processing action to optimize the pre-trained language model, the multi-task learning model, and the processing strategy.

[0014] The further technical solution is as follows: The pre-trained language model is used to extract semantic, sentiment, intent, and risk entity multi-dimensional features from the pre-processed speech text, and these multi-dimensional features are fused to obtain a comprehensive violation risk representation, including:

[0015] The deep contextual semantic representation and topic distribution of the preprocessed speech text are obtained using a pre-trained language model to obtain the first feature;

[0016] Identify the overall sentiment tendency and intensity of the preprocessed text and detect specific negative emotions to obtain a second feature;

[0017] The preprocessed text of the speech is analyzed to determine potential intentions in order to obtain a third feature;

[0018] The high-risk entities and their relationships in the preprocessed speech text are detected and understood to obtain the fourth feature;

[0019] Identify hidden expressions that evade detection in the preprocessed speech text to obtain the fifth feature;

[0020] The first feature, the second feature, the third feature, the fourth feature, and the fifth feature are fused into a comprehensive feature representation through vector concatenation or attention mechanism to obtain a comprehensive violation risk characterization.

[0021] The further technical solution is as follows: the comprehensive violation risk representation is classified into main violation types and fine-grained label identification using a multi-task learning model, and a confidence score reflecting the reliability of the prediction is calculated to obtain the prediction result, including:

[0022] The comprehensive violation risk representation is used as input and fed into a multi-task learning model designed to handle multiple related tasks simultaneously.

[0023] The multi-task learning model determines the type of violation to which the speech text belongs based on the comprehensive violation risk representation;

[0024] The multi-task learning model identifies specific violations in the speech text based on the comprehensive violation risk representation;

[0025] Based on the internal uncertainty, feature anomalies, and context-dependent factors of the multi-task learning model, a confidence score reflecting the reliability of the prediction and a severity level of the violation are calculated.

[0026] The output includes review results, including severity level and confidence score, to obtain prediction results, wherein the prediction results also include the main violation type and fine-grained labels.

[0027] Its further technical solution is: the treatment strategy maps the prediction results to different treatment actions and dynamically adjusts the treatment actions, including:

[0028] The engine for the tiered treatment strategy is activated based on the prediction results;

[0029] The engine determines whether to conduct a detailed interactive evaluation based on the prediction results to determine the final handling measures and thus the handling strategy;

[0030] Select and implement specific actions based on the aforementioned handling strategy;

[0031] The severity of the handling measures will be adjusted based on users' historical behavior and the breadth of content dissemination.

[0032] The confidence threshold is adjusted in real time based on model performance feedback and current environmental factors to optimize response decisions.

[0033] Its further technical solution is as follows: the engine determines whether to conduct a detailed interactive evaluation based on the prediction results to determine the final handling measures, thereby determining the handling strategy, including:

[0034] The engine performs non-linear interactive judgment to determine the handling measures by using a specific combination of confidence intervals and severity levels. The response to severity is adjusted according to the confidence level, with low confidence levels resulting in downgraded handling and high confidence levels resulting in escalated handling intensity. Corresponding handling standards are adopted for different types of violations to determine the handling strategy.

[0035] The further technical solution is as follows: collecting the processing results obtained after executing the processing action to optimize the pre-trained language model, the multi-task learning model, and the processing strategy includes:

[0036] Record the results of each action and related information to obtain the outcome of the action;

[0037] Manually review low-confidence or sensitive cases in the model, and clearly indicate the review results and reasons;

[0038] The processed results are cleaned and desensitized, and then converted into a format suitable for machine learning processing.

[0039] The manually reviewed data is used as new training samples and combined with the converted data to form an updated dataset.

[0040] The pre-trained language model and multi-task learning model were retrained and optimized using the updated dataset;

[0041] The rules within the engine are analyzed based on the processing results to adjust the processing strategy.

[0042] The present invention also provides an NLP-based device for processing inappropriate speech, comprising:

[0043] The acquisition unit is used to acquire the speech text and preprocess it to obtain the preprocessed speech text.

[0044] The extraction and fusion unit is used to extract semantic, sentiment, intent, and risk entity multi-dimensional features from the pre-processed speech text using a pre-trained language model, and to fuse the multi-dimensional features to obtain a comprehensive violation risk representation.

[0045] The prediction unit is used to classify the main violation type and identify fine-grained labels on the comprehensive violation risk representation through a multi-task learning model, and calculate a confidence score that reflects the reliability of the prediction to obtain the prediction result. The prediction result includes a confidence score and a severity level. The confidence score is used to characterize the degree of certainty and reliability of the multi-task learning model in determining the correctness of the current result.

[0046] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0047] The advantages of this invention compared to existing technologies are as follows: This invention acquires and preprocesses spoken text, utilizes an advanced pre-trained language model to extract multi-dimensional features covering semantics, sentiment, intent, and risk entities, and integrates these features to form a comprehensive violation risk representation. Subsequently, a multi-task learning model is used to analyze this representation, which can not only accurately identify the main violation type and fine-grained labels, but also calculate the confidence score and severity level reflecting the reliability of the prediction. This method, through comprehensive coverage and intelligent hierarchical handling of various complex violation types, greatly enhances the system's adaptability to emerging violation forms and low-resource language, significantly reduces the need for and pressure of manual review, and effectively solves the problems of low accuracy and efficiency faced by existing technologies in handling illegal online speech, thus providing a more efficient, accurate, and automated solution for managing illegal content.

[0048] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 This is a schematic diagram illustrating an application scenario of the NLP-based method for handling inappropriate speech provided in an embodiment of the present invention.

[0051] Figure 2 A flowchart illustrating the NLP-based method for handling inappropriate speech provided in an embodiment of the present invention;

[0052] Figure 3 A flowchart illustrating an NLP-based method for handling inappropriate speech, as provided in another embodiment of the present invention. Figure 1 ;

[0053] Figure 4 A flowchart illustrating an NLP-based method for handling inappropriate speech, as provided in another embodiment of the present invention. Figure 2 ;

[0054] Figure 5 A schematic block diagram of an NLP-based illegal speech processing device provided in an embodiment of the present invention;

[0055] Figure 6 Schematic diagram of an NLP-based illegal speech processing device provided for another embodiment of the present invention Figure 1 ;

[0056] Figure 7 Schematic diagram of an NLP-based illegal speech processing device provided for another embodiment of the present invention Figure 2 ;

[0057] Figure 8 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0059] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0060] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0061] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0062] Please see Figure 1 and Figure 2 , Figure 1This is a schematic diagram illustrating an application scenario of the NLP-based method for handling inappropriate speech provided in an embodiment of the present invention. Figure 2 This is a schematic flowchart illustrating the NLP-based method for handling inappropriate speech provided in this embodiment of the invention. This NLP-based method is applied to a server. The server interacts with the terminal, extracting multi-dimensional features from the speech text using a pre-trained language model and fusing these features to obtain a comprehensive risk representation of the violation. Subsequently, a multi-task learning model is used to classify the main violation type and identify fine-grained labels in the text, while simultaneously calculating a confidence score reflecting the reliability of the prediction, thereby achieving accurate identification and intelligent graded handling of inappropriate speech. The system dynamically adjusts the handling strategy based on the prediction results. For low-confidence or sensitive cases, manual review is introduced, and the review results are fed back to the system to optimize the model and strategy. This method not only enhances adaptability to new forms of violations and low-resource language but also significantly reduces the cost and pressure of manual review, effectively solving the problems of low accuracy and efficiency in handling inappropriate online speech in existing technologies. Through continuous data collection, model retraining, and strategy adjustment, the system can continuously evolve, improving its ability and effectiveness in handling inappropriate speech.

[0063] Figure 2 This is a flowchart illustrating the NLP-based method for handling inappropriate speech provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S130.

[0064] S110. Obtain the speech text and preprocess it to obtain the preprocessed speech text.

[0065] In this embodiment, speech text refers to user-generated content from various online platforms and channels, including raw text data containing diverse information such as communication, expression of opinions, and emotional expression.

[0066] Diverse sources: The texts can originate from various platforms and channels, such as social media comment sections, forum posts, instant messaging chat logs, blog articles, etc. These texts may include exchanges between users, expressions of opinions, and emotional outpourings, covering a wide range of topics and styles.

[0067] Varying formats: Due to different sources, the formats of spoken texts also vary, including but not limited to plain text, rich text with HTML tags, and text containing special symbols or emoticons.

[0068] Preprocessed speech text refers to standardized text data that has undergone a series of processes such as cleaning, standardization, word segmentation, and entity recognition to remove noise and be transformed into a normalized text data suitable for subsequent natural language processing tasks.

[0069] Preprocessing is essential for improving the performance of subsequent natural language processing tasks, ensuring that the data input into the model is clean and easy to analyze. The preprocessed text undergoes the following steps:

[0070] Clean the text:

[0071] Remove irrelevant characters: Delete non-printable characters, extra spaces, newlines, etc. from the text.

[0072] Remove HTML tags: If the text contains HTML tags from a webpage, these tags need to be removed to extract the plain text content.

[0073] Clean up special symbols: Depending on the actual situation, it may be necessary to remove or convert specific punctuation marks, emojis, etc., to facilitate subsequent analysis.

[0074] Optional word segmentation and standardization:

[0075] For languages ​​that require word segmentation, such as Chinese, use appropriate word segmentation tools (such as Jieba) to divide sentences into words or phrases.

[0076] Performing lemmatization or stemming can help reduce the impact of lexical morphological changes on the analysis, especially for languages ​​like English.

[0077] Text standardization, such as unifying capitalization and converting traditional Chinese characters to simplified Chinese characters, makes text more standardized and consistent.

[0078] Identify and process entity information:

[0079] Identify entities in text, such as names of people, places, and organizations, and mark or replace them to protect privacy while facilitating subsequent analysis.

[0080] Handle sensitive words (such as contact information), links, etc., and decide whether to retain or block them based on business needs.

[0081] Characterization processing:

[0082] For some specific application scenarios, further feature processing may be required, such as calculating text length and counting keyword frequency, to provide additional information support for subsequent tasks.

[0083] Through the aforementioned series of preprocessing steps, the final preprocessed text not only removes noise and improves data quality, but also better meets the input requirements of machine learning models, thus enhancing model performance and accuracy. This preprocessing result lays a solid foundation for subsequent steps such as multi-dimensional feature extraction and fusion, and multi-task violation detection and classification.

[0084] S120. Using a pre-trained language model, extract semantic, emotional, intentional, and risk entity multi-dimensional features from the pre-processed speech text, and fuse the multi-dimensional features to obtain a comprehensive violation risk representation.

[0085] In this embodiment, comprehensive violation risk characterization refers to obtaining comprehensive information reflecting the likelihood of text violations from preprocessed speech text through multi-dimensional feature extraction and fusion. This information includes, but is not limited to, semantics, sentiment, intent, risk entities, and covert expressions, ultimately forming a comprehensive feature representation that can quantitatively assess the risk of text violations.

[0086] In one embodiment, step S120 described above may include steps S121 to S126.

[0087] S121. Use a pre-trained language model to obtain the deep contextual semantic representation and topic distribution of the pre-processed speech text, so as to obtain the first feature.

[0088] In this embodiment, the first feature refers to using a pre-trained language model (such as BERT, RoBERTa, etc.) to perform deep contextual semantic analysis on the pre-processed speech text to obtain word vectors or sentence vectors of the text, and further extracting the topic distribution of the text through a topic model (such as LDA, Top2Vec). This step aims to capture the deep semantic structure of the text and the topic domains under discussion.

[0089] S122. Identify the overall emotional tendency and intensity of the preprocessed speech text, and detect specific negative emotions to obtain a second feature.

[0090] In this embodiment, the second feature refers to identifying the overall sentiment (positive / negative / neutral) and intensity of the text, with particular attention to high-intensity negative emotions that may be associated with offensive speech, such as anger, hatred, and disgust. This step helps to understand the emotional tone behind the text and its potential impact.

[0091] S123. Analyze the preprocessed speech text to determine potential intentions in order to obtain a second feature.

[0092] In this embodiment, the third feature refers to analyzing the text to determine its underlying intent, such as whether it contains intentions of insult, threat, discrimination, or sexual enticement. Combining the results of semantic and sentiment analysis, this step can more accurately determine the true purpose and motivation of the text.

[0093] S124. Detect and understand the high-risk entities involved in the preprocessed speech text and the relationships between them to obtain the fourth feature.

[0094] In this embodiment, the fourth feature refers to detecting and understanding high-risk entities mentioned in the text (such as specific individuals or group names) and the relationships between them (such as attacks or defamation). This feature is crucial for identifying malicious behavior targeting specific individuals or groups.

[0095] S125. Identify the hidden expressions in the preprocessed speech text that evade detection, in order to obtain the fifth feature.

[0096] In this embodiment, the fifth feature refers to identifying hidden expressions in the text that attempt to evade detection, such as the use of variants, homophones, word splitting, symbol insertion, metaphors, etc. This feature helps to discover clever expressions that attempt to bypass the automatic detection system.

[0097] S126. The first feature, the second feature, the third feature, the fourth feature, and the fifth feature are fused into a comprehensive feature representation through vector concatenation or attention mechanism to obtain a comprehensive violation risk representation.

[0098] In step S126, the first to fifth features mentioned above are fused into a comprehensive feature representation, namely a comprehensive violation risk representation, through methods such as vector concatenation or attention mechanisms. This comprehensive feature not only covers multi-dimensional information of the text but also considers the interaction between different features, thus providing a comprehensive and detailed basis for violation risk assessment.

[0099] In step S120, multi-dimensional features that can comprehensively reflect the risk of violations are extracted from the preprocessed speech text, and these features are effectively integrated to form a comprehensive feature representation.

[0100] Pre-trained models such as BERT, RoBERTa, ERNIE, and XLNet are used to obtain deep contextual semantic representations of text. These models, trained on large-scale corpora, are able to capture complex relationships between words and the deep meaning of sentences.

[0101] Methods such as LDA (Latent Dirichlet Allocation) or Top2Vec can be used to analyze the topic distribution of texts. This helps identify the main topics discussed in the texts, providing support for further understanding of the text's context.

[0102] Specialized sentiment analysis algorithms are applied to determine the overall sentiment tendency (positive, negative, or neutral) of text and quantify its sentiment intensity. This is particularly important for detecting statements containing strong negative emotions.

[0103] Further identification of specific emotional types in the text, such as anger, hatred, and disgust, especially negative emotions highly correlated with violations, enhances insight into the potential harm of the text.

[0104] Based on the results of semantic and sentiment analysis, combined with domain knowledge or a fine-tuned classifier, the potential intent of the text is determined, including abusive language, threats, discrimination, and sexual enticement. This process requires careful consideration of the text's context to accurately capture the author's true intent.

[0105] Identifying specific individuals, groups, or other sensitive information mentioned in text and assessing the relationships between them (attacks, defamation, etc.) is crucial for protecting personal privacy and preventing malicious attacks against specific individuals or groups.

[0106] Natural language processing techniques are used to identify various methods used in text to circumvent automatic detection systems, such as variations, homophones, word splitting, and symbol insertion. This step aims to enhance the system's adaptability to new types of violations.

[0107] This involves fusing all the extracted features from different dimensions (semantic vectors, sentiment scores, intent labels, risk entity information, style features, etc.). Features can be directly combined using vector concatenation, or an attention mechanism can be employed to dynamically adjust weights based on feature importance for more effective fusion. Furthermore, multimodal fusion methods can be explored when other types of data, such as images and audio, are present.

[0108] Through this series of detailed feature extraction and fusion steps, a comprehensive and detailed text violation risk representation can be constructed, providing strong support for subsequent violation determination and graded handling. This multi-level and multi-angle approach not only improves the accuracy of identification but also enhances the robustness and adaptability of the system.

[0109] S130. The comprehensive violation risk representation is classified into main violation types and identified with fine-grained labels using a multi-task learning model, and a confidence score reflecting the reliability of the prediction is calculated to obtain the prediction result. The prediction result includes a confidence score and a severity level. The confidence score is used to characterize the degree of certainty and reliability of the multi-task learning model in determining the correctness of the current result.

[0110] In this embodiment, the prediction result refers to a series of output information obtained after processing the input speech text through a multi-task learning model. This information specifically includes, but is not limited to, the following aspects:

[0111] Primary Violation Type: This refers to the type of content norms or guidelines that the speech primarily violates, as determined by a multi-task learning model based on a comprehensive violation risk profile. For example, the speech might be categorized as abusive, discriminatory, violent, or pornographic.

[0112] Fine-grained labels: In addition to identifying the main violation type, the model also identifies more specific details of the violation. This helps to more accurately describe the content and nature of the offending speech, such as distinguishing between racial discrimination and sexism, and whether it involves personal attacks or terrorist content.

[0113] Confidence Score: This is a metric reflecting the correctness and reliability of the model's current judgment. It is not solely based on classification probabilities (such as the maximum probability in Softmax), but is a composite metric that comprehensively considers multiple factors, including internal model uncertainty, feature space anomalies, and contextual dependencies. The confidence score helps assess how confident the model is in determining whether a particular statement violates regulations.

[0114] Severity Level: Based on the specific nature of the violation (such as the malicious intent, the intensity of the emotion, the sensitivity of the entity involved, etc.), a level is assigned to indicate the severity of the violation. This level can be used to guide subsequent handling strategies, such as different levels like minor, moderate, serious, or extremely serious.

[0115] In step S130, a multi-task learning (MTL) model designed to handle multiple related tasks is used to further analyze the comprehensive violation risk representation after feature extraction and fusion. The core objective of this step is to determine the main violation type and fine-grained label of the text, and to calculate a confidence score and severity level that reflect the reliability of the model's predictions.

[0116] In one embodiment, step S130 described above may include steps S131 to S135.

[0117] S131. The comprehensive violation risk representation is used as input and fed into a multi-task learning model designed to handle multiple related tasks simultaneously.

[0118] First, the comprehensive violation risk representation generated in step S120 (including semantic vectors, sentiment scores, intent labels, risk entity information, style features, etc.) is fed into a pre-trained MTL model. This model is typically based on a deep neural network architecture, such as BERT or other pre-trained language models, and is designed to handle multiple related subtasks simultaneously.

[0119] S132. The multi-task learning model determines the type of violation to which the speech text belongs based on the comprehensive violation risk representation.

[0120] Next, the MTL model uses a shared underlying encoder to process the comprehensive violation risk representation of the input, and then outputs separate prediction results for different subtasks. At this stage, the model determines whether the slang text belongs to one or more specific violation types (e.g., abusive, discriminatory, violent, pornographic, etc.) based on the comprehensive feature vector. This is a multi-classification problem, where each category represents a primary violation.

[0121] S133. The multi-task learning model identifies specific violations of the speech text based on the comprehensive violation risk representation.

[0122] In addition to identifying the primary violation type, the MTL model also identifies more specific details of the violation (i.e., fine-grained labels). This step helps to more accurately describe the content and nature of the offending speech, such as distinguishing between racial discrimination and sexism, or clearly indicating whether there is personal attack or terrorist content. This process also relies on the model's ability to learn from the input features.

[0123] S134. Based on the internal uncertainty, feature anomalies, and context-dependent factors of the multi-task learning model, calculate the confidence score and the severity level of the violation, which reflect the reliability of the prediction.

[0124] Internal uncertainty: Evaluate the consistency of the outputs of each task within the model and the level of confidence in the current input.

[0125] Feature anomalies: Check for conflicts or anomalies between different dimensions in the fused feature vector.

[0126] Context-dependent: Adjust the final confidence score by considering factors such as conversation history and user profile.

[0127] Based on the above factors, and combining the model's classification results for the main violation type and the fine-grained label recognition results, a comprehensive confidence score is calculated. Furthermore, a corresponding severity level is assigned according to the specific nature of the violation (such as the degree of malice, the importance of the sensitive entity involved, etc.).

[0128] S135. Output the review results, including severity level and confidence score, to obtain the prediction results, wherein the prediction results also include the main violation type and fine-grained label.

[0129] Finally, the MTL model outputs review results including severity level and confidence score. This information not only reflects the model's determination of whether the text violates regulations but also provides important metrics on the model's predictive accuracy and reliability. Based on these outputs, subsequent steps can determine what actions to take (e.g., removing content, warning users, or submitting for manual review) according to a pre-defined tiered handling strategy.

[0130] Therefore, in step S130 of this embodiment, the MTL model first needs to determine whether the text contains violations and identify the most likely major violation type. These types include, but are not limited to, abusive language, discrimination, violence, pornography, misinformation, advertising, and privacy violations. This process can be viewed as a multi-class or hierarchical classification problem, aiming to provide a specific violation category label for each input.

[0131] In addition to identifying the primary violation type, the MTL model needs to further refine the violation description and output more detailed violation labels. For example, after confirming "discrimination" as the primary type, finer-grained labels might further specify whether it is racial discrimination or gender discrimination, etc. This fine-grained classification helps to more accurately understand the specific manifestations of the violation.

[0132] This is one of the core innovations in the entire process. For each statement, the MTL model not only provides the probability that it belongs to a certain violation category, but also a confidence score (between 0 and 1) reflecting the reliability of the judgment, and a severity level (mild, moderate, severe, extremely severe) based on a comprehensive assessment of multiple factors such as malicious intent, emotional intensity, sensitivity of the entities involved, and estimated scope of dissemination. Importantly, the confidence score here does not solely rely on the maximum value of the classification probability, but is comprehensively measured through the following methods:

[0133] Internal model uncertainty: The reliability of model predictions is assessed by comparing the consistency of outputs across different tasks. When internal discrepancies are detected within the model, the overall confidence level will decrease accordingly.

[0134] Prediction variance / entropy: This calculates the variance or entropy of the model's predictions, serving as a measure of model uncertainty. High variance or entropy indicates a lack of confidence in the model regarding the current input.

[0135] Evidence accumulation: Employing theoretical frameworks such as the Dirichlet distribution, this approach not only outputs classification probabilities but also considers the amount of evidence supporting the prediction. Lower evidence levels correspond to higher uncertainty.

[0136] In addition, the confidence score also considers factors such as feature-level anomalies, conflict signals, and contextual dependencies, thereby ensuring that the score accurately reflects the actual violations in the text and their potential risks.

[0137] In summary, through these three tasks, the MTL model can comprehensively and meticulously analyze text content, not only identifying its main violation types and fine-grained labels, but also assessing the severity and confidence level of violations based on a series of complex algorithms. This is of great significance for the effective management and monitoring of cyberspace.

[0138] The aforementioned NLP-based method for handling inappropriate speech acquires and preprocesses the speech text, then utilizes an advanced pre-trained language model to extract multi-dimensional features encompassing semantics, sentiment, intent, and risk entities. These features are then fused to form a comprehensive representation of the inappropriate risk. Subsequently, a multi-task learning model is employed to analyze this representation, enabling accurate identification of the primary violation type and fine-grained labels, as well as calculation of confidence scores and severity levels reflecting prediction reliability. This method, through comprehensive coverage of various complex violation types and intelligent hierarchical handling, significantly enhances the system's adaptability to emerging violation forms and low-resource language, substantially reducing the need for and pressure of manual review. It effectively addresses the accuracy and efficiency issues faced by existing technologies in handling inappropriate online speech, thus providing a more efficient, accurate, and automated solution for managing inappropriate content.

[0139] Figure 3 This is a flowchart illustrating a method for handling inappropriate speech based on NLP, provided in another embodiment of the present invention. Figure 3 As shown, the NLP-based method for handling inappropriate speech in this embodiment includes steps S210-S240. Steps S210-S230 are similar to steps S110-S130 in the previous embodiment and will not be described again here. The following details the additional step S240 in this embodiment.

[0140] S240. Based on the handling strategy, the prediction results are mapped to different handling actions, and the handling actions are dynamically adjusted.

[0141] In one embodiment, step S240 described above may include steps S241 to S245.

[0142] S241. Activate the engine of the graded treatment strategy based on the prediction results.

[0143] In this embodiment, when the system receives the prediction results (including the main violation type, fine-grained label, confidence score, and severity level) output in step S230, it will activate the designed hierarchical handling strategy engine. This engine is the core of the entire process, responsible for making complex decision-making judgments based on the multi-dimensional features of the input.

[0144] S242. The engine determines whether to conduct a detailed interactive evaluation based on the prediction results to determine the final disposal measures, thereby determining the disposal strategy.

[0145] In this embodiment, the engine performs non-linear interactive judgment to determine the handling measures by using a specific combination of confidence intervals and severity levels. The response to severity is adjusted according to the confidence level, with low confidence levels resulting in downgraded handling and high confidence levels resulting in escalated handling intensity. Corresponding handling standards are adopted for different types of violations to determine the handling strategy.

[0146] The tiered handling strategy engine determines whether further interactive evaluation is needed to refine the final handling measures based on the specific combination of confidence level and severity. For example, in some cases, although the confidence level is low, the severity level is high, and the system may choose to mark the case as priority for manual review in order to more accurately assess the authenticity and potential harm of the content.

[0147] S243. Select and implement specific handling actions based on the aforementioned handling strategy.

[0148] Once the handling strategy is determined, the system will execute the corresponding actions based on that strategy. This may include, but is not limited to, immediately deleting content, banning accounts, warning users, or reducing content visibility. Each action is designed to effectively manage violations while minimizing disruption to legitimate users.

[0149] S244. Adjust the severity of the handling measures based on users' historical behavior and the breadth of content dissemination.

[0150] In this embodiment, this step takes into account the user's historical behavior records and the scope of content dissemination, using these as the basis for adjusting the severity of the handling measures. For example, the system may impose more severe penalties on users who repeatedly violate the rules; while for widely disseminated content, stricter review standards may be adopted to ensure the safety and health of the platform environment.

[0151] S245. Adjust the confidence threshold in real time based on model performance feedback and current environmental factors to optimize the handling decision.

[0152] Finally, the system dynamically adjusts the confidence threshold based on factors such as the model's online performance (e.g., false positive rate (FPR) and false negative rate (FNR), changes in public opinion over a specific time period, and user group characteristics. This not only helps improve the system's accuracy but also enhances its adaptability, making decision-making more aligned with reality, thereby significantly improving user experience and the platform's overall management level.

[0153] This multi-layered, dynamically adjusted approach effectively addresses the inaccuracy and inefficiency issues inherent in traditional methods. It enhances the system's adaptability to emerging forms of violations and low-resource language, while significantly reducing the need for and pressure of manual review. This enables accurate identification and intelligent management of online speech violations.

[0154] In this embodiment, the core innovation of this engine lies in its combination of confidence scores and severity levels, mapping them to different action sequences. The following is an example strategy:

[0155] Confidence level > High threshold AND Severity = Extremely high: Immediately delete content + permanently ban account + report to regulatory authorities.

[0156] Confidence > High Threshold AND Severity = High: Immediately delete content + temporarily ban the account (e.g., 7 days) + user warning.

[0157] Confidence > Medium threshold AND Severity = Medium: Delete content + user warning or short-term ban (e.g., 24 hours).

[0158] Confidence level > low threshold AND severity = low: Collapse content / reduce visibility / mark as "content questionable" + user prompt. Or enter the manual review queue.

[0159] Confidence level < low threshold: Do not process / release temporarily. Or enter the low-priority manual sampling queue.

[0160] The strategy can be dynamically adjusted based on the specific circumstances of the violation type. For example, content involving illegality should be immediately deleted and the account suspended and reported, even if the confidence level is moderate. In addition, the strategy can combine the user's historical violation records for additional penalties and consider the scope of content dissemination to determine stricter handling measures.

[0161] The core innovation of the tiered response strategy engine lies in the fact that confidence scores and severity levels are not used independently or simply linearly, but are achieved through complex conditional interactive judgments:

[0162] Nonlinear mapping: The selection of action is not a simple multiplication of confidence level by severity weight, but a multidimensional decision table or decision tree / rule engine. For example, "high confidence (>0.9) + very high severity" triggers the most severe action; "medium confidence (0.6-0.8) + high severity" may trigger deletion + manual review priority queue; "low confidence (<0.4) + high severity" may trigger marking + advanced manual review; "low confidence + low severity" may simply be recorded or ignored.

[0163] Correction of severity judgment by confidence level: When the confidence level is low, even if the initial judgment of severity is high, the handling strategy may be downgraded because the model is not very confident in this judgment. Conversely, extremely high confidence level may amplify the impact of severity. For example, for confirmed child pornography, even if the severity score judged by the model is not the highest, the most stringent handling may be implemented.

[0164] Violation type specificity: Different violation types have different tolerance levels for confidence and severity. For example, content involving terrorism or child safety may trigger immediate removal and reporting even with a moderate confidence level; while minor advertising harassment requires a higher confidence level to be removed.

[0165] The concept of "risk tolerance" is introduced: the strategy engine has built-in tolerance thresholds for different types of risks (such as security risks, legal risks, and user experience risks). The combination of confidence level and severity level ultimately maps to whether the tolerance threshold for a specific risk type has been exceeded, thereby triggering corresponding actions. This mapping relationship is a highly nontrivial and carefully designed business rule.

[0166] Dynamic threshold adjustment: The confidence thresholds (high / medium / low) are not fixed values, but can be dynamically adjusted based on the model's online performance (through feedback loop monitoring of FPR, FNR), specific time periods (such as peak periods), or specific user groups / regions. This allows the system to adapt to constantly changing environments and needs, optimizing decision-making.

[0167] Figure 4 This is a flowchart illustrating a method for handling inappropriate speech based on NLP, provided in another embodiment of the present invention. Figure 4 As shown, the NLP-based method for handling inappropriate speech in this embodiment includes steps S310-S350. Steps S310-S340 are similar to steps S210-S240 in the previous embodiment and will not be described again here. The following details the additional step S350 in this embodiment.

[0168] S350. Collect the processing results obtained after executing the processing action to optimize the pre-trained language model, the multi-task learning model, and the processing strategy.

[0169] In one embodiment, step S350 described above may include steps S351 to S356.

[0170] S351. Record the results of each action and related information to obtain the outcome of the action.

[0171] In this embodiment, the processing result refers to the result obtained after each processing action is performed.

[0172] The system needs to record in detail each action decision made and its results. This includes not only direct actions (such as deleting content or banning users), but also all information related to the decision, such as confidence score, severity level, violation type, and user history. This information provides rich data support for subsequent analysis.

[0173] S352. Manually review low-confidence or sensitive cases of the model, and clearly indicate the review results and reasons.

[0174] For low-confidence predictions or cases involving sensitive topics, manual review is required. During the review process, reviewers must not only confirm or overturn the model's initial judgments but also provide detailed reasons and annotations for the review. This high-quality manually annotated data is a key resource for improving model performance.

[0175] S353. Clean and desensitize the processing results, and convert them into a format suitable for machine learning processing.

[0176] To protect user privacy and ensure data security, all collected processing results must undergo cleaning and anonymization. Simultaneously, this data needs to be converted into a format suitable for machine learning algorithms to facilitate subsequent data processing and model training.

[0177] S354. The manually reviewed data is used as new training samples and combined with the converted data to form an updated dataset.

[0178] The cleaned and desensitized data, along with newly trained samples that have undergone manual review, are integrated to form an updated dataset. This new dataset contains the latest information and more accurate annotations, effectively improving the model's learning ability and accuracy.

[0179] S355. Retrain and optimize the pre-trained language model and multi-task learning model using the updated dataset.

[0180] Retraining pre-trained language models (such as BERT-base-Chinese) and multi-task learning models using updated datasets helps the models better understand and adapt to changing linguistic environments and violation patterns, thereby improving their accuracy and robustness in real-world applications.

[0181] S356. Analyze the rules within the engine based on the processing results to adjust the processing strategy.

[0182] Finally, the rules within the tiered handling strategy engine are analyzed based on the handling results, and the handling strategy is adjusted accordingly. This method allows for dynamic optimization of strategy parameters, making them more aligned with actual circumstances and business needs. Furthermore, new risk tolerance thresholds can be introduced or existing thresholds adjusted to further refine the handling measures.

[0183] By continuously collecting and feeding back the results of the processing, especially the data from manual review, the model can learn the correct judgment methods in more complex situations, thereby significantly improving the accuracy of identification and classification.

[0184] As the model continuously receives new data and is retrained, it becomes better able to cope with new types of violations and changes in language expression, demonstrating stronger adaptability.

[0185] A refined, tiered approach to handling violations not only improves the efficiency of processing inappropriate content but also reduces the negative impact of misjudgments, maintains a healthy platform environment, and enhances user experience.

[0186] Automated processes reduce reliance on extensive manual review, lower labor costs, and make content management more efficient and organized.

[0187] In summary, step S350 and its sub-steps constitute a complete feedback loop mechanism, ensuring that the entire process from data collection and model training to strategy adjustment can be effectively managed and continuously optimized. This is crucial for building an intelligent and efficient system for managing illegal content.

[0188] In this embodiment, relevant platform interfaces are invoked to perform actions such as deleting content, banning users, sending warning notifications, logging violations, and reporting. The key is to establish a closed-loop feedback mechanism:

[0189] Collect and process the results: Pay special attention to the results of manual review (confirming or overturning the model's judgment).

[0190] High-quality labeled data feedback: The results of manual review and user appeals are used as high-quality labeled data and continuously fed back into the training dataset.

[0191] Model optimization: Regularly update and optimize the model (feature extraction model of S2, multi-task decision model of S3) and adjust the handling strategy (strategy engine parameters of S4) using feedback data.

[0192] Performance monitoring: Monitor the model's performance metrics (accuracy, recall, F1 score, AUC, false positive rate, false negative rate) and the effectiveness of the measures taken.

[0193] Deep NLP models (especially pre-trained models) combine multi-dimensional feature fusion to significantly improve the ability to understand complex semantics, context, intent, irony and metaphor, and greatly reduce misjudgment and missed judgment.

[0194] The multi-task learning framework can effectively identify multiple types of inappropriate speech simultaneously.

[0195] It innovatively combines confidence level with severity level to achieve intelligent and tiered handling strategies, avoiding a "one-size-fits-all" approach and balancing security and user experience.

[0196] It has a better ability to detect circumvention methods such as variations, homophones, and hidden expressions; it can quickly adapt to new forms of violations through continuous learning.

[0197] Automated processing of massive amounts of content greatly reduces the burden of manual review and lowers operating costs.

[0198] It can be combined with model interpretation technologies (such as LIME, SHAP) to provide judgment criteria (such as which words and phrases triggered violations), assisting in manual review and user education.

[0199] The model framework can be transferred to different languages ​​(with fine-tuning through corpora) and different vertical fields (such as e-commerce, games, and communities).

[0200] Specifically, we will collect a massive amount of labeled text data, covering various types of offensive speech (positive samples) and normal speech (negative samples). Labeling must include: the main violation type, fine-grained tags, severity level, and suggested actions. The data must be cleaned and anonymized.

[0201] Choose a pre-trained language model (such as BERT-base-Chinese) and pre-train it on a large-scale general corpus (or directly use an open-source pre-trained model).

[0202] Domain adaptation is performed on compliant / non-compliant corpora in specific domains (such as social media) to improve their understanding of target scenarios.

[0203] Train the sentiment analysis model and the intent recognition model (using a specific architecture or fine-tuning based on a pre-trained model).

[0204] Train the entity recognition model (NER).

[0205] Design an MTL model architecture (such as sharing the underlying encoder - BERT, and setting up output headers for different tasks in the upper layer).

[0206] The fused feature vector after processing (or the original text input can be used directly to allow the model to learn the features internally).

[0207] Design a suitable loss function for each task (e.g., use cross-entropy for T1 / T2, and mean squared error or ranking loss for T3), and perform weighted joint training.

[0208] Use the collected labeled data for training, validation, and testing. Optimize hyperparameters.

[0209] Deploy the trained model (feature extraction model, MTL model) as a callable API service or an embedded content processing pipeline.

[0210] Develop or integrate the handling strategy engine (S4) and configure policy rules and thresholds.

[0211] Develop a module to execute actions and connect it to the platform backend.

[0212] Build a manual review backend system to receive content with low confidence or specific types.

[0213] Build a data pipeline to enable the automatic collection, cleaning, and feedback of data.

[0214] A user commented: "[A certain celebrity] is an [insulting term], and their fans are all [discriminatory terms]. I suggest using [violent terms]!"

[0215] Clean the text and perform word segmentation.

[0216] Semantic model: It understands that the text contains strong negative evaluations and attacks.

[0217] Emotional model: Outputs extremely high levels of anger and disgust.

[0218] Intent Model: Identifies intentions of "personal attack", "group discrimination", and "incitement to violence".

[0219] Risky Entities: Identify celebrity names (high-risk entities), and identify insulting, discriminatory, and violent words.

[0220] Style: No obvious variations, direct expression.

[0221] Fusion characteristics: The above information is combined to form a high-risk feature vector for violations.

[0222] Main type = "Abusive attack" + "Discrimination and hatred".

[0223] Fine-grained tags = "personal insult", "group denigration", "incitement to harm".

[0224] Confidence level = 0.98 (very high), severity = "extremely serious" (due to the involvement of specific individuals and incitement to violence).

[0225] The strategy engine makes the following decisions based on (Main type = abusive / discriminatory, confidence level = 0.98, severity level = extremely severe): Immediately delete the content + permanently ban the account + report the incident.

[0226] Perform deletion and ban. This judgment has a high confidence level and usually does not require manual review. Log the transaction.

[0227] When scoring confidence for training tasks, it is necessary not only to label samples as violating or normal, but also to annotate the "uncertainty" or "difficulty of judgment" of the samples (if possible). For example, human annotators can record their own level of confidence in their judgments during annotation, or annotate samples with differing opinions.

[0228] In addition to teaching the model to predict the main class and severity, the loss function needs to explicitly include optimizations for uncertainty estimation. For example:

[0229] For evidence accumulation methods, loss can be estimated using methods such as maximum likelihood estimation.

[0230] Regarding model divergence, the loss function is designed to encourage a negative correlation between inter-task consistency and prediction confidence (i.e., prediction confidence should be low when divergence is high).

[0231] Training with adversarial or difficult samples forces the model to output low confidence on these samples.

[0232] Describe how to implement this in detail (e.g., train an auxiliary sub-model or use rules specifically to detect inconsistencies between key feature pairs such as semantic-sentiment and semantic-intent, and quantify them as a conflict score to be input into the confidence calculation).

[0233] The specific OOD detection technology used (such as EnergyScore, MahalanobisDistance, or a specially trained OOD detector) is used, and its output is used as an important input factor for confidence calculation.

[0234] A comprehensive reflection of the nontriviality of confidence level

[0235] This embodiment explicitly points out that traditional probability P(class) cannot fully reflect model reliability, feature consistency, context dependence, and OOD risk. It systematically elaborates on the specific technical implementation methods for the above four "core nontrivialities" (internal model uncertainty, feature space anomalies, context dependence, and nonlinear interaction with severity). This is the core difference from existing technologies. Confidence is a synthetic metric obtained through complex calculations (fusing multiple sub-modules / signals), rather than the direct output of a single model.

[0236] In order to train a model that can accurately output such nontrivial confidence scores, the special requirements for training data labeling include, but are not limited to:

[0237] Is uncertainty labeling required?

[0238] Difficult sample identification;

[0239] Multiple annotation disagreement records;

[0240] Describe in detail the design of the confidence loss function, and how to integrate objectives such as uncertainty modeling, feature conflict penalty, and OOD detection.

[0241] The tiered handling strategy engine provides more specific examples of decision-making logic or pseudocode / flowcharts, clearly demonstrating how multiple input variables such as confidence level, severity, violation type, and user history jointly determine the final handling action through non-linear, conditional rules. It emphasizes that this strategy engine embeds the platform's risk management strategy, and its rule design is the result of a deep integration of business logic and technical indicators, exhibiting high customization and non-triviality.

[0242] For the method in this embodiment, a novel confidence scoring mechanism is proposed. This mechanism achieves a deep quantitative evaluation of the reliability of the model's judgment results by comprehensively modeling the following aspects:

[0243] Internal model uncertainty: Considering the uncertainties in the model prediction process, such as changes in data distribution and ambiguity of samples, the method in this embodiment uses a variety of technical means (such as evidence accumulation methods and adversarial training) to optimize the model's perception and expression of uncertainty.

[0244] Feature space conflict: A feature conflict detection module was specifically designed to identify and quantify inconsistencies between different feature dimensions, thereby adjusting the confidence score.

[0245] Context-dependent: By gaining a deeper understanding of the context and background knowledge of the text content, the model's ability to understand complex semantic structures (such as irony and metaphor) is enhanced, ensuring more accurate confidence scores.

[0246] Sample distribution characteristics: Introducing OOD detection technology, based on methods such as Energy Score and Mahalanobis Distance, to assess whether a sample belongs to a known category, further improving the effectiveness and robustness of confidence scores.

[0247] Compared with traditional confidence estimation methods based on single classification probability, the confidence scoring mechanism proposed in this invention can more comprehensively and accurately evaluate the reliability of the model output results, especially when dealing with illegal speech recognition tasks in complex scenarios.

[0248] Another core innovation lies in the hierarchical response strategy engine based on non-linear, conditional interaction rules of confidence and severity levels. This engine is designed to ensure that responses are both precise and appropriate, specifically in the following aspects:

[0249] Accuracy: By considering information from multiple dimensions such as confidence level and severity, and combining factors such as user history behavior, more detailed handling rules are formulated to avoid misjudgments caused by simple threshold settings.

[0250] Reasonableness: It not only focuses on immediate risk control, but also emphasizes long-term user experience maintenance, striving to minimize interference with normal users while ensuring content security.

[0251] Flexibility: The strategy parameters can be flexibly adjusted according to different application scenarios and business needs, supporting the deployment of customized risk management strategies.

[0252] In summary, this embodiment effectively solves the main problems existing in the prior art through the above-mentioned innovative measures, and provides a brand-new solution framework for achieving efficient and intelligent content violation detection and handling.

[0253] In another embodiment, step S110 is changed to:

[0254] It acquires multimedia content including text, images, audio, and video, extracts global features, scene features, object features, facial expression features, and text features recognized by OCR from the images, and performs preprocessing to obtain preprocessed information.

[0255] During processing, consistent with step S120, for text features, deep semantic representations are obtained using pre-trained language models (such as BERT);

[0256] For visual features, pre-trained convolutional neural networks (CNNs) (such as ResNet, EfficientNet), Vision Transformer (ViT) or object detection models (such as YOLO, Faster R-CNN) are used to extract global features, scene features, object features, facial expression features, and text features in images recognized by OCR.

[0257] For auditory features, acoustic features (such as MFCCs and spectrograms) are extracted from audio.

[0258] Pre-trained models are used to analyze speech emotion (tone, intensity), detect audio events (such as gunshots, screams), and perform ASR (Automatic Speech Recognition) to convert the speech into text before NLP analysis.

[0259] The subsequent steps are processed according to step S130. This method is particularly suitable for modern social network environments with rich multimedia content. It can significantly improve the understanding of complex linguistic phenomena such as implied meaning, irony, and humor, while also enhancing the effective capture of visual and auditory cues.

[0260] Figure 5 This is a schematic block diagram of an NLP-based illegal speech processing device 300 provided in an embodiment of the present invention. Figure 5As shown, corresponding to the above-described NLP-based method for handling inappropriate speech, the present invention also provides an NLP-based device 300 for handling inappropriate speech. This NLP-based device 300 includes a unit for executing the above-described NLP-based method for handling inappropriate speech, and the device can be configured in a desktop computer, tablet computer, laptop computer, or other terminal. Specifically, please refer to... Figure 5 The NLP-based illegal speech processing device 300 includes an acquisition unit 301, an extraction and fusion unit 302, and a prediction unit 303.

[0261] The acquisition unit 301 is used to acquire the speech text and preprocess it to obtain the preprocessed speech text; the extraction and fusion unit 302 is used to extract semantic, sentiment, intent, and risk entity multi-dimensional features from the preprocessed speech text using a pre-trained language model, and fuse the multi-dimensional features to obtain a comprehensive violation risk representation; the prediction unit 303 is used to classify the main violation type and identify fine-grained labels on the comprehensive violation risk representation through a multi-task learning model, and calculate a confidence score reflecting the reliability of the prediction to obtain a prediction result, wherein the prediction result includes a confidence score and a severity level, wherein the confidence score is used to characterize the degree of grasp and reliability of the multi-task learning model on the correctness of the current result.

[0262] In one embodiment, the extraction and fusion unit 302 includes:

[0263] The first extraction subunit is used to obtain the deep contextual semantic representation and topic distribution of the preprocessed speech text using a pre-trained language model to obtain a first feature; the second extraction subunit is used to identify the overall sentiment tendency and intensity of the preprocessed speech text and detect specific negative emotions to obtain a second feature; the third extraction subunit is used to analyze the preprocessed speech text to determine potential intentions to obtain a third feature; the fourth extraction subunit is used to detect and understand the high-risk entities involved in the preprocessed speech text and their relationships to obtain a fourth feature; the fifth extraction subunit is used to identify the hidden expressions in the preprocessed speech text that evade detection to obtain a fifth feature; and the fusion subunit is used to fuse the first feature, the second feature, the third feature, the fourth feature, and the fifth feature into a comprehensive feature representation through vector concatenation or attention mechanism to obtain a comprehensive violation risk representation.

[0264] In one embodiment, the prediction unit 303 includes:

[0265] The system comprises the following subunits: an input subunit, which takes the comprehensive violation risk representation as input and feeds it into a multi-task learning model designed to handle multiple related tasks simultaneously; a type determination subunit, which uses the multi-task learning model to determine the violation type of the speech text based on the comprehensive violation risk representation; a behavior determination subunit, which uses the multi-task learning model to identify the specific violation behavior of the speech text based on the comprehensive violation risk representation; a severity determination subunit, which calculates a confidence score reflecting the reliability of the prediction and the severity level of the violation based on the internal uncertainty, feature anomalies, and context-dependent factors of the multi-task learning model; and an output subunit, which outputs the review results, including the severity level and confidence score, to obtain the prediction results, wherein the prediction results also include the main violation type and fine-grained labels.

[0266] Figure 6 This is a schematic block diagram of an NLP-based illegal speech processing device 300 provided in another embodiment of the present invention. Figure 6 As shown, the NLP-based illegal speech processing device 300 in this embodiment is based on the above embodiment with the addition of an adjustment unit 304.

[0267] The adjustment unit 304 is used to map the prediction results to different handling actions based on the handling strategy, and to dynamically adjust the handling actions.

[0268] In one embodiment, the adjustment unit 304 includes:

[0269] The system comprises the following sub-units: an activation sub-unit for activating the engine of the tiered response strategy based on the prediction results; an evaluation sub-unit for determining whether to conduct a detailed interactive evaluation based on the prediction results to determine the final response measures and thus the response strategy; a selection sub-unit for selecting and implementing specific response actions based on the response strategy; an adjustment sub-unit for adjusting the strictness of the response measures by referring to the user's historical behavior and the breadth of content dissemination; and a threshold adjustment sub-unit for adjusting the confidence threshold in real time based on model performance feedback and current environmental factors to optimize the response decision.

[0270] In one embodiment, the evaluation subunit is used by the engine to perform non-linear interactive judgment to determine the handling measures through a specific combination of confidence intervals and severity levels. The response to severity is adjusted according to the confidence level, with low confidence levels resulting in downgraded handling and high confidence levels resulting in escalated handling intensity. Corresponding handling standards are adopted for different types of violations to determine the handling strategy.

[0271] Figure 7 This is a schematic block diagram of an NLP-based illegal speech processing device 300 provided in another embodiment of the present invention. Figure 7As shown, the NLP-based illegal speech processing device 300 in this embodiment is based on the above embodiment with the addition of an optimization unit 305.

[0272] The optimization unit 305 is used to collect the processing results obtained after executing the processing action in order to optimize the pre-trained language model, the multi-task learning model, and the processing strategy.

[0273] In one embodiment, the optimization unit 305 includes:

[0274] The system comprises the following sub-units: a recording sub-unit, which records the results of each action and related information to obtain the action outcome; a review sub-unit, which manually reviews low-confidence or sensitive cases of the model and annotates the review results and reasons in detail; a processing sub-unit, which cleans and de-identifies the action outcome and converts it into a format suitable for machine learning processing; a combining sub-unit, which uses the manually reviewed data as new training samples and combines it with the converted data to form an updated dataset; a retraining sub-unit, which uses the updated dataset to retrain and optimize the pre-trained language model and multi-task learning model; and a policy adjustment sub-unit, which analyzes the rules within the engine based on the action outcome to adjust the processing policy.

[0275] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned NLP-based illegal speech processing device 300 and its various units can be referred to the corresponding descriptions in the foregoing method embodiments. For the sake of convenience and brevity, these details will not be repeated here.

[0276] The aforementioned NLP-based violation processing device 300 can be implemented as a computer program, which can, for example... Figure 8 It runs on the computer device shown.

[0277] Please see Figure 8 , Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0278] See Figure 8 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0279] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform an NLP-based method for handling inappropriate speech.

[0280] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0281] The internal memory 504 provides an environment for the execution of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an NLP-based method for handling inappropriate speech.

[0282] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0283] The processor 502 is used to run a computer program 5032 stored in the memory to implement all the steps of the NLP-based method for handling inappropriate speech.

[0284] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0285] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0286] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform all the steps of the NLP-based violation speech processing method.

[0287] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0288] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0289] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0290] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0291] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0292] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for handling inappropriate speech based on NLP, characterized in that, include: The text of the speech is obtained and preprocessed to obtain the preprocessed text of the speech. A pre-trained language model is used to extract semantic, sentiment, intent, and risk entity multi-dimensional features from the pre-processed speech text, and the multi-dimensional features are fused to obtain a comprehensive violation risk representation. The comprehensive violation risk representation is classified into main violation types and identified with fine-grained labels using a multi-task learning model. A confidence score reflecting the reliability of the prediction is then calculated to obtain the prediction result. The prediction result includes a confidence score and a severity level. The confidence score is used to characterize the degree to which the multi-task learning model is confident in the correctness of the current result and to measure its reliability. The process involves extracting semantic, sentiment, intent, and risk entity multi-dimensional features from the pre-processed speech text using a pre-trained language model, and fusing these multi-dimensional features to obtain a comprehensive violation risk representation, including: The deep contextual semantic representation and topic distribution of the preprocessed speech text are obtained using a pre-trained language model to obtain the first feature; Identify the overall sentiment tendency and intensity of the preprocessed speech text, and detect negative emotions to obtain a second feature; The preprocessed text of the speech is analyzed to determine potential intentions in order to obtain a third feature; The high-risk entities and their relationships in the preprocessed speech text are detected and understood to obtain the fourth feature; Identify hidden expressions that evade detection in the preprocessed speech text to obtain the fifth feature; The first feature, the second feature, the third feature, the fourth feature, and the fifth feature are fused into a comprehensive feature representation through vector concatenation or attention mechanism to obtain a comprehensive violation risk characterization; The process of using a multi-task learning model to classify the main violation type, identify fine-grained labels, and calculate a confidence score reflecting the reliability of the prediction to obtain the prediction result includes: The comprehensive violation risk representation is used as input and fed into a multi-task learning model designed to handle multiple related tasks simultaneously. The multi-task learning model determines the type of violation to which the speech text belongs based on the comprehensive violation risk representation; The multi-task learning model identifies specific violations in the speech text based on the comprehensive violation risk representation; Based on the internal uncertainty, feature anomalies, and context-dependent factors of the multi-task learning model, a confidence score reflecting the reliability of the prediction and a severity level of the violation are calculated. The output includes review results, including severity level and confidence score, to obtain prediction results, wherein the prediction results also include the main violation type and fine-grained labels.

2. The method for handling inappropriate speech based on NLP according to claim 1, characterized in that, After classifying the main violation type and identifying fine-grained labels using a multi-task learning model to determine the comprehensive violation risk representation and calculating a confidence score reflecting the prediction reliability to obtain the prediction result, the method further includes: Based on the treatment strategy, the prediction results are mapped to different treatment actions, and the treatment actions are dynamically adjusted.

3. The method for handling inappropriate speech based on NLP according to claim 2, characterized in that, After mapping the prediction results to different handling actions and dynamically adjusting the handling actions, the method further includes: Collect the processing results obtained after performing the processing action to optimize the pre-trained language model, the multi-task learning model, and the processing strategy.

4. The method for handling inappropriate speech based on NLP according to claim 3, characterized in that, The disposal strategy maps the prediction results to different disposal actions and dynamically adjusts the disposal actions, including: The engine for the tiered treatment strategy is activated based on the prediction results; The engine determines whether to conduct a detailed interactive evaluation based on the prediction results to determine the final handling measures and thus the handling strategy; Select and implement specific actions based on the aforementioned handling strategy; The severity of the handling measures will be adjusted based on users' historical behavior and the breadth of content dissemination. The confidence threshold is adjusted in real time based on model performance feedback and current environmental factors to optimize response decisions.

5. The method for handling inappropriate speech based on NLP according to claim 4, characterized in that, The process of determining whether to conduct a detailed interactive evaluation based on the prediction results to determine the final action and action strategy includes: The engine performs non-linear interactive judgment to determine the handling measures by using a specific combination of confidence intervals and severity levels. The response to severity is adjusted according to the confidence level, with low confidence levels resulting in downgraded handling and high confidence levels resulting in escalated handling intensity. Corresponding handling standards are adopted for different types of violations to determine the handling strategy.

6. The method for handling inappropriate speech based on NLP according to claim 4, characterized in that, The step of collecting the processing results obtained after executing the processing action to optimize the pre-trained language model, the multi-task learning model, and the processing strategy includes: Record the results of each action and related information to obtain the outcome of the action; Manually review low-confidence or sensitive cases in the model, and clearly indicate the review results and reasons; The processed results are cleaned and desensitized, and then converted into a format suitable for machine learning processing. The manually reviewed data is used as new training samples and combined with the converted data to form an updated dataset. The pre-trained language model and multi-task learning model were retrained and optimized using the updated dataset; The rules within the engine are analyzed based on the processing results to adjust the processing strategy.

7. A device for handling inappropriate speech based on NLP, characterized in that, The device uses the NLP-based method for handling inappropriate speech as described in any one of claims 1 to 6, including: The acquisition unit is used to acquire the speech text and preprocess it to obtain the preprocessed speech text. The extraction and fusion unit is used to extract semantic, sentiment, intent, and risk entity multi-dimensional features from the pre-processed speech text using a pre-trained language model, and to fuse the multi-dimensional features to obtain a comprehensive violation risk representation. The prediction unit is used to classify the main violation type and identify fine-grained labels on the comprehensive violation risk representation through a multi-task learning model, and calculate a confidence score that reflects the reliability of the prediction to obtain the prediction result. The prediction result includes a confidence score and a severity level. The confidence score is used to characterize the degree of certainty and reliability of the multi-task learning model in determining the correctness of the current result.

8. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 6.

Images