Content detection method and device, equipment, medium and product

Abstract

The invention discloses a content detection method and device, equipment, a medium and a product, and is applied to the field of illegal content processing. The method comprises the steps that to-be-detected content and an identification rule strategy are acquired, the identification rule strategy comprises a first identification rule and a second identification rule, and the identification complexity of the second identification rule is higher than that of the first identification rule; taking the first identification rule as auxiliary screening, under the condition that the to-be-detected content is not hit by the first identification rule, predicting the to-be-detected content through an illegal content detection model, and outputting the confidence coefficient of the to-be-detected content; and under the condition that the confidence coefficient of the to-be-detected content is in the target confidence coefficient interval, taking the second recognition rule as auxiliary screening, and sequentially calling at least one type of lexicon through the illegal content detection model to determine that the to-be-detected content is the illegal content. According to the layered progressive detection method, the detection accuracy of the content to be detected can be improved.

Description

Technical Field

[0001] This application relates to the field of handling illegal content, and in particular to a content detection method, apparatus, equipment, medium, and product. Background Technology

[0002] To meet the needs of cyberspace governance and mitigate the social risks posed by the spread of harmful information, the detection of illegal content has become one of the core tasks in maintaining network content security.

[0003] In related technologies, content violation detection mainly relies on pre-set lexicon rules or large language models. Taking detection using pre-set lexicon rules as an example, it utilizes a sensitive word library and keyword matching rules to detect content violations. However, this method is inefficient and lacks accuracy, making it difficult to identify various complex and subtle variations of violations. Taking detection based on a large language model (LLM) as another example, LLM has strong generalization capabilities, utilizing its contextual semantic understanding and transfer learning abilities to handle complex and subtle variations of violations. However, LLM inference and output are susceptible to illusions, and the sheer size of LLM models results in high computational resource consumption, making them unsuitable for handling massive amounts of data.

[0004] Therefore, how to improve the accuracy of detecting illegal content while reducing the consumption of computing resources is an urgent problem to be solved. Summary of the Invention

[0005] This application provides a content detection method, apparatus, device, medium, and product. The technical solution is as follows: On the one hand, a content detection method is provided, the method comprising: The content to be detected and the recognition rule strategy are obtained. The recognition rule strategy includes a first recognition rule and a second recognition rule. The recognition complexity of the second recognition rule is higher than that of the first recognition rule. Using the first identification rule as an auxiliary filter, if the content to be detected is not hit by the first identification rule, the violation content detection model is used to predict the content to be detected and output the confidence level of the content to be detected. If the confidence level of the content to be detected is within the target confidence level range, the second identification rule is used as an auxiliary filter, and the violation content detection model sequentially calls at least one type of thesaurus to determine that the content to be detected is the violation content. The at least one type of thesaurus includes at least one of the following: a thesaurus of illegal content, a thesaurus of normal content with difficult examples, and a thesaurus of illegal content with difficult examples.

[0006] On the other hand, a content detection device is provided, the device comprising: The acquisition module is used to acquire the content to be detected and the recognition rule strategy. The recognition rule strategy includes a first recognition rule and a second recognition rule. The recognition complexity of the second recognition rule is higher than that of the first recognition rule. The prediction module is used to use the first identification rule as an auxiliary filter. When the content to be detected is not hit by the first identification rule, the module predicts the content to be detected through the violation content detection model and outputs the confidence level of the content to be detected. The determination module is used to determine the content to be detected as the illegal content by using the second identification rule as an auxiliary filter when the confidence level of the content to be detected is within the target confidence level range, and by sequentially calling at least one type of thesaurus through the illegal content detection model. The at least one type of thesaurus includes at least one of the following: a thesaurus of illegal content, a thesaurus of normal content with difficult examples, and a thesaurus of illegal content with difficult examples.

[0007] On the other hand, a computer device is provided, the computer device comprising: a processor and a memory, the memory storing a computer program, the computer program being loaded and executed by the processor to implement the content detection method as described above.

[0008] On the other hand, a computer-readable storage medium is provided that stores a computer program, which is loaded and executed by a processor to implement the content detection method described above.

[0009] On the other hand, a computer program product is provided, the computer program product including computer instructions stored in a computer-readable storage medium, wherein a processor obtains the computer instructions from the computer-readable storage medium, causing the processor to load and execute them to implement the content detection method as described above.

[0010] The beneficial effects of the technical solutions provided in this application include at least the following: The content detection method proposed in this application, after obtaining the content to be detected, uses an identification rule strategy as an auxiliary filter. The method sequentially calls at least one type of thesaurus through a violation content detection model, and combines the confidence level of the content to be detected to determine whether the content is violation content. The identification rule strategy includes a first identification rule and a second identification rule, with the second identification rule having higher identification complexity than the first. For example, the first identification rule can identify whether the content to be detected contains sensitive words, while the second identification rule can identify whether the content to be detected contains complex variations of sensitive words. Specifically, the first identification rule is first used to quickly filter the content to be detected. If the content to be detected is not matched by the first identification rule, it is determined to be suspected violation content (content to be confirmed as violation), and the content to be detected is input into the violation content detection model for prediction. If the confidence level predicted by the violation content detection model is within the target confidence level range, the second identification rule (such as performing complex variation word matching) is then activated. The second identification rule is used as an auxiliary filter, and the violation content detection model calls at least one type of thesaurus to cross-validate the content to be detected. The analysis results of the second identification rule and the matching results of at least one type of thesaurus are combined to determine whether the content to be detected is violation content. This layered and progressive detection method can improve the detection efficiency of the content to be detected and reduce the consumption of computing resources.

[0011] On the one hand, by using the first identification rule as a preliminary screening checkpoint, normal content without controversy can be quickly filtered out, allowing the violation content detection model to only process suspected violation content, which greatly reduces the computational load and running time of the violation content detection model and avoids the waste of resources when all content is fed into the model.

[0012] On the other hand, for suspected illegal content in areas with ambiguous confidence levels that the illegal content detection model cannot accurately determine, a second identification rule and cross-validation using multiple lexicons (at least one type) are introduced to solve the problem of inaccurate identification of complex variants (variant words, metaphorical words). The difficult example lexicon for normal content can filter out normal content that is easily misjudged (such as academic content containing sensitive words), while the difficult example lexicon for illegal content can identify variant illegal texts that evade detection (such as homophones or words with added interference characters), thus improving the detection accuracy of the content to be detected. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1This is a block diagram of a computer system provided in an exemplary embodiment; Figure 2 This is a schematic diagram of an exemplary embodiment of a detection architecture for illegal content. Figure 3 A flowchart of a content detection method provided in an exemplary embodiment; Figure 4 A flowchart of a content detection method provided in an exemplary embodiment; Figure 5 A flowchart of a content detection method provided in an exemplary embodiment; Figure 6 A flowchart of a content detection method provided in an exemplary embodiment; Figure 7 A block diagram of a content detection apparatus provided in an exemplary embodiment; Figure 8 This is a structural block diagram of a computer device provided in an exemplary embodiment. Detailed Implementation

[0015] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0016] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0017] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any and all possible combinations of one or more of the associated listed items.

[0018] It should be understood that although the terms first, second, etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, a first parameter may also be referred to as a second parameter, and similarly, a second parameter may also be referred to as a first parameter. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to determination."

[0019] It should be noted that this application may display prompt interfaces, pop-ups, or output voice prompts before and during the collection of user, user account, and image-related data. These prompt interfaces, pop-ups, or voice prompts are used to inform the user that their data is being collected. This ensures that the application only begins the steps for collecting user data after receiving confirmation from the user regarding the prompt interface or pop-up; otherwise (i.e., without user confirmation), the steps for collecting user data end, meaning no user data is collected. In other words, all user data collected in this application is collected with the user's consent and authorization, and the collection, use, and processing of related user data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0020] First, a brief introduction to the terms used in the embodiments of this application will be given.

[0021] The list of prohibited content terms consists of words that clearly violate regulations. Optionally, prohibited content includes, but is not limited to: vulgar or pornographic content, content that promotes illegal traffic, or other prohibited content.

[0022] Normal content difficult example vocabulary: This consists of normal words that are easily misidentified as illegal content. In this embodiment of the application, the normal content difficult example vocabulary can also be called the normal sample difficult example vocabulary.

[0023] Difficult-to-identify violation vocabulary: This consists of violation words that are easily missed in identifying as violation content.

[0024] Figure 1 This is a block diagram of a computer system provided as an exemplary embodiment. The computer system 100 can be implemented as a system architecture for a content detection method. The computer system 100 includes a terminal 120 and a server 140.

[0025] Terminal 120 can be an electronic device such as a mobile phone, tablet computer, vehicle terminal (vehicle system), wearable device, PC (Personal Computer), unmanned reservation terminal, smart home appliance, smart voice interaction device, unmanned vending terminal, etc. A client application for the target application can be installed and running on terminal 120. This target application can be a program for detecting illegal content, or other applications that provide illegal content detection functions; this application embodiment does not limit this. Furthermore, this application embodiment does not limit the form of the target application, including but not limited to Apps (Applications), mini-programs, etc., installed on terminal 120, and can also be in web page form.

[0026] Server 140 can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. Server 140 can be the backend server for the aforementioned target application, providing backend services to the client of the target application. In some optional embodiments, server 140 can also be implemented as a node in a blockchain system.

[0027] Terminal 120 and server 140 can communicate via a network, such as a wired or wireless network.

[0028] The content detection method provided in this application embodiment can be executed by a computer device, which refers to an electronic device with data computing, processing, and storage capabilities. Figure 1 Taking the implementation environment of the scheme shown as an example, the content detection method can be executed by the terminal 120. For example, the client of the target application installed and running in the terminal 120 can execute the content detection method, or the server 140 can execute the content detection method, or the terminal 120 and the server 140 can interact and cooperate to execute it. This application embodiment does not limit this.

[0029] Those skilled in the art will understand that the number of terminals 120 can be more or less. For example, there may be only one terminal 120, or there may be dozens or hundreds of terminals 120, or more. The embodiments of this application do not limit the number of terminals 120 or the type of device.

[0030] Content violation detection has become one of the core tasks in maintaining network content security. Among related technologies, content violation detection typically employs the following three methods: (1) Detection method based on thesaurus + policy rules: This method uses open-source sensitive word libraries for illegal content or sensitive word libraries for illegal content built based on specific business scenarios and combines them with relevant policy rules to detect illegal content text. This type of method has low technical implementation difficulty, low computational requirements, and is easy to deploy quickly. Moreover, the rules are transparent and clear, and it is easy to adjust and maintain them in a timely manner according to policy requirements. It is suitable for scenarios with high real-time requirements such as machine review and reduction. However, existing thesaurus and rule construction methods usually rely on manual compilation and rule matching. This type of method is inefficient, has narrow coverage, and is difficult to cope with constantly changing online language. In addition, illegal content often has variations, pinyin substitutions, word order inversions, etc. This type of method is difficult to cover comprehensively, and has insufficient flexibility, high false alarm rate, and difficulty in identifying various types of variant samples and obscure or implicit illegal information.

[0031] (2) Detection method based on small model: use deep learning models (such as BERT, RoBERTa and other language models) to segment and extract features of text content, and then perform binary classification to determine whether the text content belongs to illegal content. This type of method uses a small-scale deep learning model with low computational resource requirements, can be iterated quickly, and is suitable for scenarios with high real-time requirements such as machine review and reduction. It can filter through a high-confidence white-and-black strategy to reduce the number of review samples. However, due to the small model capacity, the accuracy will drop significantly when the confidence is low, but the recall will drop significantly when the confidence is high.

[0032] (3) Detection methods based on large language models (LLM): These methods utilize large-scale pre-trained language models, combined with strategies such as prompting engineering, parameter fine-tuning, and reinforcement learning, to endow the relevant large models with the ability to understand the complex needs of specific scenarios, detect illegal content, and then provide reasons for the judgment. Because large language models have strong contextual semantic understanding capabilities, they exhibit strong transfer learning capabilities and high generalization performance, making them suitable for combating illegal content data with high difficulty. However, due to the massive size and computational demands of large language models, they lead to high resource consumption, complex deployment, and inability to handle large amounts of data, making them unsuitable for review platforms with massive amounts of data.

[0033] Current solutions in related technologies primarily rely on natural language processing or rule-based lexicon-based content identification. However, most existing solutions suffer from simple rules, limited lexicons, and rudimentary strategies, making them ill-equipped to handle complex variations, lack of contextual understanding, and iterative delays. While large language models possess powerful contextual semantic understanding capabilities and demonstrate strong transfer learning abilities, their reasoning and output may produce illusions, generating inaccurate or erroneous responses. Furthermore, large language models are massive in size, computationally intensive, resource-intensive, and complex to deploy, making them unsuitable for large-scale data processing platforms with vast amounts of data.

[0034] Based on the problems existing in related technologies, this application provides a content detection method. Figure 2 This is a schematic diagram of an exemplary embodiment of a violation content detection architecture. Figure 2 The architecture shown can be derived from Figure 1 At least one of terminal 120 and server 140 is executed. The illegal content detection architecture provided in this application includes a first phase 10 and a second phase 20.

[0035] In the first stage 10, a dynamic data source 11 is obtained, and at least one type of word library is constructed by combining the dynamic data source 11 with an open-source illegal content word library 12. The at least one type of word library includes at least one of an illegal content word library 13, a normal content difficult example word library 14, and an illegal content difficult example word library 15. Among them, the illegal content word library 13 consists of illegal vocabulary with obvious illegal content. The normal content difficult example word library 14 consists of normal vocabulary that is easily misidentified as illegal content. The illegal content difficult example word library 15 consists of illegal vocabulary that is easily missed as illegal content. Optionally, the illegal content includes but is not limited to at least one of vulgar and pornographic content, traffic diversion content, or other illegal content. This application does not make a limitation in this regard.

[0036] For example, the illegal content word library 13 includes illegal vocabulary, and the illegal vocabulary is "brushing orders". The normal content difficult example word library 14 includes normal difficult example vocabulary, and the normal content difficult example vocabulary is "good product recommendation". The illegal content difficult example word library 15 includes illegal difficult example vocabulary, and the illegal content difficult example vocabulary is "shuadan" (a homophonic variant corresponding to the illegal content vocabulary "brushing orders").

[0037] Taking the at least one type of word library including the illegal content word library 13, the normal content difficult example word library 14, and the illegal content difficult example word library 15 as an example, an identification rule strategy is determined based on the illegal content word library 13, the normal content difficult example word library 14, and the illegal content difficult example word library 15. Optionally, the identification rule strategy includes a first identification rule and / or a second identification rule. Among them, the identification complexity of the second identification rule is higher than that of the first identification rule. For example, the first identification rule identifies illegal content A, and the second identification rule identifies a complex variant of illegal content A.

[0038] After obtaining the illegal content word library 13, the illegal content word library 13 is optimized. Optionally, each vocabulary in the illegal content word library 13 is expanded in at least one dimension by a large language model to obtain an expanded illegal content word library. At least one candidate vocabulary is collected from the dynamic data source 11 every preset time period, and the at least one candidate vocabulary is added to the expanded illegal content word library to obtain an optimized illegal content word library 16. The optimized illegal content word library 16 is a dynamically iteratively updated word library. The above at least one dimension includes but is not limited to at least one of a synonym dimension; a pinyin dimension; a word order reversal dimension; a word order recombination dimension; a homophone replacement dimension; an emoji expansion dimension; an interference character insertion dimension; a context expansion dimension. This application does not make a limitation in this regard.

[0039] In the second stage 20, an illegal content recall model 21 is constructed based on the optimized illegal content lexicon 16, the normal content difficult example lexicon 14, and the identification rule strategy. Optionally, the first identification rule is used as an auxiliary filter. The illegal content recall model 21, combined with the optimized illegal content lexicon 16 and the normal content difficult example lexicon 14, filters the dynamic data source 11 to obtain an enhanced data source 17. The data concentration of illegal content in the enhanced data source 17 is higher than that in the dynamic data source 11. The enhanced data source 17 is labeled to obtain enhanced training samples 22 for training the illegal content detection model 23. The illegal content detection model 23 is iteratively trained using the enhanced training samples 22.

[0040] In the second stage 20, the identification rule strategy is used as an auxiliary screening method. The content to be detected 24 is predicted by the violation content detection model 23 in combination with at least one type of lexicon. The content to be detected 24 is determined to be violation content based on the confidence level output by the violation content detection model 23.

[0041] Optionally, the first identification rule is used as an auxiliary filter. If the content to be detected 24 is not matched by the first identification rule, the violation content detection model 23 predicts the content to be detected 24 and outputs the confidence level of the content to be detected 24. If the confidence level of the content to be detected 24 is within the target confidence level range, the second identification rule is used as an auxiliary filter. The violation content detection model sequentially calls at least one of the violation content lexicon 13, the normal content difficult example lexicon 14, and the violation content difficult example lexicon 15 to determine that the content to be detected 24 is violation content.

[0042] In this embodiment, a dual-lexicon chain filtering method is used to filter dynamic data sources through a constructed lexicon of illegal content and a lexicon of difficult examples of normal content. This is combined with identification rule strategies to assist in the filtering process, resulting in an enhanced data source. The illegal content lexicon ensures the recall of illegal content, while the difficult example lexicon of normal content improves the accuracy of the recalled illegal content. The enhanced data source recalled by the illegal content recall model has a high concentration of illegal content. Labeling the enhanced data source significantly reduces labeling costs. By training the illegal content detection model with the enhanced training samples obtained after labeling the enhanced data source, and then combining at least one type of lexicon and identification rule strategies to identify the content to be detected, the predictive accuracy of the illegal content detection model can be greatly improved.

[0043] The following section provides a detailed introduction to content detection methods.

[0044] Figure 3 This is a flowchart illustrating a content detection method provided in an exemplary embodiment. The method is executed by a computer device, which may be... Figure 1The terminal 120 and / or server 140. The computer device stores a violation content detection model, or has the ability to invoke a violation content detection model, which refers to a trained detection model. The method includes at least some of steps 220, 240, and 260.

[0045] Step 220: Obtain the content to be detected and the recognition rule strategy, which includes the first recognition rule and the second recognition rule.

[0046] The content to be detected is the content used to identify whether it is illegal.

[0047] In some embodiments, the content to be detected includes text content; or, the content to be detected includes string content; or, the content to be detected includes link content; or, the content to be detected includes multilingual mixed text; or, the content to be detected includes emoji content; or, the content to be detected includes multimodal fusion content. The above are merely illustrative examples, and this application does not limit the specific form of the content to be detected.

[0048] The identification rule strategy refers to the judgment rules used to filter non-violation content (normal content). The identification rule strategy is used to quickly filter uncontroversial normal content.

[0049] In some embodiments, the identification rule strategy includes a first identification rule and a second identification rule. Optionally, the second identification rule has a higher identification complexity than the first identification rule. The first identification rule is a basic rule with low identification complexity, and the second identification rule is an advanced rule with high identification complexity. Identification complexity indicates the difficulty of determining the identification rule.

[0050] In some embodiments, recognition complexity is represented by logical complexity; or, recognition complexity is represented by computational complexity. Logical complexity indicates whether the first recognition rule / second recognition rule needs to be combined with context or variant pattern matching. Computational complexity indicates the resource and time costs required to execute the first recognition rule / second recognition rule.

[0051] In some embodiments, the identification rule strategy is constructed based on a dynamic data source. Optionally, a dynamic data source is obtained, at least one type of lexicon is constructed based on the dynamic data source, and an identification rule strategy is determined based on the at least one type of lexicon.

[0052] For example, the first identification rule does not require contextual or variant pattern matching, while the second identification rule does require contextual or variant pattern (variant vocabulary or metaphorical vocabulary of the sensitive word) matching. Optionally, the resource and time costs required to execute the first identification rule are lower than those required for the second identification rule.

[0053] The first identification rule described above can also be called the basic rule or the simple rule. The second identification rule described above can also be called the complex rule or the advanced rule.

[0054] In some embodiments, the first identification rule / second identification rule includes, but is not limited to: sensitive word negation rule, semantic matching rule, and context exclusion rule. This application does not limit this.

[0055] For example, the first identification rule / second identification rule includes a sensitive word negation rule. If the text content in the dynamic data source does not contain sensitive words, the text content is determined to be normal content. For instance, if the text content in the dynamic data source is "The weather is nice today, let's go hiking together," this text content does not contain any sensitive words and is therefore determined to be normal content.

[0056] For example, the first identification rule / second identification rule includes a semantic matching rule. The text content in the dynamic data source is semantically matched against sensitive words (or variations of sensitive words). If a semantic match fails, the text content is determined to be normal content.

[0057] For example, the first / second identification rule includes a context exclusion rule. If the text content in the dynamic data source contains explicit contextual markers (normal domain markers), the text content is determined to be normal content. For instance, if the text content in the dynamic data source is "Medical Science Popularization: The physiological structure of the chest is composed of multiple glands," where "chest" is a sensitive word and violates regulations, but this is normal medical content, then the text content is determined to be normal content.

[0058] Step 240: Using the first identification rule as an auxiliary filter, if the content to be detected is not matched by the first identification rule, the violation content detection model is used to predict the content to be detected and output the confidence level of the content to be detected.

[0059] The first identification rule is a basic rule with low identification complexity. It is used to filter non-violation content (normal content) using a low-complexity judgment rule. The first identification rule can also be called a basic rule or a simple rule. Auxiliary filtering refers to a rule used to filter non-violation content or narrow the detection range. For example, the first identification rule can be used as an auxiliary filter to filter non-violation content in the initial stage of the content to be detected.

[0060] In some embodiments, the first identification rule includes, but is not limited to: a first sensitive word negation rule, a first semantic matching rule, and a first context exclusion rule. This application does not limit this.

[0061] For example, the first identification rule includes a first sensitive word negation principle, where the first sensitive word is a fixed sensitive word. For example, the first sensitive word is "pornography". For example, the first identification rule includes a first semantic matching rule, where the first semantic is used to indicate the semantics that directly match the first sensitive word. For example, the first identification rule includes a first context exclusion rule, where the first context is used to indicate the current context without needing to consider the surrounding context.

[0062] In some embodiments, the first identification rule is used as an auxiliary filter to determine whether the content to be detected is matched by the first identification rule. If the content to be detected is not matched by the first identification rule, it is determined that the content to be detected is suspected violation content (or violation content to be determined, or candidate violation content). The content to be detected (in this case, suspected violation content) is input into the violation content detection model, which predicts the content to be detected and outputs the confidence level of the content to be detected.

[0063] The confidence score indicates the probability that the detected content is illegal. The confidence score ranges from 0 to 1. A higher confidence score indicates a higher probability that the illegal content detection model will determine that the detected content is illegal; a lower confidence score indicates a lower probability that the illegal content detection model will determine that the detected content is illegal.

[0064] For example, a violation detection model is a Bidirectional Encoder Representations from Transformers (BERT) model. The BERT model uses bidirectional encoded representations of the content to be detected by a transformer.

[0065] It should be noted that the above-mentioned content violation detection model is a fully trained model. A content violation detection model can also be referred to as a fully trained content violation detection model. The training process of the content violation detection model will be described in the following examples.

[0066] Step 260: When the confidence level of the content to be detected is within the target confidence level range, the second identification rule is used as an auxiliary screening method, and the violation content detection model sequentially calls at least one type of thesaurus to determine that the content to be detected is violation content.

[0067] In some embodiments, the content to be detected (in this case, suspected illegal content) is input into the illegal content detection model, which then predicts the content and outputs its confidence level. If the confidence level of the content to be detected falls within the target confidence level range, a second identification rule is used as an auxiliary filter, and the illegal content detection model sequentially calls at least one type of thesaurus to determine whether the content to be detected is illegal.

[0068] The target confidence interval is used to indicate the target probability interval for determining whether the content to be detected is illegal. The target confidence interval includes a first boundary value and a second boundary value. The first boundary value is the maximum value of the target confidence interval, and the second boundary value is the minimum value. When the confidence level of the content to be detected falls within the target confidence interval, it indicates that the prediction result of the illegal content detection model is ambiguous, and the content to be detected is suspected of being illegal, but it cannot be directly determined whether it is illegal.

[0069] The second identification rule is an advanced rule with high identification complexity. It is used to filter non-violation content (normal content) using a more complex judgment rule. The second identification rule can also be called a complex rule or an advanced rule. Auxiliary filtering refers to a rule used to filter non-violation content or narrow the detection range. For example, the second identification rule can be used as an auxiliary filter to narrow the detection range at an intermediate stage of the content to be detected.

[0070] In some embodiments, the second identification rule includes, but is not limited to: a second sensitive word negation rule, a second semantic matching rule, and a second context exclusion rule. This application does not limit this.

[0071] For example, the second identification rule includes a second sensitive word negation principle. The second sensitive word is a variant sensitive word or a metaphorical sensitive word, and the identification complexity of the second sensitive word is higher than that of the first sensitive word. For example, the first sensitive word is "pornography," and the second sensitive words are "eroticism" or "seqing."

[0072] For example, the second identification rule includes a second semantic matching rule, whereby the second semantic is used to indicate the semantics that directly or indirectly match the second sensitive word (a variant of the first sensitive word).

[0073] For example, the second identification rule includes a second context exclusion rule, where the second context is used to indicate the preceding and following contexts that need to be considered. For instance, the second context is "This book explores the social impact of the pornography industry," which, although containing the word "pornography," is considered to be within the context of an academic discussion.

[0074] In some embodiments, at least one type of thesaurus includes at least one of a violation content thesaurus, a normal content difficult example thesaurus, and a violation content difficult example thesaurus. Optionally, when the confidence level of the content to be detected is within the target confidence level range, a second identification rule is used as an auxiliary screening method, and the violation content detection model sequentially calls at least one of the violation content thesaurus, the normal content difficult example thesaurus, and the violation content difficult example thesaurus to determine that the content to be detected is violation content.

[0075] Among them, the illegal content word library consists of illegal words with obvious illegal content. The normal content difficult example word library consists of normal words that are easily misidentified as illegal content. The illegal content difficult example word library consists of illegal words that are easily missed as illegal content.

[0076] Exemplarily, the illegal content word library includes illegal words, and the illegal word is "shua dan" (false transaction), and "shua dan" is illegal content. The normal content difficult example word library includes normal difficult example words, and the normal content difficult example word is "good product recommendation", and "good product recommendation" is a real sharing by users and is easily misjudged as illegal content of the drainage type. The illegal content difficult example word library includes illegal difficult example words, and the illegal content difficult example word is "shua dan" (a homophone variant corresponding to the illegal content word "shua dan"), and "shua dan" is easily missed as illegal content.

[0077] In some embodiments, in the case where the content to be detected is determined to be illegal content, a blacklisting operation is performed on the content to be detected. Among them, the blacklisting operation is the final disposal operation for the content to be detected.

[0078] Optionally, the blacklisting operation includes one or more of the following operations: direct interception, content shielding, illegal marking (illegal content marking, or illegal content difficult example marking), and associated disposal. Among them, direct interception means refusing to publish illegal content (such as a post cannot be published). Content shielding means hiding the published illegal content, and other accounts cannot view it. Illegal marking means marking the illegal content and incorporating it into the illegal content library for subsequent iterative optimization of the illegal content detection model. Associated disposal means warning and traffic limiting the account that publishes the illegal content. This application does not make a limitation on this.

[0079] It should be noted that the embodiments of this application are executed by a computer device, and the computer device stores an illegal content detection model, or has the ability to call an illegal content detection model. In some embodiments, the computer device may be a server, that is, the server stores an illegal content detection model, or the server has the ability to call an illegal content detection model. In some other embodiments, the embodiments of this application may also be executed by an illegal content detection model in local deployment (local terminal, or an application program running in the local terminal), or local deployment (local terminal, or an application program running in the local terminal) has the ability to call an illegal content detection model.

[0080] In summary, the content detection method proposed in this application, after obtaining the content to be detected, uses an identification rule strategy as an auxiliary screening method. It sequentially calls at least one type of thesaurus through a violation content detection model and combines the confidence level of the content to be detected to determine whether the content is violation content. The identification rule strategy includes a first identification rule and a second identification rule, with the second identification rule having higher identification complexity than the first. For example, the first identification rule can identify whether the content to be detected contains sensitive words, while the second identification rule can identify whether the content to be detected contains complex variations of sensitive words. Specifically, the content to be detected is first quickly filtered using the first identification rule. If the content to be detected is not matched by the first identification rule, it is determined to be suspected violation content (content to be confirmed as violation content), and then input into the violation content detection model for prediction. If the confidence level predicted by the violation detection model falls within the target confidence level range, a second identification rule (such as complex variant word matching) is then activated. This second identification rule serves as an auxiliary filter. The violation detection model calls at least one type of thesaurus to cross-validate the content to be detected. Combining the analysis results of the second identification rule with the matching results of at least one type of thesaurus, it is determined whether the content to be detected is violation content. This hierarchical and progressive detection method can improve the detection efficiency of the content and reduce the consumption of computational resources.

[0081] On the one hand, by using the first identification rule as a preliminary screening checkpoint, normal content without controversy can be quickly filtered out, allowing the violation content detection model to only process suspected violation content, which greatly reduces the computational load and running time of the violation content detection model and avoids the waste of resources when all content is fed into the model.

[0082] On the other hand, for suspected illegal content in areas with ambiguous confidence levels that the illegal content detection model cannot accurately determine, a second identification rule and cross-validation using multiple lexicons (at least one type) are introduced to solve the problem of inaccurate identification of complex variants (variant words, metaphorical words). The difficult example lexicon for normal content can filter out normal content that is easily misjudged (such as academic content containing sensitive words), while the difficult example lexicon for illegal content can identify variant illegal texts that evade detection (such as homophones or words with added interference characters), thus improving the detection accuracy of the content to be detected.

[0083] Determine the identification rule strategy and at least one type of lexicon In some embodiments, the identification rule strategy is built based on a dynamic data source.

[0084] Figure 4 This is a flowchart of a content detection method provided by an exemplary embodiment. Step 210 is included before step 220.

[0085] Step 210: Obtain dynamic data sources, construct at least one type of lexicon based on the dynamic data sources, and determine recognition rule strategies based on at least one type of lexicon.

[0086] The dynamic data source is the full dataset within the target business scenario. It is used to construct at least one type of thesaurus. Optionally, the target business scenario includes, but is not limited to, at least one of the following: communication scenarios, live streaming with bullet comments, channel scenarios, and community / forum scenarios. This application does not limit this. The aforementioned dynamic data source can also be referred to as dashboard data (dashboard data source).

[0087] In some embodiments, at least one type of thesaurus is constructed based on a dynamic data source. Optionally, the constructed at least one type of thesaurus includes at least one of a thesaurus of prohibited content, a thesaurus of normal content with difficult examples, and a thesaurus of prohibited content with difficult examples. This application does not limit this.

[0088] The aforementioned list of prohibited content terms consists of words containing clearly offensive or offensive content. The list of normal content terms (difficult examples) consists of normal words that are easily misidentified as prohibited content. The list of prohibited content terms (difficult examples) consists of prohibited words that are easily missed in identification as prohibited content.

[0089] Optionally, the prohibited content includes, but is not limited to, at least one of the following: vulgar or pornographic content, content that redirects traffic, or other prohibited content. This application does not limit this.

[0090] In some embodiments, a dynamic data source is obtained, and at least one type of thesaurus is constructed based on the dynamic data source and an open-source thesaurus of prohibited content. The dynamic data source is matched based on the open-source thesaurus of prohibited content, and at least one type of thesaurus is constructed based on the matching results (or hit results).

[0091] In some embodiments, a violation content thesaurus is constructed based on the violation content detected in the dynamic data source. Optionally, the open-source violation content thesaurus is matched against the dynamic data source. If the content in the dynamic data source successfully matches the content in the open-source violation content thesaurus, and the successfully matched content (i.e., the thesaurus matching result) is consistent with the actual attributes of the content (in which case it is a correct match), the successfully matched content is identified as violation content and classified into the violation content thesaurus.

[0092] In some embodiments, a normal content difficult example thesaurus is constructed based on the falsely invoked violation content in the dynamic data source. Optionally, the open-source violation content thesaurus is matched against the dynamic data source. If the content in the dynamic data source successfully matches the content in the open-source violation content thesaurus, and the successfully matched content (i.e., the thesaurus matching result) contradicts the actual attributes of the content (in this case, it is an incorrect match), the matched content is confirmed as a normal content difficult example and added to the normal content difficult example thesaurus.

[0093] In some embodiments, a difficult example dictionary of illegal content is constructed based on the illegal content missed in the dynamic data source. Optionally, the open-source illegal content dictionary is matched against the dynamic data source. If the content in the dynamic data source and the content in the open-source illegal content dictionary do not match successfully (i.e., the match fails), and the unmatched content (i.e., the dictionary matching result) contradicts the actual attributes of the content (in this case, a match omission), the matched content is identified as a difficult example of illegal content and added to the difficult example dictionary of illegal content.

[0094] In some embodiments, an identification rule strategy is determined based on at least one type of thesaurus. Here, the identification rule strategy refers to the judgment rules used to filter non-violation content (normal content). The identification rule strategy is used to quickly filter uncontroversial normal content.

[0095] In some embodiments, the identification rule strategy includes a first identification rule and / or a second identification rule. Optionally, the second identification rule has a higher identification complexity than the first identification rule.

[0096] For details on the identification rules and strategies, please refer to the relevant description in step 220 above, which will not be repeated here.

[0097] Determine whether the content to be detected is illegal. • The content to be detected was determined to be illegal. In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the second identification rule is used as an auxiliary screening method, and the violation content detection model sequentially calls at least one type of thesaurus to determine that the content to be detected is violation content. The at least one type of thesaurus includes at least one of a violation content thesaurus, a normal content difficult example thesaurus, and a violation content difficult example thesaurus.

[0098] Step 260 above can be replaced by step 262, or by step 264.

[0099] Step 262: If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon in sequence through the violation content detection model, the second identification rule does not hit the content to be detected, and the normal content difficult example lexicon does not hit the content to be detected, then the content to be detected is determined to be violation content.

[0100] The target confidence interval is used to indicate the target probability interval for determining that the content to be detected is illegal. For example, the target confidence interval is [0.7, 0.95], where 0.95 is the first boundary value and 0.7 is the second boundary value.

[0101] In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the second identification rule is used as an auxiliary filter, and the violation content detection model sequentially calls at least one of the violation content lexicon, the normal content difficult example lexicon, and the violation content difficult example lexicon to filter (or screen) the content to be detected.

[0102] In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the violation content detection model first calls a violation content lexicon to filter the content to be detected. The violation content lexicon consists of violation words with obvious violation content. If the violation content lexicon matches the content to be detected, that is, if the violation content lexicon matches a word in the content to be detected, it is confirmed that the content to be detected has a first violation possibility (the content to be detected has basic violation characteristics). The violation content detection model then calls a second identification rule to filter the content to be detected. The second identification rule is used to filter non-violation content through a judgment rule with high recognition complexity (advanced rules, such as variant sensitive word matching, context semantic association). If the second identification rule does not match the content to be detected, it is confirmed that the content to be detected has a second violation possibility (the violation suspicion of the content to be detected cannot be eliminated by the complex second identification rule). The violation detection model continues to call the normal content difficult example dictionary to filter the content to be detected. The normal content difficult example dictionary consists of normal words that are easily misidentified as violation content. If the normal content difficult example dictionary does not match the content to be detected, the content to be detected is determined to be violation content (the content to be detected does not belong to the "normal content difficult example that is easily misjudged", and there is no situation where it is mistakenly marked as violation, so it is confirmed as violation content).

[0103] Step 264: If the confidence level of the content to be detected is within the target confidence level range, and the violation content detection model sequentially calls the violation content thesaurus to hit the content to be detected, the second identification rule does not hit the content to be detected, the normal content difficult example thesaurus hits the content to be detected, and the violation content difficult example thesaurus hits the content to be detected, then the content to be detected is determined to be violation content.

[0104] The target confidence interval is used to indicate the target probability interval for determining that the content to be detected is illegal content.

[0105] In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the violation content detection model first calls a violation content lexicon to filter the content to be detected. The violation content lexicon consists of violation words with obvious violation content. If the violation content lexicon matches the content to be detected, that is, if the violation content lexicon matches a word in the content to be detected, it is confirmed that the content to be detected has a first violation possibility (the content to be detected has basic violation characteristics). The violation content detection model then calls a second identification rule to filter the content to be detected. The second identification rule is used to filter non-violation content through a judgment rule with high recognition complexity (advanced rules, such as variant sensitive word matching, context semantic association). If the second identification rule does not match the content to be detected, it is confirmed that the content to be detected has a second violation possibility (the violation suspicion of the content to be detected cannot be eliminated by the complex second identification rule). The violation detection model continues to filter the content to be detected by calling the normal content difficult example dictionary. The normal content difficult example dictionary consists of normal words that are easily misidentified as violation content. If the normal content difficult example dictionary matches the content to be detected, it is confirmed that the content to be detected has a third possibility of violation (the content to be detected belongs to "normal difficult examples that are easily misjudged," resulting in a judgment contradiction, and further judgment is required). The violation content detection model continues to filter the content to be detected by calling the violation content difficult example dictionary. The violation content difficult example dictionary consists of violation words that are easily missed as violation content. If the violation content difficult example dictionary matches the content to be detected, it is determined that the content to be detected is violation content.

[0106] In this embodiment, for the content to be detected within the fuzzy confidence interval (target confidence interval), where the violation content detection model cannot directly and accurately determine the suspected violation content, the detection accuracy is achieved through a multi-condition combination verification using a violation content lexicon, a second identification rule, a normal content difficult example lexicon, and a violation content difficult example lexicon. This approach adapts to suspected violation scenarios without contradictory features (as described in step 262 above) and complex violation scenarios with contradictory features (as described in step 264 above), significantly reducing computational resource consumption while maintaining accuracy. Furthermore, the aforementioned lexicons can be dynamically updated, quickly adapting to new types of online slang and violating rules, thus avoiding detection capability lag.

[0107] In some embodiments, if the confidence level of the content to be detected is greater than a first boundary value of the target confidence level interval, the content to be detected is determined to be illegal content.

[0108] The first boundary value is the maximum value of the target confidence interval. For example, if the target confidence interval is [0.7, 0.95], then 0.95 is the first boundary value.

[0109] In some embodiments, if the confidence level of the content to be detected is greater than the first boundary value of the target confidence level interval, the violation content detection model determines that the content to be detected is violation content with a very high probability. At this time, there is no need to start the second recognition rule or call various thesaurus for cross-validation.

[0110] In some embodiments, during the prediction of the content to be detected by the violation detection model, falsely recalled and missed content to be detected are obtained; at least one type of lexicon is updated based on the falsely recalled and missed content to be detected, that is, the normal content difficult example lexicon is updated based on the falsely recalled content to be detected, and the violation content difficult example lexicon is updated based on the missed content to be detected. Optionally, relevant rules are extracted from the falsely recalled and missed content to be detected, and the extracted relevant rules are added to the second identification rules. Here, the relevant rules refer to executable judgment logic extracted from the falsely recalled and missed content to be detected.

[0111] For example, the features of falsely recalled and missed content to be detected are analyzed and transformed into executable relevant rules. For instance, the relevant rule extracted from the falsely recalled content "Medical Science Popularization: Reproductive System Anatomy" is: if the text contains the sensitive word "reproduction" and is prefixed with "medical science popularization," it is determined to be normal content. As another example, the relevant rule extracted from the missed content "Get V for Late-Night Benefits" is: if the text contains the combined features of "V" + "Late-Night Benefits," it is determined to be illegal content. These newly extracted relevant rules are integrated into the second identification rule (i.e., complex rule, advanced rule) of the detection process for filtering new content to be detected within the target confidence range in the next iteration.

[0112] In this embodiment, by learning from erroneous samples (falsely recalled and missed detected content), the same type of error is avoided from recurring, while the reliance on the prediction of the violation content detection model is reduced, thereby improving the stability and efficiency of boundary scene determination.

[0113] • Confirm that the content to be detected is non-violation content. In some embodiments, the content to be detected is either illegal or non-illegal, that is, the content to be detected is normal.

[0114] In some embodiments, if the content to be detected is determined to be non-violation content, a release operation (also known as a whitelisting operation) is performed on the content to be detected. The release operation is the final processing operation for the content to be detected. Optionally, the release operation includes one or more of the following operations: allow publication, no additional marking, difficult example marking (normal content difficult example marking), and fast approval. This application does not limit this.

[0115] In some embodiments, the determination that the content to be detected is non-violation content includes at least one of the following situations: Case 1 In some embodiments, the first identification rule is used as an auxiliary filter, and if the content to be detected is matched by the first identification rule, the content to be detected is determined to be non-violation content.

[0116] The first identification rule is a basic rule with low identification complexity. It is used to filter non-violation content (normal content) using a low-complexity judgment rule. The first identification rule can also be called a basic rule or a simple rule.

[0117] In some embodiments, the first identification rule is used as an auxiliary filter to determine whether the content to be detected is matched by the first identification rule. If the content to be detected is matched by the first identification rule, the content to be detected is determined to be non-violation content.

[0118] Case 2 In some embodiments, if the confidence level of the content to be detected is less than the second boundary value of the target confidence level interval, the content to be detected is determined to be non-violation content.

[0119] The second boundary value is the minimum value of the target confidence interval. For example, if the target confidence interval is [0.7, 0.95], then 0.7 is the second boundary value.

[0120] In some embodiments, if the confidence level of the content to be detected is less than the second boundary value of the target confidence level interval, the violation content detection model determines the content to be detected as violation content with a very low probability, that is, determines the content to be detected as non-violation content.

[0121] Case 3 In some embodiments, if the confidence level of the content to be detected is within the target confidence level range and the content to be detected is not matched by calling the violation content thesaurus through the violation content detection model, the content to be detected is determined to be non-violation content.

[0122] In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the violation content detection model sequentially calls at least one of the violation content thesaurus, the normal content difficult example thesaurus, and the violation content difficult example thesaurus to filter (or screen) the content to be detected.

[0123] In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the content to be detected is first filtered by calling the violation content lexicon through the violation content detection model. The violation content lexicon consists of violation words with obvious violation content. If the violation content lexicon does not match the content to be detected, that is, if the violation content lexicon does not match any words in the content to be detected, it is confirmed that the content to be detected is not violation content (the content to be detected does not have basic violation characteristics).

[0124] Case 4 In some embodiments, if the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by calling the violation content lexicon through the violation content detection model, and the second identification rule hits the content to be detected, then the content to be detected is determined to be non-violation content.

[0125] In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the violation content detection model first calls a violation content lexicon to filter the content to be detected. The violation content lexicon consists of violation words with obvious violation content. If the violation content lexicon matches the content to be detected, that is, if the violation content lexicon matches a word in the content to be detected, it is confirmed that the content to be detected has a first violation probability (the content to be detected has basic violation characteristics). The violation content detection model then calls a second identification rule to filter the content to be detected. The second identification rule is used to filter non-violation content through a judgment rule with high recognition complexity (advanced rules, such as variant sensitive word matching, context semantic association). If the second identification rule matches the content to be detected, it is confirmed that the content to be detected is non-violation content.

[0126] For example, the content to be detected is "Academic paper: On the impact of the pornography industry on society", with a confidence level of 0.85. The violation word database matches "pornography". The academic context association rule of the second identification rule is activated, and the context of "academic paper" is matched, so the content to be detected is determined to be non-violation content.

[0127] Case 5 In some embodiments, if the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon called by the violation content detection model, the content to be detected is not hit by the second identification rule, the content to be detected is hit by the normal content difficult example lexicon, and the content to be detected is not hit by the violation content difficult example lexicon, then the content to be detected is determined to be non-violation content.

[0128] In some embodiments, when the confidence level of the content to be detected is within the target confidence level range, the violation content detection model first calls a violation content lexicon to filter the content to be detected. The violation content lexicon consists of violation words with obvious violation content. If the violation content lexicon matches the content to be detected, that is, if the violation content lexicon matches a word in the content to be detected, it is confirmed that the content to be detected has a first violation possibility (the content to be detected has basic violation characteristics). The violation content detection model then calls a second identification rule to filter the content to be detected. The second identification rule is used to filter non-violation content through a judgment rule with high recognition complexity (advanced rules, such as variant sensitive word matching, context semantic association). If the second identification rule does not match the content to be detected, it is confirmed that the content to be detected has a second violation possibility (the violation suspicion of the content to be detected cannot be eliminated by the complex second identification rule). The violation detection model continues to filter the content to be detected by calling the normal content difficult example dictionary. The normal content difficult example dictionary consists of normal words that are easily misidentified as violation content. If the normal content difficult example dictionary matches the content to be detected, it is confirmed that the content to be detected has a third possibility of violation (the content to be detected belongs to "normal difficult examples that are easily misjudged," resulting in a judgment contradiction, and further judgment is required). The violation detection model continues to filter the content to be detected by calling the violation content difficult example dictionary. The violation content difficult example dictionary consists of violation words that are easily missed as violation content. If the violation content difficult example dictionary does not match the content to be detected, it is determined that the content to be detected is non-violation content.

[0129] Training a violation detection model • Train the violation detection model by enhancing training samples In some embodiments, the violation detection model is trained by augmenting training samples.

[0130] In some embodiments, augmented training samples are obtained and input into a general violation content detection model (also known as a violation content detection model to be trained). The predicted results output by the general violation content detection model are obtained. Based on the predicted results and the ground truth labels, a sample loss is obtained. The model parameters of the general violation content detection model are trained using the sample loss to obtain the violation content detection model. The augmented training samples are labeled with ground truth labels, which indicate whether the augmented training samples are violation content.

[0131] For example, if the real label in the augmented training sample is the first type label "0", it indicates that the augmented training sample is non-violation content (normal content); if the real label in the augmented training sample is the second type label "1", it indicates that the augmented training sample is violation content.

[0132] For example, if the real label in the augmented training sample is a first-class label "0", it indicates that the augmented training sample is illegal content; if the real label in the augmented training sample is a second-class label "1", it indicates that the augmented training sample is non-illegal content (normal content). This application does not limit this.

[0133] In some embodiments, augmented training samples are input into a general violation detection model to obtain the prediction results output by the general violation detection model. These prediction results indicate the predicted labels for the augmented training samples. Optionally, a sample loss (cross-entropy loss) is obtained based on the true labels of the augmented training samples and the prediction results. This sample loss indicates the error between the prediction results output by the general violation detection model and the true labels of the augmented training samples. The sample loss is used as the loss value during training, and the general violation detection model is trained based on this loss value to obtain the violation content detection model, i.e., the trained violation content detection model.

[0134] In this embodiment, in the scenario of detecting prohibited content, the number of high-quality labeled samples (especially variant prohibited samples and normal difficult examples) is often small. Augmenting the training samples can quickly expand the sample size. By simulating various forms of circumvention of prohibited content (such as adding interfering characters and homophonic variations), the prohibited content detection model learns more comprehensive prohibited features, avoiding the model only recognizing standard prohibited words while missing variant prohibited content and misjudging normal difficult examples. This training method greatly improves the prediction accuracy of the prohibited content detection model.

[0135] • Enhanced training samples were obtained through the violation content recall model. In some embodiments, the enhanced training samples are obtained through a violation content recall model.

[0136] In some embodiments, the first identification rule is used as an auxiliary filter. The dynamic data source is filtered using a violation content recall model combined with an optimized violation content lexicon and a normal content difficult example lexicon to obtain an enhanced data source. The enhanced data source is then labeled to obtain enhanced training samples for training the violation content detection model. Optionally, the data concentration of violation content in the enhanced data source is higher than that in the dynamic data source. Data concentration indicates the proportion of violation content data to the total amount of data in the dataset (here, the dynamic data source or the enhanced data source).

[0137] The first identification rule is a basic rule with low identification complexity. It is used to filter non-violation content (normal content) using a low-complexity judgment rule. The optimized violation content lexicon refers to the optimized lexicon obtained by expanding and supplementing the existing violation content lexicon. The enhanced data source refers to a data source with a higher concentration of violation content after filtering dynamic data sources. Labeling processing refers to the manual or automated tagging operation performed on each piece of data in the enhanced data source.

[0138] For example, after filtering dynamic data sources, a large amount of uncontroversial normal content is removed, while high-probability violations and easily misjudged difficult-to-identify content are retained, increasing the concentration of violation content in the dynamic data source to be higher than that in the dynamic data source. For instance, if a dynamic data source contains 10 million data entries, of which 10,000 to 20,000 are violation entries, the violation content concentration in the dynamic data source is approximately 0.1% to 0.2%. After filtering using a high-recall model for violation content, 50,000 data entries are selected from the 10 million data entries, of which 10,000 to 20,000 are violation entries, increasing the violation content concentration in the dynamic data source to approximately 20% to 40%.

[0139] In some embodiments, the specific method for obtaining the enhanced data source is as follows: the dynamic data source is filtered by calling the first identification rule through the violation content recall model, and the first part of the data source that is not matched by the first identification rule is retained; the first part of the data source is filtered by calling the optimized violation content lexicon through the violation content recall model, and the second part of the data source that is not matched by the optimized violation content lexicon is retained; the second part of the data source is filtered by calling the normal content difficult example lexicon through the violation content recall model, and the third part of the data source that is not matched by the normal content difficult example lexicon is retained, and the third part of the data source is used as the enhanced data source.

[0140] The first data source refers to the dataset retained after filtering by the first recognition rule in the dynamic data source. The second data source refers to the dataset retained after filtering by the optimized illegal content lexicon in the first data source. The third data source refers to the dataset retained after filtering by the normal content difficult example lexicon in the second data source; the third data source is the final enhanced data source. The first data source is a subset of the dynamic data source, the second data source is a subset of the first data source, and the third data source is a subset of the second data source.

[0141] In this embodiment, the enhanced data source obtained through the high recall model for illegal content has two advantages. First, it significantly increases the concentration of effective data (referring to illegal content). Second, the high concentration of effective data can be directly used for the training and iteration of the subsequent illegal content detection model. Compared with dynamic data sources, the enhanced data source greatly reduces the annotation cost, eliminating the need for a large investment in manual annotation. It also reduces the time and resource costs for training the subsequent illegal content detection model.

[0142] In some embodiments, the optimized list of prohibited content is a dynamically updated list.

[0143] In some embodiments, the optimized illegal content lexicon is generated as follows: An illegal content lexicon is obtained; each word in the lexicon is expanded in at least one dimension using a large language model to obtain an expanded illegal content lexicon. At least one candidate word is collected from a dynamic data source at preset time intervals, and this candidate word is added to the expanded illegal content lexicon to obtain the optimized illegal content lexicon.

[0144] In some embodiments, the vocabulary in the illegal content lexicon is expanded in at least one dimension using a large language model to obtain an expanded illegal content lexicon. Optionally, the vocabulary in the illegal content lexicon is expanded in at least one dimension using a large language model to obtain expanded vocabulary corresponding to each vocabulary; the vocabulary and its corresponding expanded vocabulary are then merged and deduplicated to obtain the expanded illegal content lexicon.

[0145] Optionally, at least one dimension may include one or more of the following: synonym dimension; pinyin dimension; word order reversal dimension; word order reorganization dimension; homophone substitution dimension; emoji expansion dimension; interference character insertion dimension; and context expansion dimension. This application does not limit this dimension.

[0146] For example, at least one dimension includes a synonym dimension. After obtaining the violation content lexicon, the lexicon is expanded using a large language model to include synonyms, resulting in an expanded first violation content lexicon. Synonym expansion refers to expanding synonymous or near-synonymous violation expressions based on the semantics of the violation words (basic violation words), covering violation expressions with different wording. For example, if the violation words in the lexicon include "brushing orders and receiving rebates," after expanding the violation words with synonyms, the violation words become "brushing orders and reviews," "order cashback," and "volume boosting commission."

[0147] Exemplarily, at least one dimension includes the pinyin dimension. After obtaining the illegal content word library, the large language model is used to perform pinyin dimension expansion on each word in the illegal content word library, resulting in an expanded second illegal content word library. Pinyin dimension expansion refers to expanding the full pinyin, initials abbreviation, and segmented pinyin form of illegal words, covering illegal expressions that avoid detection by using pinyin. For example, if the illegal word in the illegal content word library is "brush order for rebate", after performing pinyin dimension expansion on this illegal word, the illegal word is expanded to "shua dan fan li".

[0148] Exemplarily, at least one dimension includes the word order reversal dimension. After obtaining the illegal content word library, the large language model is used to perform word order reversal dimension expansion on each word in the illegal content word library, resulting in an expanded third illegal content word library. Word order reversal dimension expansion refers to reversing the word order of illegal words while retaining the core illegal semantics, covering circumvention expressions with reorganized word order. For example, if the illegal word in the illegal content word library is "brush order for rebate", after performing word order reversal dimension expansion on this illegal word, the illegal word is expanded to "rebate for brush order".

[0149] Exemplarily, at least one dimension includes the word order reorganization dimension. After obtaining the illegal content word library, the large language model is used to perform word order reorganization dimension expansion on each word in the illegal content word library, resulting in an expanded fourth illegal content word library. Word order reorganization dimension expansion refers to flexibly reorganizing multi-character illegal words and generating illegal expressions with different word orders by combining simple auxiliary words. For example, if the illegal word in the illegal content word library is "brush order for rebate", after performing word order reorganization dimension expansion on this illegal word, the illegal word is expanded to "do brush order to get rebate", "brush an order and you can get rebate".

[0150] Exemplarily, at least one dimension includes the homophone replacement dimension. After obtaining the illegal content word library, the large language model is used to perform homophone replacement dimension expansion on each word in the illegal content word library, resulting in an expanded fifth illegal content word library. Homophone replacement dimension expansion refers to replacing some Chinese characters in illegal words with homophones or near-homophones, covering illegal expressions that avoid detection by using homophonic words. For example, if the illegal word in the illegal content word library is "brush order for rebate", after performing homophone replacement expansion on this illegal word, the illegal word is expanded to "刷單返俐", "唰单返立".

[0151] Exemplarily, at least one dimension includes an emoji expansion dimension. After obtaining the illegal content vocabulary, each vocabulary in the illegal content vocabulary is expanded in the emoji expansion dimension through a large language model to obtain an expanded sixth illegal content vocabulary. The emoji expansion dimension expansion refers to inserting emojis related to the illegal vocabulary into the illegal vocabulary to simulate illegal expressions with symbol interference. For example, the illegal vocabulary in the illegal content vocabulary includes "brush orders and get rebates". After expanding this illegal vocabulary in the emoji expansion dimension, the illegal vocabulary is expanded to "brush orders [gold coins] and get rebates".

[0152] Exemplarily, at least one dimension includes an interfering character insertion dimension. After obtaining the illegal content vocabulary, each vocabulary in the illegal content vocabulary is expanded in the interfering character insertion dimension through a large language model to obtain an expanded seventh illegal content vocabulary. The interfering character insertion dimension expansion refers to inserting meaningless symbols, numbers, spaces and other interfering characters into the illegal vocabulary to cover the circumvention expressions of character interference. For example, the illegal vocabulary in the illegal content vocabulary includes "brush orders and get rebates". After expanding this illegal vocabulary in the interfering character insertion dimension, the illegal vocabulary is expanded to "brush * orders and get % rebates".

[0153] Exemplarily, at least one dimension includes a context expansion dimension. After obtaining the illegal content vocabulary, each vocabulary in the illegal content vocabulary is expanded in the context expansion dimension through a large language model to obtain an expanded eighth illegal content vocabulary. The context expansion dimension expansion refers to supplementing common context for the illegal vocabulary to generate illegal expressions of complete illegal short sentences. For example, the illegal vocabulary in the illegal content vocabulary includes "brush orders and get rebates". After expanding this illegal vocabulary in the context expansion dimension, the illegal vocabulary is expanded to "Come if you want to earn pocket money. Brush orders and get rebates, earn 200 yuan a day".

[0154] It should be noted that this application does not limit this. The above at least one dimension is an exemplary illustration, and this application does not limit the specific form of the at least one dimension. Taking a total of eight dimensions as an example (in fact, there may be more or fewer), at least one dimension includes the expansion of any one of the above dimensions, or includes the expansion of any two of the above dimensions, or includes the expansion of any three of the above dimensions, or includes the expansion of any four of the above dimensions, or includes the expansion of any five of the above dimensions, or includes the expansion of any six of the above dimensions, or includes the expansion of any seven of the above dimensions, or includes the expansion of the above eight dimensions.

[0155] In some embodiments, at least one candidate word is collected from a dynamic data source at preset time intervals. The feature extraction model is used to extract features from each word and at least one candidate word in the expanded illegal content lexicon to obtain the feature vectors corresponding to each word and the feature vectors corresponding to at least one candidate word. Based on the feature vectors corresponding to each word, the center feature vector of the expanded illegal content lexicon is obtained, and a first distance between the feature vectors corresponding to at least one candidate word and the center feature vector is calculated. If the first distance is less than a first distance threshold, at least one candidate word is added to the expanded illegal content lexicon to obtain an optimized illegal content lexicon.

[0156] Here, the central feature vector refers to the mean vector of all word feature vectors in the expanded illegal content lexicon. The first distance refers to the similarity distance (spatial distance, calculated using cosine or Euclidean distance) between the feature vector of a candidate word and the central feature vector; this first distance measures the semantic similarity between them. The first distance threshold is a preset distance judgment threshold used to distinguish whether a candidate word belongs to the expanded illegal content lexicon.

[0157] In some embodiments, if the first distance is less than (or less than or equal to) a first distance threshold, the candidate word is indicated to be semantically similar to the violation content lexicon, the candidate word is determined to be a violation expression of the same type, and the candidate word is added to the expanded violation content lexicon.

[0158] In some embodiments, if the first distance is greater than (or greater than or equal to) the first distance threshold, it indicates that the candidate word has a large semantic difference from the violation content lexicon, and the candidate word is determined to be a different type of violation expression. In this case, it is not necessary to add the candidate word to the expanded violation content lexicon.

[0159] In some embodiments, a second distance is calculated between the feature vector corresponding to each word and the central feature vector; if the second distance is less than a second distance threshold, the words corresponding to the second distance less than the second distance threshold are removed from the expanded illegal content word library to obtain the optimized illegal content word library.

[0160] The second distance refers to the similarity distance (spatial distance, calculated using cosine or Euclidean distance) between the feature vectors of each word in the expanded violating content lexicon and the central feature vector. This second distance measures the semantic similarity between the feature vectors of each candidate word and the central feature vector in the expanded violating content lexicon. The magnitude of the second distance reflects the degree of semantic fit between individual words in the expanded violating content lexicon and the overall violating semantics of the lexicon. The second distance threshold is a preset distance judgment threshold used to distinguish whether to retain or remove individual words in the expanded violating content lexicon.

[0161] It should be noted that the above feature extraction model is a trained model. In an exemplary training method, the training process of the general feature extraction model is as follows: Illegal words are selected from the expanded illegal content lexicon, sentences containing illegal words are constructed, and these sentences are used as positive samples; normal words are selected from the dynamic data source, sentences containing normal words are constructed, and these sentences are used as negative samples. A text feature extractor extracts feature vectors from the positive and negative samples, mapping the feature vectors corresponding to the positive and negative samples to the feature space. The general feature extraction model is trained based on a contrastive learning approach, aiming to maximize the distance between the feature vectors corresponding to the positive and negative samples in the feature space.

[0162] In this embodiment, a large language model is used to expand the vocabulary of each word in the illegal content lexicon in multiple dimensions, enabling the batch generation of various variants of illegal words and significantly improving the coverage of the lexicon. The optimization of the lexicon is continuous and iterative. At preset time intervals (such as daily / weekly), new illegal candidate words (such as newly added internet slang) are collected from dynamic data sources and added to the expanded illegal content lexicon to obtain an optimized illegal content lexicon, ensuring that the optimized illegal content lexicon can adapt to the dynamic changes of illegal expressions.

[0163] Application Scenario Examples The following explanation uses vulgar and pornographic content as the violation and a vulgar and pornographic content detection model as the violation detection model.

[0164] Step 1: Obtain the content to be detected and the recognition rules and strategies. The recognition rules and strategies include the first recognition rule and the second recognition rule.

[0165] The content to be detected is used to identify whether it is vulgar or pornographic. The identification rule strategy refers to the judgment rules used to filter out non-vulgar or pornographic content (normal content). The identification rule strategy is used to quickly filter out undisputed normal content.

[0166] Optionally, vulgar pornographic content includes one or more of the following: vulgar pornographic text, vulgar pornographic images, vulgar pornographic strings, vulgar pornographic links, vulgar pornographic multilingual mixed text, and vulgar pornographic emojis. The above are merely illustrative examples, and this application does not limit the specific form of the vulgar pornographic content package.

[0167] In some embodiments, the identification rule strategy includes a first identification rule and a second identification rule. Optionally, the second identification rule has a higher identification complexity than the first identification rule. The first identification rule is a basic rule with low identification complexity, and the second identification rule is an advanced rule with high identification complexity. Identification complexity indicates the difficulty of determining the identification rule.

[0168] In some embodiments, recognition complexity is represented by logical complexity; or, recognition complexity is represented by computational complexity. Logical complexity indicates whether the first recognition rule / second recognition rule needs to be combined with context or variant pattern matching. Computational complexity indicates the resource and time costs required to execute the first recognition rule / second recognition rule.

[0169] In some embodiments, the identification rule strategy is constructed based on a dynamic data source. Optionally, a dynamic data source is obtained, at least one type of lexicon is constructed based on the dynamic data source, and an identification rule strategy is determined based on the at least one type of lexicon.

[0170] Let's take the first / second identification rule, which includes the context exclusion rule, as an example. If the text content in the dynamic data source contains explicit context markers (normal domain markers), the text content is determined to be normal content. For example, if the dynamic data source includes the text "Medical Science Popularization: The physiological structure of the chest is composed of multiple glands," where "chest" is a sensitive word indicating vulgar or pornographic content, but this is normal medical content, then the text content is determined to be normal content.

[0171] Step 2: Using the first identification rule as an auxiliary filter, if the content to be detected is not matched by the first identification rule, the vulgar and pornographic detection model is used to predict the content to be detected and output the confidence level of the content to be detected.

[0172] The first identification rule is a basic rule with low identification complexity. It is used to filter out non-pornographic content (normal content) using a low-complexity judgment rule. The first identification rule can also be called a basic rule or a simple rule. Auxiliary filtering refers to a rule used to filter non-pornographic content or narrow the detection range. For example, the first identification rule can be used as an auxiliary filter to filter non-pornographic content in the initial stage of content detection.

[0173] In some embodiments, the first identification rule includes, but is not limited to: a first sensitive word negation rule, a first semantic matching rule, and a first context exclusion rule. This application does not limit this.

[0174] In some embodiments, the first identification rule is used as an auxiliary filter to determine whether the content to be detected is matched by the first identification rule. If the content to be detected is not matched by the first identification rule, it is determined that the content to be detected is suspected vulgar and pornographic content (or vulgar and pornographic content to be determined, or candidate vulgar and pornographic content). The content to be detected (in this case, suspected vulgar and pornographic content) is input into the vulgar and pornographic detection model, which predicts the content to be detected and outputs the confidence level of the content to be detected.

[0175] The confidence score indicates the probability that the detected content is classified as vulgar or pornographic. The confidence score ranges from 0 to 1. A higher confidence score indicates a higher probability that the vulgar / pornographic detection model classifies the content as vulgar or pornographic; a lower confidence score indicates a lower probability.

[0176] It should be noted that the above-mentioned pornography detection model is a fully trained model. A pornography detection model can also be referred to as a fully trained pornography detection model.

[0177] Step 3: When the confidence level of the content to be detected is within the target confidence level range, the second identification rule is used as an auxiliary screening method, and the content to be detected is determined to be illegal by sequentially calling at least one type of thesaurus through the illegal content detection model.

[0178] In some embodiments, the content to be detected (in this case, suspected vulgar and pornographic content) is input into a vulgar and pornographic detection model. The model predicts the content to be detected and outputs the confidence level of the content. If the confidence level of the content to be detected is within the target confidence level range, a second identification rule is used as an auxiliary filter. The vulgar and pornographic detection model sequentially calls at least one type of lexicon to determine whether the content to be detected is vulgar and pornographic.

[0179] The target confidence interval is used to indicate the target probability interval for determining whether the content to be detected is vulgar or pornographic. The target confidence interval includes a first boundary value and a second boundary value. The first boundary value is the maximum value of the target confidence interval, and the second boundary value is the minimum value of the target confidence interval. When the confidence level of the content to be detected falls within the target confidence interval, it indicates that the prediction result of the vulgar and pornographic detection model is ambiguous; the content to be detected is suspected to be vulgar or pornographic, and it cannot be directly determined whether it belongs to vulgar or pornographic content.

[0180] The second identification rule is an advanced rule with high identification complexity. It is used to filter out non-pornographic content (normal content) using a more complex judgment rule. The second identification rule can also be called a complex rule or an advanced rule.

[0181] In some embodiments, the second identification rule includes, but is not limited to: a second sensitive word negation rule, a second semantic matching rule, and a second context exclusion rule. This application does not limit this.

[0182] Taking the second identification rule, which includes the second sensitive word negation principle, as an example, the second sensitive word is a variant sensitive word or a metaphorical sensitive word. The identification complexity of the second sensitive word is higher than that of the first sensitive word. For example, if the first sensitive word is "pornography", the second sensitive words are "eroticism" or "seqing".

[0183] In some embodiments, at least one type of thesaurus includes at least one of a vulgar pornography thesaurus, a normal content difficult example thesaurus, and a vulgar pornography difficult example thesaurus. Optionally, when the confidence level of the content to be detected is within the target confidence level range, a second identification rule is used as an auxiliary screening method, and the vulgar pornography detection model sequentially calls at least one of the vulgar pornography thesaurus, the normal content difficult example thesaurus, and the vulgar pornography difficult example thesaurus to determine that the content to be detected is vulgar pornography.

[0184] The dictionary is divided into three parts: a vulgar and pornographic terminology database, a normal content and difficult example database, and a vulgar and pornographic terminology database. The vulgar and pornographic terminology database consists of words that are easily misidentified as vulgar and pornographic. The vulgar and pornographic terminology database consists of words that are easily missed as vulgar and pornographic.

[0185] In some embodiments, if the content to be detected is determined to be vulgar or pornographic, a blacklisting operation is performed on the content to be detected. This blacklisting operation is the final processing step for the content to be detected.

[0186] Optionally, the blacklisting operation includes one or more of the following: direct interception, content blocking, violation marking (marking of vulgar and pornographic content, or marking of vulgar and pornographic content as an example), and related handling. Direct interception refers to refusing the posting of vulgar and pornographic content (e.g., preventing posts from being published). Content blocking refers to hiding already posted vulgar and pornographic content, making it inaccessible to other accounts. Violation marking refers to marking vulgar and pornographic content and including it in a vulgar and pornographic content database for subsequent iteration and optimization of the vulgar and pornographic detection model. Related handling refers to issuing warnings or limiting the reach of accounts that post vulgar and pornographic content. This application does not limit this aspect.

[0187] In some embodiments, the above-described identification rule strategy is built based on a dynamic data source.

[0188] In some embodiments, before step 1 above, the method further includes: obtaining a dynamic data source, constructing at least one type of lexicon based on the dynamic data source, and determining a recognition rule strategy based on at least one type of lexicon.

[0189] The dynamic data source is the full dataset within the target business scenario. It is used to construct at least one type of thesaurus. Optionally, the target business scenario includes, but is not limited to, at least one of the following: communication scenarios, live streaming with bullet comments, channel scenarios, and community / forum scenarios. This application does not limit this. The aforementioned dynamic data source can also be referred to as dashboard data (dashboard data source).

[0190] In some embodiments, a dynamic data source is obtained, and at least one type of thesaurus is constructed based on the dynamic data source and an open-source vulgar and pornographic thesaurus. The dynamic data source is matched against the open-source vulgar and pornographic thesaurus, and at least one type of thesaurus is constructed based on the matching results (or hit results). The constructed at least one type of thesaurus includes at least one of a vulgar and pornographic thesaurus, a normal content difficult example thesaurus, and a vulgar and pornographic difficult example thesaurus. This application does not limit this.

[0191] In some embodiments, if the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the vulgar and pornographic word library sequentially called by the vulgar and pornographic detection model, the content to be detected is not hit by the second identification rule, and the content to be detected is not hit by the normal content difficult example word library, then the content to be detected is determined to be vulgar content.

[0192] For example, when the confidence level of the content to be detected is within the target confidence level range, the vulgar and pornographic detection model first calls a vulgar and pornographic lexicon to filter the content. The vulgar and pornographic lexicon consists of vulgar and pornographic words with obvious vulgar and pornographic content. If the vulgar and pornographic lexicon matches the content to be detected, that is, if the vulgar and pornographic lexicon matches a vulgar and pornographic word in the content to be detected, it is confirmed that the content to be detected has a first vulgar and pornographic possibility (the content to be detected has basic vulgar and pornographic features). The vulgar and pornographic detection model then calls a second identification rule to filter the content to be detected. The second identification rule is used to filter non-vulgar and pornographic content (normal content) through a judgment rule with high recognition complexity (advanced rules, such as variant sensitive word matching, context semantic association). If the second identification rule does not match the content to be detected, it is confirmed that the content to be detected has a second vulgar and pornographic possibility (the vulgar and pornographic suspicion of the content to be detected cannot be eliminated by the complex second identification rule). The vulgar and pornographic detection model continues to filter the content to be detected by calling the normal content difficult example dictionary. The normal content difficult example dictionary consists of normal words that are easily misidentified as vulgar and pornographic content. If the normal content difficult example dictionary does not match the content to be detected, the content to be detected is determined to be vulgar and pornographic content (the content to be detected does not belong to the "normal content difficult example that is easily misjudged", and there is no situation where it is mistakenly labeled as vulgar and pornographic, so it is confirmed as vulgar and pornographic content).

[0193] In some embodiments, if the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the vulgar and pornographic word library in sequence through the vulgar and pornographic detection model, the content to be detected is not hit by the second identification rule, the content to be detected is hit by the normal content difficult example word library, and the content to be detected is hit by the vulgar and pornographic difficult example word library, then the content to be detected is determined to be vulgar and pornographic content.

[0194] For example, when the confidence level of the content to be detected is within the target confidence level range, the vulgar and pornographic detection model first calls a vulgar and pornographic lexicon to filter the content. The vulgar and pornographic lexicon consists of vulgar and pornographic words with obvious vulgar and pornographic content. If the vulgar and pornographic lexicon matches the content to be detected, that is, if the vulgar and pornographic lexicon matches a vulgar and pornographic word in the content to be detected, it is confirmed that the content to be detected has a first vulgar and pornographic possibility (the content to be detected has basic vulgar and pornographic features). The vulgar and pornographic detection model then calls a second identification rule to filter the content to be detected. The second identification rule is used to filter non-vulgar and pornographic content through a judgment rule with high recognition complexity (advanced rules, such as variant sensitive word matching, context semantic association). If the second identification rule does not match the content to be detected, it is confirmed that the content to be detected has a second vulgar and pornographic possibility (the vulgar and pornographic suspicion of the content to be detected cannot be eliminated by the complex second identification rule). The vulgar and pornographic detection model continues to filter the content to be detected by calling a normal content difficult example dictionary. This normal content difficult example dictionary consists of normal words that are easily misidentified as vulgar and pornographic content. If the normal content difficult example dictionary matches the content to be detected, it is confirmed that the content to be detected has a third possibility of being vulgar and pornographic (the content to be detected belongs to "normal difficult examples that are easily misjudged," resulting in a judgment contradiction, requiring further judgment). The vulgar and pornographic detection model then continues to filter the content to be detected by calling a vulgar and pornographic difficult example dictionary. This vulgar and pornographic difficult example dictionary consists of vulgar and pornographic words that are easily missed as vulgar and pornographic content. If the vulgar and pornographic difficult example dictionary matches the content to be detected, the content to be detected is confirmed as vulgar and pornographic content.

[0195] In some embodiments, if the confidence level of the content to be detected is greater than the first boundary value of the target confidence level interval, the content to be detected is determined to be vulgar and pornographic content. When the confidence level of the content to be detected is greater than the first boundary value of the target confidence level interval, the vulgar and pornographic detection model determines that the content to be detected is vulgar and pornographic content with a very high probability, and at this time, there is no need to activate a second recognition rule or call various thesaurus for cross-validation.

[0196] In some embodiments, the content to be detected is non-vulgar or pornographic content, that is, the content to be detected is normal content.

[0197] In some embodiments, if it is determined that the content to be detected is not vulgar or pornographic, a release operation (also known as a whitelisting operation) is performed on the content to be detected. The release operation is the final processing operation for the content to be detected. Optionally, the release operation includes one or more of the following operations: allow publication, no additional marking, hard example marking (hard example marking for normal content), and fast approval. This application does not limit this.

[0198] In some embodiments, the determination that the content to be detected is not vulgar or pornographic includes at least one of the following situations: • The first identification rule is used as an auxiliary filter. If the content to be detected is matched by the first identification rule, it is determined that the content to be detected is not vulgar or pornographic.

[0199] • If the confidence level of the content to be detected is less than the second boundary value of the target confidence level interval, the content to be detected is determined to be non-vulgar and pornographic content.

[0200] • If the confidence level of the content to be detected is within the target confidence level range, and the vulgar and pornographic word library is not matched when the vulgar and pornographic detection model is called, the content to be detected is determined to be non-vulgar and pornographic content.

[0201] • If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the vulgar and pornographic word library called by the vulgar and pornographic detection model, and the second recognition rule is hit by the content to be detected, then the content to be detected is determined to be non-vulgar and pornographic content.

[0202] • If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the vulgar and pornographic word library called by the vulgar and pornographic detection model, the content to be detected is not hit by the second identification rule, the content to be detected is hit by the normal content difficult example word library, and the content to be detected is not hit by the vulgar and pornographic difficult example word library, then the content to be detected is determined to be non-vulgar and pornographic content.

[0203] In some embodiments, the above-described pornography detection model is trained by augmenting training samples.

[0204] In some embodiments, enhanced training samples are obtained and input into a general pornography detection model (also known as a pornography detection model to be trained). The predicted results output by the general pornography detection model are obtained. Based on the predicted results and the ground truth labels, a sample loss is obtained. The model parameters of the general detection model are trained using the sample loss to obtain the pornography detection model. The enhanced training samples are labeled with ground truth labels, which indicate whether the enhanced training samples contain pornography.

[0205] In some embodiments, enhanced training samples are input into a general pornography detection model to obtain the prediction results output by the general pornography detection model. These prediction results are used to indicate the predicted labels for the enhanced training samples. Optionally, a sample loss (cross-entropy loss) is obtained based on the true labels of the enhanced training samples and the prediction results. This sample loss indicates the error between the prediction results output by the general pornography detection model and the true labels of the enhanced training samples. The sample loss is used as the loss value during training, and the general pornography detection model is trained based on this loss value to obtain the pornography detection model.

[0206] In some embodiments, the enhanced training samples are obtained through a pornographic recall model.

[0207] In some embodiments, the first identification rule is used as an auxiliary filter. The dynamic data source is filtered using a vulgar and pornographic recall model combined with an optimized vulgar and pornographic lexicon and a normal content difficult example lexicon to obtain an enhanced data source. The enhanced data source is then labeled to obtain enhanced training samples for training the vulgar and pornographic detection model. Optionally, the data concentration of vulgar and pornographic content in the enhanced data source is higher than that in the dynamic data source. Data concentration indicates the proportion of vulgar and pornographic content to the total amount of data in the dataset (here, the dynamic data source or the enhanced data source).

[0208] The optimized vulgar and pornographic terminology database refers to an improved database obtained by expanding and supplementing the existing vulgar and pornographic terminology database. The enhanced data source refers to a data source with a higher concentration of vulgar and pornographic content obtained after filtering dynamic data sources.

[0209] For example, after filtering the dynamic data source, a large amount of uncontroversial normal content is removed, while high-probability vulgar and pornographic content and easily misjudged difficult-to-identify content are retained, thus increasing the concentration of vulgar and pornographic content in the enhanced data source to be higher than that in the dynamic data source. For instance, if a dynamic data source contains 10 million data entries, of which 10,000 to 20,000 are vulgar and pornographic, the concentration of such content in the dynamic data source is approximately 0.1% to 0.2%. After filtering using a vulgar and pornographic recall model, 50,000 data entries are selected from the 10 million data entries, of which 10,000 to 20,000 are still vulgar and pornographic, increasing the concentration of such content in the enhanced data source to approximately 20% to 40%.

[0210] In some embodiments, the specific method for obtaining the enhanced data source is as follows: The dynamic data source is filtered by calling the first identification rule through the vulgar and pornographic recall model, retaining the first part of the data source that was not matched by the first identification rule; the first part of the data source is filtered by calling the optimized vulgar and pornographic lexicon through the vulgar and pornographic recall model, retaining the second part of the data source that was not matched by the optimized vulgar and pornographic lexicon; the second part of the data source is filtered by calling the normal content difficult example lexicon through the vulgar and pornographic recall model, retaining the third part of the data source that was not matched by the normal content difficult example lexicon, and using the third part of the data source as the enhanced data source.

[0211] The first data source refers to the dataset retained after filtering by the first recognition rule in the dynamic data source. The second data source refers to the dataset retained after filtering by an optimized vulgar and pornographic word list in the first data source. The third data source refers to the dataset retained after filtering by a normal content word list in the second data source; this third data source is the final enhanced data source. The first data source is a subset of the dynamic data source, the second data source is a subset of the first data source, and the third data source is a subset of the second data source.

[0212] In some embodiments, the optimized vulgar pornography lexicon is a dynamically updated lexicon.

[0213] In some embodiments, the optimized vulgar and pornographic lexicon is generated as follows: A vulgar and pornographic lexicon is obtained; each word in the lexicon is expanded in at least one dimension using a large language model to obtain an expanded vulgar and pornographic lexicon. At least one candidate word is collected from a dynamic data source at preset time intervals, and this candidate word is added to the expanded vulgar and pornographic lexicon to obtain the optimized vulgar and pornographic lexicon.

[0214] In some embodiments, the vocabulary in the vulgar and pornographic lexicon is expanded in at least one dimension using a large language model to obtain an expanded vulgar and pornographic lexicon. Optionally, the vocabulary in the vulgar and pornographic lexicon is expanded in at least one dimension using a large language model to obtain expanded vocabulary corresponding to each vocabulary; the vocabulary and its corresponding expanded vocabulary are then merged and deduplicated to obtain the expanded vulgar and pornographic lexicon.

[0215] Optionally, at least one dimension may include one or more of the following: synonym dimension; pinyin dimension; word order reversal dimension; word order reorganization dimension; homophone substitution dimension; emoji expansion dimension; interference character insertion dimension; and context expansion dimension. This application does not limit this dimension.

[0216] In some embodiments, at least one candidate word is collected from a dynamic data source at preset time intervals. The feature extraction model is used to extract features from each word and at least one candidate word in the expanded vulgar and pornographic word library to obtain the feature vectors corresponding to each word and the feature vectors corresponding to at least one candidate word. Based on the feature vectors corresponding to each word, the center feature vector of the expanded vulgar and pornographic word library is obtained, and a first distance between the feature vectors corresponding to at least one candidate word and the center feature vector is calculated. If the first distance is less than a first distance threshold, at least one candidate word is added to the expanded vulgar and pornographic word library to obtain an optimized vulgar and pornographic word library.

[0217] Here, the central feature vector refers to the mean vector of all word feature vectors in the expanded vulgar and pornographic lexicon. The first distance refers to the similarity distance (spatial distance, calculated using cosine or Euclidean distance) between the feature vector of a candidate word and the central feature vector; this first distance measures the semantic similarity between them. The first distance threshold is a preset threshold used to distinguish whether a candidate word belongs to the expanded vulgar and pornographic lexicon.

[0218] In some embodiments, if the first distance is less than (or less than or equal to) a first distance threshold, the candidate word is indicated to be semantically similar to the vulgar pornographic lexicon, the candidate word is determined to be a vulgar pornographic expression of the same kind, and the candidate word is added to the expanded vulgar pornographic lexicon.

[0219] In some embodiments, if the first distance is greater than (or greater than or equal to) the first distance threshold, it indicates that the candidate word has a large semantic difference from the vulgar pornography lexicon, and the candidate word is determined to be a different type of vulgar pornography expression. In this case, it is not necessary to add the candidate word to the expanded vulgar pornography lexicon.

[0220] In some embodiments, a second distance is calculated between the feature vector corresponding to each word and the central feature vector; if the second distance is less than a second distance threshold, the words corresponding to the second distance less than the second distance threshold are removed from the expanded vulgar and pornographic word library to obtain the optimized vulgar and pornographic word library.

[0221] The second distance refers to the similarity distance (spatial distance, calculated using cosine or Euclidean distance) between the feature vectors of each word in the expanded vulgar and pornographic lexicon and the central feature vector. This second distance measures the semantic similarity between the feature vectors of each candidate word and the central feature vector in the expanded vulgar and pornographic lexicon. The magnitude of the second distance reflects the degree of fit between the semantics of individual words in the expanded vulgar and pornographic lexicon and the overall vulgar and pornographic lexicon. The second distance threshold is a preset distance judgment threshold used to distinguish whether to retain or remove individual words in the expanded vulgar and pornographic lexicon.

[0222] In summary, the content detection method proposed in this application employs a dual-lexicon chain filtering approach on dynamic data sources using a constructed vulgar and pornographic lexicon and a normal content difficult example lexicon. This is further enhanced by combining this with identification rule strategies to obtain an augmented data source. The vulgar and pornographic lexicon ensures the recall of such content, while the normal content difficult example lexicon improves the accuracy of the recalled vulgar and pornographic content. The augmented data source recalled by the vulgar and pornographic recall model has a high concentration of illegal content, and labeling the augmented data source significantly reduces labeling costs. By training the vulgar and pornographic detection model with augmented training samples obtained after labeling the augmented data source, and then combining at least one type of lexicon and identification rule strategies to identify the content to be detected, the predictive accuracy of the vulgar and pornographic model can be greatly improved.

[0223] The following is an overall explanation of content detection methods.

[0224] In the following embodiments, the high-recall model is used as the violation content recall model, the high-accuracy model is used as the violation content detection model, and the violation content is vulgar and pornographic text (a representation of vulgar and pornographic content) as an example for illustration.

[0225] (a) Product Realization This application proposes a method for detecting illegal content based on adaptive multi-strategy fusion, which integrates a BERT language model, a lexicon (high-accuracy lexicon and high-call lexicon), and recognition rule strategies to achieve high-accuracy detection of illegal content. The method is illustrated using an example where the recognition rule strategies include basic rules (the first recognition rule in the previous embodiment) and complex rules (the second recognition rule in the previous embodiment).

[0226] This application can be applied in the following business scenarios: (1) In the scenario of machine review of vulgar and pornographic text in message queue, the hit samples are directly judged as black, thereby reducing the amount of review data, reducing the accumulation of review samples, and improving review efficiency.

[0227] (2) Migrate to other scenarios, such as channel messages, comments, live barrages, etc., and adapt to specific business scenarios to achieve single development and multi-scenario applicability, thereby reducing development costs.

[0228] (3) The overall process, methods, strategies and thematic database construction methods can be transferred to other violation types of review scenarios, such as replacing the "vulgar and pornographic" category in this application with categories such as "illegal traffic generation", and constructing a new tag detection algorithm according to the technical process of this application.

[0229] (II) Technical Implementation This application proposes a violation content detection method based on adaptive multi-strategy fusion. The method mainly consists of two steps: Stage 1 data cold start and Stage 2 model training and iterative update.

[0230] The following explanation uses group message dashboard data as a dynamic data source as an example.

[0231] Stage 1 Data Cold Start: The purpose of this step is to construct the initial lexicon and design basic rules using group message dashboard data and an open-source vulgar and pornographic lexicon, as detailed below: First, an open-source vulgar and pornographic thesaurus was used to filter the group message dashboard data. The filtering results were analyzed, and three initial versions of the thesaurus were built, along with basic vulgar and pornographic identification rules and strategies. The specific process is as follows: ① Construct a vulgar and pornographic lexicon using the hit vulgar and pornographic data; ② Construct a normal sample difficult example vocabulary using falsely invoked vulgar and pornographic data; ③ Construct a vulgar and pornographic difficult-to-example lexicon using the missed vulgar and pornographic data; ④ Conduct preliminary analysis of hit, missed, and false positive data to construct identification rules and strategies for vulgar and pornographic content.

[0232] In some embodiments, the vulgar and pornographic terminology database is expanded to obtain an expanded vulgar and pornographic terminology database, and the expanded vulgar and pornographic terminology database is optimized to obtain an optimized vulgar and pornographic terminology database.

[0233] The strategy for building and optimizing a thesaurus based on LLM mainly consists of the following five steps, taking a vulgar and pornographic thesaurus as an example: Step 1: Build a vulgar and pornographic lexicon.

[0234] • Collect comprehensive data and extract high-frequency pornographic terms from publicly available pornographic websites, forums, social media platforms, and mobile texts from three dramas.

[0235] • Integrate open-source vulgar and pornographic terminology databases, collect existing open-source vulgar and pornographic terminology databases, and perform deduplication and merging.

[0236] • After cleaning and deduplication, a vulgar and pornographic word database is formed (denoted as V0).

[0237] Step 2: Expand the vocabulary using LLM.

[0238] • Expand each word in the vulgar and pornographic lexicon using a large language model to generate multiple variant forms, including: synonym dimension expansion; pinyin dimension expansion; word order reversal dimension expansion; word order reorganization dimension expansion; homophone replacement dimension expansion; emoticon expansion dimension expansion; interference character insertion dimension expansion; and context expansion dimension expansion.

[0239] • Merge and deduplicate the generated extended vocabulary with the vulgar pornography lexicon (V0) to form the expanded vulgar pornography lexicon (denoted as V1).

[0240] Step 3: Build a training dataset and train the feature extraction model.

[0241] • Select pornographic information from the overall data and words from the expanded vulgar pornographic lexicon, and construct sentences containing the words as positive samples. Select information from normal text (normal text in the overall data) as negative samples. Balance the positive and negative samples to ensure that the number of samples of the two types is equal.

[0242] • Train RoBERTa as a text feature extractor, which maps text to a feature space.

[0243] Training objective: To separate the feature vectors of positive samples (vulgar and pornographic text) and negative samples (normal text) as much as possible in the feature space.

[0244] • Training method: Use comparative learning.

[0245] Step 4: Obtain the optimized vulgar and pornographic word library.

[0246] • Use a trained feature extractor to extract words from the expanded vulgar pornography lexicon.

[0247] • Collect a batch of new candidate words (including newly emerging words on the Internet) from the group message dashboard data, and use a feature extractor to extract their feature vectors.

[0248] • Calculate the distance between the feature vector of each candidate word and the pornographic feature center (obtained by the mean of the feature vectors of each word in the expanded vulgar pornographic lexicon, the center feature vector in the previous embodiment). If the distance is less than the set pornographic threshold (the first distance threshold in the previous embodiment), then add the candidate word to the expanded vulgar pornographic lexicon.

[0249] Similarly, words in the expanded vulgar and pornographic word library are filtered, and their distance from the feature center of normal text is calculated. If the weak distance is less than the set normal text threshold (the second distance threshold in the previous embodiment), then the words are removed from the expanded vulgar and pornographic word library.

[0250] After the above optimizations, an optimized vulgar and pornographic word library is formed (denoted as V2).

[0251] Step 5: Iterative update.

[0252] • Repeat steps 2 through 4 periodically, updating the lexicon with new data and expanding and optimizing it with updated LLM and feature extraction models to adapt to the ever-changing network environment.

[0253] Stage 2 Model Training & Iterative Updates: This step mainly involves building two versions of the model: a high-recall model for improving the effective data concentration of group message dashboards and a high-accuracy model for machine review of vulgar and pornographic texts. The high-accuracy model is then iteratively updated according to business needs.

[0254] For the high-recall model: An optimized vulgar / pornographic terminology library and a normal content difficult-example terminology library are used to perform dual-lexicon chain filtering on the group message dashboard data, combined with basic rules to assist in selection. The vulgar / pornographic terminology library ensures the recall of vulgar / pornographic data, while the normal content difficult-example terminology library improves the accuracy of the recalled data. Furthermore, simple basic rules are used to improve processing efficiency. The specific process is as follows: Figure 5 As shown, there are three filtering steps: Step 1: Enter the content to be detected.

[0255] Step 2: Determine whether the basic rules have been successfully used for filtering.

[0256] Step 3: Release the vehicle if the screening is successful.

[0257] Successful filtering refers to the basic rules being met.

[0258] Step 4: If the filtering is unsuccessful, determine whether the vulgar and pornographic keyword database has been matched.

[0259] "Not filtered" means that the basic rules were not met.

[0260] Step 5: If the vulgar and pornographic keyword database is matched, determine whether the normal content keyword database is matched.

[0261] Step 6: If the normal content is not matched by the difficult example dictionary, it is judged as vulgar and pornographic.

[0262] Use basic rules for filtering; if a match is found, allow the data directly. For data not matched by the basic rules, filter using a vulgar / pornographic keyword database; if not matched, allow the data directly. For data matched by the vulgar / pornographic keyword database, filter using a normal content keyword database; if matched, allow the data directly; otherwise, classify the data as vulgar / pornographic.

[0263] The high-recall model can significantly increase the concentration of effective data in the overall data and greatly reduce the labeling cost. Through offline data testing, the accuracy is about 15%, the recall is about 95%, and the effective data concentration is increased from 0.1% to 15%, which is an improvement of up to 150 times.

[0264] For the high-accuracy model: it integrates the BERT model, recognition rule strategy (basic rules and complex rules), and a triple high-accuracy vocabulary database (vulgar and pornographic vocabulary database, normal content difficult example vocabulary database, and vulgar and pornographic difficult example vocabulary database), with an accuracy rate of up to 98%+.

[0265] First, a high-recall model is used to filter the group message dashboard data, obtaining high-concentration data of vulgar and pornographic content (the enhanced data source in the previous example). Then, this data is labeled for training the BERT model. The specific process is as follows: Figure 6 As shown: Step 1: Enter the content to be detected.

[0266] Step 2: Determine if the basic rules have been successfully used for filtering.

[0267] Step 3: Release the vehicle if the screening is successful.

[0268] Successful filtering refers to the basic rules being met.

[0269] Step 4: Enter the BERT model if the filtering is unsuccessful.

[0270] "Unsuccessful filtering" means that the basic rules were not met.

[0271] Step 5: Release the vehicle if the confidence level is less than the second boundary value.

[0272] Step 6: If the confidence level is greater than the first boundary value, it is determined to be vulgar and pornographic.

[0273] Step 7: Determine that the confidence level is within the target confidence level range.

[0274] Step 8: If the confidence level falls within the target confidence level range, determine whether the vulgar and pornographic keyword database has been matched.

[0275] Step 9: Allow passage if the vulgar and pornographic keywords do not match.

[0276] Step 10: If the vulgar and pornographic word list is matched, determine whether the complex rule is matched.

[0277] Step 11: If the complex rule is not hit, determine whether the normal content difficult example vocabulary is hit.

[0278] Step 12: If the normal content difficult example vocabulary is matched, determine whether the vulgar and pornographic difficult example vocabulary is matched.

[0279] Step 13: If the word "vulgar and pornographic" is matched in the dictionary of difficult examples, it is determined to be vulgar and pornographic.

[0280] For example, first, the content to be detected is input, and basic rules are used for filtering. If a match is found, it is allowed directly; otherwise, the data is input into the BERT model for processing. The data that does not match is then input into the trained BERT model for prediction. Taking a target confidence interval of [0.7, 0.95] as an example, 0.95 is the first boundary value, and 0.7 is the second boundary value. The data is then dynamically processed based on the predicted confidence level (conf). a) Samples with a confidence level less than 0.7 (conf<0.7) are allowed to pass directly.

[0281] b) For samples with confidence levels in the range of 0.7 to 0.95 (0.7 ≤ conf ≤ 0.95), filter them by sequentially calling at least one of the following: vulgar pornography dictionary, complex rules, normal content difficult example dictionary, and vulgar pornography difficult example dictionary.

[0282] c) Samples with a confidence level greater than or equal to 0.95 (conf>0.95) are directly identified as vulgar and pornographic text.

[0283] Content to be detected with a confidence level between 0.7 and 0.95 is first filtered using a vulgar and pornographic terminology database. Data that does not match the database is allowed to proceed, while data that matches is filtered using a more complex rule. If the complex rule matches, the content is allowed to proceed; otherwise, it is filtered using a more complex rule. If no match is found, the content is directly classified as vulgar and pornographic, while the content that matches is filtered using a more complex rule. Content that matches the complex rule is directly classified as vulgar and pornographic, while content that does not match is allowed to proceed.

[0284] In addition, the high-precision model is iteratively updated according to steps 1 to 13 above. During the update process, the three major word libraries (the high-precision version of the vulgar and pornographic word library, the normal content difficult example word library, and the vulgar and pornographic difficult example word library) are updated based on the false positive and false negative data. Relevant rules are extracted and added to the complex rules to further improve the accuracy of the high-precision model.

[0285] Figure 7 This is a block diagram of a content detection apparatus provided in an exemplary embodiment. The content detection apparatus includes: an acquisition module 910, a prediction module 920, and a determination module 930.

[0286] The acquisition module 910 is used to acquire the content to be detected and the recognition rule strategy. The recognition rule strategy includes a first recognition rule and a second recognition rule. The recognition complexity of the second recognition rule is higher than that of the first recognition rule. The prediction module 920 is used to use the first identification rule as an auxiliary filter, and when the content to be detected is not hit by the first identification rule, it predicts the content to be detected through the violation content detection model and outputs the confidence level of the content to be detected. The determination module 930 is used to determine the content to be detected as the illegal content by using the second identification rule as an auxiliary filter when the confidence level of the content to be detected is within the target confidence level range, and by sequentially calling at least one type of thesaurus through the illegal content detection model. The at least one type of thesaurus includes at least one of the following: a thesaurus of illegal content, a thesaurus of normal content with difficult examples, and a thesaurus of illegal content with difficult examples.

[0287] In some embodiments, the determining module 930 is further configured to be at least one of the following: If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon sequentially called by the violation content detection model, the second identification rule does not hit the content to be detected, and the normal content difficult example lexicon does not hit the content to be detected, then the content to be detected is determined to be the violation content. If the confidence level of the content to be detected is within the target confidence level range, and the violation content detection model sequentially calls the violation content lexicon to hit the content to be detected, the second identification rule does not hit the content to be detected, the normal content difficult example lexicon hits the content to be detected, and the violation content difficult example lexicon hits the content to be detected, then the content to be detected is determined to be the violation content.

[0288] In some embodiments, the determining module 930 is further configured to be at least one of the following: If the confidence level of the content to be detected is greater than the first boundary value of the target confidence level interval, the content to be detected is determined to be the illegal content.

[0289] In some embodiments, the determining module 930 is further configured to be at least one of the following: Using the first identification rule as an auxiliary filter, if the content to be detected is matched by the first identification rule, the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is less than the second boundary value of the target confidence level interval, the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is not matched by calling the violation content thesaurus through the violation content detection model, then the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon called by the violation content detection model, and the second identification rule hits the content to be detected, then the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon called by the violation content detection model, the second identification rule does not hit the content to be detected, the normal content difficult example lexicon hits the content to be detected, and the violation content difficult example lexicon does not hit the content to be detected, then the content to be detected is determined to be non-violation content.

[0290] In some embodiments, the identification rule strategy is constructed based on a dynamic data source; the acquisition module 910 is further configured to: The process involves acquiring the dynamic data source, constructing at least one type of lexicon based on the dynamic data source, and determining the recognition rule strategy based on the at least one type of lexicon.

[0291] In some embodiments, the device further includes a training module.

[0292] In some embodiments, the illegal content detection model is trained by augmenting training samples; the training module is further configured to: Obtain the augmented training sample, which is labeled with a real label. The real label is used to indicate whether the augmented training sample contains illegal content. The enhanced training samples are input into the general illegal content detection model to obtain the prediction results output by the general illegal content detection model; Based on the prediction results and the true labels, a sample loss is obtained. The model parameters of the general illegal content detection model are trained using the sample loss to obtain the illegal content detection model.

[0293] In some embodiments, the device further includes a screening module.

[0294] In some embodiments, the filtering module is used for: Using the first identification rule as an auxiliary filter, the dynamic data source is filtered by combining the violation content recall model with the optimized violation content lexicon and the normal content difficult example lexicon to obtain an enhanced data source. The data concentration of violation content in the enhanced data source is higher than that in the dynamic data source. The enhanced data source is labeled to obtain the enhanced training samples used to train the illegal content detection model.

[0295] In some embodiments, the filtering module is further configured to: The violation content recall model calls the first identification rule to filter the dynamic data source, and retains the first part of the data source that was not hit by the first identification rule in the dynamic data source; The optimized illegal content terminology is called through the illegal content recall model to filter the first part of the data source, and the second part of the data source that was not matched by the optimized illegal content terminology is retained in the first part of the data source. The violation content recall model calls the normal content difficult example vocabulary to filter the second part of the data source, retains the third part of the data source that was not matched by the normal content difficult example vocabulary in the second part of the data source, and uses the third part of the data source as the enhanced data source.

[0296] In some embodiments, the apparatus further includes an iteration module.

[0297] In some embodiments, the optimized lexicon of prohibited content is a dynamically updated lexicon, and the iteration module is further used for: Obtain the aforementioned list of prohibited content terms; The vocabulary in the illegal content lexicon is expanded by at least one dimension using a large language model to obtain an expanded illegal content lexicon. At a preset time interval, at least one candidate word is collected from the dynamic data source, and the at least one candidate word is added to the expanded illegal content lexicon to obtain the optimized illegal content lexicon.

[0298] In some embodiments, the iteration module is further configured to: At each preset time interval, at least one candidate word is collected from the dynamic data source. The feature extraction model is used to extract features from each word in the expanded illegal content word library and the at least one candidate word to obtain the feature vectors corresponding to each word and the feature vectors corresponding to the at least one candidate word. The center feature vector of the expanded illegal content lexicon is obtained based on the feature vectors corresponding to each word, and the first distance between the feature vector corresponding to at least one candidate word and the center feature vector is calculated. If the first distance is less than the first distance threshold, the at least one candidate word is added to the expanded illegal content lexicon to obtain the optimized illegal content lexicon.

[0299] In some embodiments, the iteration module is further configured to: Calculate the second distance between the feature vector corresponding to each word and the central feature vector; If the second distance is less than the second distance threshold, the words corresponding to the second distance being less than the second distance threshold are removed from the expanded illegal content word library to obtain the optimized illegal content word library.

[0300] In some embodiments, the at least one dimension includes one or more of the following: Synonym dimension; Pinyin dimension; Word order reversal dimension; Word order reorganization dimension; Homophone substitution dimension; Emoji expansion dimension; Interference character insertion dimension; Context expansion dimension.

[0301] It should be noted that the specific limitations of the embodiments of the one or more content detection devices provided above can be found in the limitations of the content detection method and the training method of the recommendation model based on hierarchical heterogeneous hypergraphs mentioned above, and will not be repeated here. Each module of the above device can be implemented entirely or partially by software, hardware, or a combination thereof. Each module can be embedded in the processor of the computer device in hardware form or independent of the processor, or it can be stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0302] This application also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program; the processor is used to execute the computer program in the memory to implement the content detection method provided in the above-described method embodiments.

[0303] For example, Figure 8 This is a structural block diagram of a computer device provided in an exemplary embodiment. Optionally, the computer device is a server 1000.

[0304] Typically, server 1000 includes a processor 1001 and memory 1002.

[0305] Processor 1001 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1001 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1001 may also include a main processor and a coprocessor. The main processor, also known as the Central Processing Unit (CPU), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1001 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1001 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0306] The memory 1002 may include one or more computer-readable storage media, which may be non-transitory. The memory 1002 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1002 is used to store at least one instruction, which is executed by the processor 1001 to implement the content detection method provided in the above-described method embodiments.

[0307] In some embodiments, the server 1000 may optionally include an input interface 1003 and an output interface 1004. The processor 1001, memory 1002, and input interfaces 1003 and 1004 can be connected via a bus or signal lines. Various peripheral devices can be connected to the input interfaces 1003 and 1004 via a bus, signal lines, or a circuit board. The input interfaces 1003 and 1004 can be used to connect at least one input / output (I / O) related peripheral device to the processor 1001 and memory 1002. In some embodiments, the processor 1001, memory 1002, and input interfaces 1003 and 1004 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1001, memory 1002, and input interfaces 1003 and 1004 can be implemented on separate chips or circuit boards, and this application does not limit this.

[0308] Those skilled in the art will understand that the structure shown in the figure does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0309] In an exemplary embodiment, this application also provides a chip, which includes programmable logic circuits and / or computer instructions, and when the chip is run on a computer device, it is used to implement the content detection methods provided in the above-described method embodiments.

[0310] This application also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement the content detection method provided in the above-described method embodiments.

[0311] This application also provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the processor of the computer device to load and execute the content detection method provided in the above-described method embodiments.

[0312] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0313] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0314] Those skilled in the art will recognize that the functions described in the embodiments of this application in one or more of the above examples can be implemented using hardware, software, firmware, or any combination thereof. When implemented using software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer.

[0315] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A content detection method, characterized in that, The method includes: The content to be detected and the recognition rule strategy are obtained. The recognition rule strategy includes a first recognition rule and a second recognition rule. The recognition complexity of the second recognition rule is higher than that of the first recognition rule. Using the first identification rule as an auxiliary filter, if the content to be detected is not hit by the first identification rule, the content to be detected is predicted by the violation content detection model, and the confidence level of the content to be detected is output. If the confidence level of the content to be detected is within the target confidence level range, the second identification rule is used as an auxiliary filter, and the violation content detection model sequentially calls at least one type of thesaurus to determine that the content to be detected is violation content; The at least one type of thesaurus includes at least one of the following: a thesaurus of illegal content, a thesaurus of normal content with difficult examples, and a thesaurus of illegal content with difficult examples.

2. The method according to claim 1, characterized in that, When the confidence level of the content to be detected is within the target confidence level range, the second identification rule is used as an auxiliary filter, and the violation content detection model sequentially calls at least one type of thesaurus to determine that the content to be detected is the violation content, including at least one of the following: If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon sequentially called by the violation content detection model, the second identification rule does not hit the content to be detected, and the normal content difficult example lexicon does not hit the content to be detected, then the content to be detected is determined to be the violation content. If the confidence level of the content to be detected is within the target confidence level range, and the violation content detection model sequentially calls the violation content lexicon to hit the content to be detected, the second identification rule does not hit the content to be detected, the normal content difficult example lexicon hits the content to be detected, and the violation content difficult example lexicon hits the content to be detected, then the content to be detected is determined to be the violation content.

3. The method according to claim 2, characterized in that, The method further includes: If the confidence level of the content to be detected is greater than the first boundary value of the target confidence level interval, the content to be detected is determined to be the illegal content.

4. The method according to claim 2 or 3, characterized in that, The method further includes at least one of the following: Using the first identification rule as an auxiliary filter, if the content to be detected is matched by the first identification rule, the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is less than the second boundary value of the target confidence level interval, the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is not matched by calling the violation content thesaurus through the violation content detection model, then the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon called by the violation content detection model, and the second identification rule hits the content to be detected, then the content to be detected is determined to be non-violation content. If the confidence level of the content to be detected is within the target confidence level range, and the content to be detected is hit by the violation content lexicon called by the violation content detection model, the second identification rule does not hit the content to be detected, the normal content difficult example lexicon hits the content to be detected, and the violation content difficult example lexicon does not hit the content to be detected, then the content to be detected is determined to be non-violation content.

5. The method according to any one of claims 1 to 4, characterized in that, The identification rule strategy is constructed based on a dynamic data source; the method further includes: The process involves acquiring the dynamic data source, constructing at least one type of lexicon based on the dynamic data source, and determining the recognition rule strategy based on the at least one type of lexicon.

6. The method according to any one of claims 1 to 5, characterized in that, The illegal content detection model is trained by augmenting training samples; the method also includes: Obtain the augmented training sample, which is labeled with a real label. The real label is used to indicate whether the augmented training sample contains illegal content. The enhanced training samples are input into the general illegal content detection model to obtain the prediction results output by the general illegal content detection model; Based on the prediction results and the true labels, a sample loss is obtained. The model parameters of the general illegal content detection model are trained using the sample loss to obtain the illegal content detection model.

7. The method according to claim 6, characterized in that, The method further includes: Using the first identification rule as an auxiliary filter, the dynamic data source is filtered by combining the violation content recall model with the optimized violation content lexicon and the normal content difficult example lexicon to obtain an enhanced data source. The data concentration of violation content in the enhanced data source is higher than that in the dynamic data source. The enhanced data source is labeled to obtain the enhanced training samples used to train the illegal content detection model.

8. The method according to claim 7, characterized in that, The step of using the first identification rule as an auxiliary filter, and filtering the dynamic data source by combining the violation content recall model with the optimized violation content lexicon and the normal content difficult example lexicon, to obtain an enhanced data source includes: The violation content recall model calls the first identification rule to filter the dynamic data source, and retains the first part of the data source that was not hit by the first identification rule in the dynamic data source; The optimized illegal content terminology is called through the illegal content recall model to filter the first part of the data source, and the second part of the data source that was not matched by the optimized illegal content terminology is retained in the first part of the data source. The violation content recall model calls the normal content difficult example vocabulary to filter the second part of the data source, retains the third part of the data source that was not matched by the normal content difficult example vocabulary in the second part of the data source, and uses the third part of the data source as the enhanced data source.

9. The method according to claim 7 or 8, characterized in that, The optimized terminology for prohibited content is a dynamically updated terminology, and the method further includes: Obtain the aforementioned list of prohibited content terms; The vocabulary in the illegal content lexicon is expanded by at least one dimension using a large language model to obtain an expanded illegal content lexicon. At a preset time interval, at least one candidate word is collected from the dynamic data source, and the at least one candidate word is added to the expanded illegal content lexicon to obtain the optimized illegal content lexicon.

10. The method according to claim 9, characterized in that, The process involves collecting at least one candidate word from the dynamic data source at preset time intervals, adding the at least one candidate word to the expanded illegal content lexicon, and obtaining the optimized illegal content lexicon, including: At each preset time interval, at least one candidate word is collected from the dynamic data source. The feature extraction model is used to extract features from each word in the expanded illegal content word library and the at least one candidate word to obtain the feature vectors corresponding to each word and the feature vectors corresponding to the at least one candidate word. The center feature vector of the expanded illegal content lexicon is obtained based on the feature vectors corresponding to each word, and the first distance between the feature vector corresponding to at least one candidate word and the center feature vector is calculated. If the first distance is less than the first distance threshold, the at least one candidate word is added to the expanded illegal content lexicon to obtain the optimized illegal content lexicon.

11. The method according to claim 10, characterized in that, The method further includes: Calculate the second distance between the feature vector corresponding to each word and the central feature vector; If the second distance is less than the second distance threshold, the words corresponding to the second distance being less than the second distance threshold are removed from the expanded illegal content word library to obtain the optimized illegal content word library.

12. The method according to claim 9, characterized in that, The at least one dimension includes one or more of the following: Synonym dimension; Pinyin dimension; Word order reversal dimension; Word order reorganization dimension; Homophone substitution dimension; Emoji expansion dimension; Interference character insertion dimension; Context expansion dimension.

13. A content detection device, characterized in that, The device includes: The acquisition module is used to acquire the content to be detected and the recognition rule strategy. The recognition rule strategy includes a first recognition rule and a second recognition rule. The recognition complexity of the second recognition rule is higher than that of the first recognition rule. The prediction module is used to use the first identification rule as an auxiliary filter. When the content to be detected is not hit by the first identification rule, the module predicts the content to be detected through the violation content detection model and outputs the confidence level of the content to be detected. The determination module is used to determine the content to be detected as illegal content by using the second identification rule as an auxiliary filter when the confidence level of the content to be detected is within the target confidence level range, and by sequentially calling at least one type of thesaurus through the illegal content detection model. The at least one type of thesaurus includes at least one of the following: a thesaurus of illegal content, a thesaurus of normal content with difficult examples, and a thesaurus of illegal content with difficult examples.

14. A computer device, characterized in that, The computer device includes a processor and a memory, the memory storing a computer program that is loaded and executed by the processor to implement the content detection method as described in any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is loaded and executed by a processor to implement the content detection method as described in any one of claims 1 to 12.

16. A computer program product, characterized in that, The computer program product includes computer instructions stored in a computer-readable storage medium, from which a processor retrieves the computer instructions, causing the processor to load and execute them to implement the content detection method as described in any one of claims 1 to 12.

Images