Content security auditing method and device, electronic equipment and storage medium

Abstract

The invention provides a content security auditing method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence, and the method comprises the steps: if it is determined that a scene category of an input text belongs to an exemption security auditing scene set, sending the input text as a question and answer prompt word to a question and answer large model for answering; otherwise, when it is determined that the initial risk level of the input text is a low risk, inputting the input text into the security auditing model to determine the final risk level of the input text; and if the final risk level is low risk, sending the input text as a question and answer prompt word to the question and answer large model for answering. According to the method, the differentiated auditing processing capability of strict interception of high-risk content, intelligent rewriting of medium-risk content and rapid release of low-risk content based on the scene and the user portrait is realized in a high-concurrency scene, the balance between auditing efficiency and safety is effectively balanced, and the safety of the user is improved through double-layer auditing of a safety word library rule and a safety auditing model. And the auditing accuracy and flexibility are improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a content security review method, apparatus, electronic device, and storage medium. Background Technology

[0002] Existing solutions typically employ a "one-size-fits-all" approach when conducting content security audits, applying the same risk level standards and processing procedures to all types of content. For example, all user input is scanned for keywords, and the assessment is based on those keywords.

[0003] This static security audit mechanism has the following defects: (1) low audit efficiency, low-risk content is over-audited, and computing resources are consumed at the same time; (2) the judgment is not accurate enough, the user experience is damaged, and the rigid audit mechanism does not combine with the scenario to make differentiated processing, which makes the judgment not accurate enough. For example, in terms of political matters, the usual security review method regards the mention of sensitive people's names as sensitive questions and refuses to answer them. However, in the daily work of special scenarios, it is necessary to learn the speech of sensitive people (often need to quote the speech of sensitive people). In this scenario, the rigid security audit machine makes the audit over-audit and cannot be used, resulting in a very poor user experience.

[0004] In addition, current mainstream content security review solutions rely on manually preset security thesaurus and rule engines to identify sensitive content through string matching. This can lead to insufficient semantic understanding, as it can only recognize literal matching and cannot understand the contextual semantics, resulting in misjudgments or omissions.

[0005] In view of this, there is an urgent need to provide a new content security auditing method to make up for the shortcomings of existing security audits. Summary of the Invention

[0006] This invention provides a content security review method, apparatus, electronic device, and storage medium to address the shortcomings of existing security review mechanisms, such as low review efficiency, low review accuracy, and poor user experience.

[0007] This invention provides a content security review method, comprising: Determine the scenario category of the input text to be reviewed; If it is determined that the scenario category belongs to the set of scenarios exempt from security review, then the security review of the input text is skipped, and the input text is sent as a question-and-answer prompt to the question-and-answer big model for answering; Otherwise, determine the initial risk level of the input text, and if the initial risk level is low, input the input text into the security audit model to determine the final risk level of the input text based on the output of the security audit model; If the final risk level is low, the input text will be sent as a question-and-answer prompt to the question-and-answer model for response.

[0008] According to a content security review method provided by the present invention, determining the initial risk level of the input text includes: Determine the blacklist-weighted risk value and the whitelist-weighted exemption value of the input text; The initial risk level of the input text is determined based on the blacklist weighted risk value, the whitelist weighted exemption value, and the security weight of the user who input the input text. The security weight is determined based on the user profile of the input user.

[0009] According to a content security auditing method provided by the present invention, the blacklist weighted risk value and the whitelist weighted exemption value of the input text are determined based on the following steps: Determine all the words that make up the input text; Identify all blacklisted word segments that match any blacklisted word in the security lexicon among all the word segments, and assign the preset risk weight value of any blacklisted word to the blacklisted word segment that matches it; The preset risk weight values ​​of all the blacklisted words are summed to obtain the blacklisted weighted risk value of the input text. Determine all whitelisted word segments that match any whitelisted word in the security lexicon among all the word segments, and assign the preset exemption weight value of any whitelisted word to the whitelisted word segment that matches it; The preset exemption weight values ​​of all the whitelisted word segments are summed up to obtain the whitelist-weighted exemption value of the input text; The security terminology database includes at least a blacklist terminology database and a whitelist terminology database.

[0010] According to a content security review method provided by the present invention, the step of determining the initial risk level of the input text based on the blacklist weighted risk value, the whitelist weighted exemption value, and the security weight of the input user of the input text specifically includes: Determine the risk difference between the blacklist weighted risk value and the whitelist weighted exemption value; The initial risk score is obtained by weighting the risk difference using the safety weights. The initial risk score is compared with a preset risk threshold range to map the initial risk score to one of high risk, medium risk, or low risk, which serves as the initial risk level of the input text.

[0011] According to a content security review method provided by the present invention, the step of inputting the input text into a security review model to determine the final risk level of the input text based on the output of the security review model includes: The security audit model is used to perform semantic analysis on the input text to obtain supplementary blacklist and supplementary whitelist words extracted from the input text. The preset exemption weight values ​​of all the supplementary blacklist words are summed to obtain the blacklist weighted risk supplement value, and the blacklist weighted risk value is updated using the blacklist weighted risk supplement value; The preset exemption weight values ​​of all the supplementary whitelist word segments are summed to obtain the whitelist weighted exemption supplementary value, and the whitelist weighted exemption value is updated using the whitelist weighted exemption supplementary value; The final risk score is determined based on the updated weighted risk value of the blacklist, the updated weighted exemption value of the whitelist, and the security weight of the input user. Based on the final risk score, the final risk level of the input text is determined.

[0012] According to a content security auditing method provided by the present invention, before obtaining the initial risk score, the method further includes: Monitor hot events related to the input text to determine whether to trigger a decision on the outbreak of emerging sensitive words; If it is determined that the emerging sensitive word outbreak decision is triggered, then a temporary compensation coefficient corresponding to the emerging sensitive word outbreak decision is determined; The temporary compensation coefficient is used to compensate for the initial risk score and / or the final risk score.

[0013] According to a content security review method provided by the present invention, when the initial risk level is determined to be high risk or the final risk level is determined to be high risk, a preset input interception mechanism is triggered; If the initial risk level is determined to be medium risk or the final risk level is determined to be medium risk, then the input text is rewritten by removing blacklisted words using a content rewriting model to obtain the rewritten text. The rewritten text is used as new input text and sent to the large language model for response.

[0014] According to a content security review method provided by the present invention, the security thesaurus is constructed based on the following cold start method: Send any text to multiple pre-defined general-purpose large language models; Receive candidate words and risk attributes of each candidate word from the output of the general large language model for any text; the candidate words include candidate blacklisted words and candidate whitelisted words; If the proportion of the same candidate word in the general large language model reaches a preset first threshold, then the candidate word and its risk attributes are added to the safe word library. If the proportion of the same candidate word in the general language model is less than the first threshold and greater than or equal to the preset second threshold, then manual review is triggered to determine whether the candidate word should be added to the secure word library based on the results of the manual review.

[0015] According to the content security review method provided by the present invention, the security review model is trained based on the following steps: If the number of training samples in the training corpus reaches a preset training quantity threshold, the initial general model is trained using the training corpus to obtain the security audit model. The training samples in the training corpus are obtained by filtering candidate words output by each of the generalized large language models for the input text, and then generalizing the filtered candidate words to obtain question-answer pairs.

[0016] According to the content security review method provided by the present invention, the training corpus is constructed using the following steps: Send any text to multiple general-purpose large language models respectively, and receive the candidate words output by each of the general-purpose large language models for any text; Based on the candidate words received from each of the general large language models, the consensus degree of the general large language model for the risk judgment of any text is determined; Based on the consensus level, a data generalization generation ratio is determined, wherein the consensus level and the data generalization generation ratio are negatively correlated. Based on the data generalization generation ratio, a generalized large language model is used to generalize any text to generate a corresponding number of question-answer pairs, which are then stored in the training corpus.

[0017] According to a content security review method provided by the present invention, after training an initial general model using the training corpus, the method further includes: Verify the audit effectiveness of the security audit model; If the audit effect is determined to meet the preset audit conditions, the security audit model is deployed to replace the multiple general-purpose large language models used to construct the training corpus.

[0018] According to a content security review method provided by the present invention, after sending the input text as question-and-answer prompts to a large question-and-answer model for answering, the method further includes: Obtain the response content generated by the large language model in response to the question-answering prompt words; The responses are subject to security review to determine the risk level of the output. If the output risk level is high risk, then a preset fallback response will be output to the user. If the output risk level is medium risk, a content generation model is invoked to clean up the answer content, so as to generate a cleaned answer content and output it to the user. If the output risk level is low, the answer will be directly output to the user.

[0019] The present invention also provides a content security review device, comprising: a scene type discrimination unit, a review processing unit, and a risk level discrimination unit; The scene type discrimination unit is used to determine the scene category of the input text to be reviewed; The review and processing unit is used to skip the security review of the input text and send the input text as a question-and-answer prompt to the question-and-answer big model for answering when it is determined that the scenario category belongs to the set of scenarios exempt from security review. Otherwise, the risk level determination unit determines the initial risk level of the input text, and if the initial risk level is low, the input text is input into the security audit model to determine the final risk level of the input text based on the output of the security audit model; If the final risk level is low, the review and processing unit will send the input text as a question-and-answer prompt to the question-and-answer model for response.

[0020] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the content security auditing method described above.

[0021] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the content security auditing method as described above.

[0022] The content security review method, device, electronic device, and storage medium provided by this invention achieve differentiated review processing capabilities in high-concurrency scenarios, including strict interception of high-risk content based on scenarios and user profiles, intelligent rewriting of medium-risk content, and rapid release of low-risk content. This effectively balances review efficiency and security, and improves the accuracy and flexibility of review through a two-layer review process of security terminology rules and security review models. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0024] Figure 1 This is one of the overall flowcharts of the content security auditing method provided by the present invention.

[0025] Figure 2 This is the second schematic diagram of the overall process of the content security auditing method provided by the present invention.

[0026] Figure 3 This is a schematic diagram of the overall process for determining the initial risk level provided by the present invention.

[0027] Figure 4 This is a schematic diagram illustrating the calculation process of the blacklist weighted risk value and the whitelist weighted exemption value provided by the present invention.

[0028] Figure 5 This is a schematic diagram of the process for determining the initial risk level based on the user's security weight, provided by the present invention.

[0029] Figure 6 This is a schematic diagram of the final risk level determination process provided by the present invention.

[0030] Figure 7 This is a schematic diagram of the process for final risk score compensation based on hot events provided by the present invention.

[0031] Figure 8 This is one of the schematic diagrams of the overall process of constructing the cold start security thesaurus provided by the present invention.

[0032] Figure 9 This is the second schematic diagram of the overall process of constructing the cold start security thesaurus provided by the present invention.

[0033] Figure 10 This is a schematic diagram of the overall process for constructing a training corpus provided by the present invention.

[0034] Figure 11 This is a flowchart illustrating the process of reviewing the answers in a large question-and-answer model provided by the present invention.

[0035] Figure 12 This is the third schematic diagram of the overall process of the content security auditing method provided by the present invention.

[0036] Figure 13 This is a schematic diagram of the architecture of the content security review system provided by the present invention.

[0037] Figure 14 This is a structural schematic diagram of the content security review device provided by the present invention.

[0038] Figure 15 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0040] It should be noted that, in the description of this invention, the terms "comprising," "including," or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0041] The terms "first," "second," etc., used in this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more.

[0042] Currently, content security moderation technology plays a crucial role in artificial intelligence applications, especially in large-scale model interaction scenarios. Most existing content security moderation solutions employ static moderation strategies, applying the same moderation standards and processes to all user input. For example, simple rule engines based on keyword matching cannot understand contextual semantics, easily leading to misjudgments and missed judgments; while performing deep model moderation on all input would excessively consume computational resources, resulting in low moderation efficiency.

[0043] Especially in certain business scenarios, such as when employees of important departments or large enterprises need to quote the speeches of sensitive individuals in their studies and work, rigid review mechanisms will treat them all as sensitive content and block them, which seriously affects the availability of the system and the user experience.

[0044] To address the technical problems of rigid review strategies in existing technologies, which fail to balance review efficiency, security, and scenario adaptability, this invention provides a content security review method. This method aims to achieve precise interception of high-risk content and rapid release of low-risk content through scenario-based and hierarchical intelligent review of input content, thereby significantly improving review efficiency and user experience while ensuring content security. The following section combines... Figures 1-15 This invention describes the content security auditing method, apparatus, electronic device, and storage medium provided by the present invention.

[0045] This invention provides a content security auditing method (hereinafter referred to as the security auditing method or security review method), the execution subject of which can be a content security auditing system (hereinafter referred to as the system), which can be deployed on a server, cloud computing platform or other electronic devices with corresponding computing capabilities. This security review method can be widely applied in various application scenarios that require interaction with large language models, such as intelligent customer service, enterprise knowledge bases, and public information publishing platforms.

[0046] Figure 1 This is one of the flowcharts illustrating the content security auditing method provided by this invention, such as... Figure 1 As shown, the method may include the following steps: First, determine the scenario category of the input text to be reviewed. This step is the foundation for subsequent differentiated and scenario-based review. The input text to be reviewed can be any form of text content entered by the user through various interactive interfaces, such as questions posed by users to intelligent question-answering systems, manuscripts submitted on content creation platforms, or comments posted in social applications.

[0047] Scenario categories are classifications based on the specific business environment, application context, or intent type of the input text. In one embodiment, scenario categories may include government information disclosure Q&A, internal corporate training, e-commerce customer inquiries, general knowledge Q&A, recommendation Q&A, or weather inquiries. There are various ways to determine scenario categories. For example, one can determine the source of the input text; input text from a specific government application programming interface (API) can be categorized as a government information disclosure Q&A scenario. In another embodiment, natural language processing techniques can be used, such as a pre-trained intent recognition model or text classification model, to analyze the content of the input text itself to determine its scenario category.

[0048] Next, it is determined whether the scenario category belongs to the set of scenarios exempt from security review. The set of scenarios exempt from security review is a predefined and configured collection containing several scenario categories deemed to have extremely low risk or requiring no input security review. The purpose of setting up this set is to quickly route obviously safe interaction requests, forming an efficient exemption review path, thereby avoiding unnecessary review resource consumption and improving the overall response speed and processing efficiency of the system. For example, scenario categories such as pre-reviewed recommended questions or weather queries with clear interaction intent and extremely low risk can be configured in this set.

[0049] If the scenario category of the input text is determined to belong to the set of scenarios exempt from security review, the security review of the input text is skipped, and it is sent as a question-and-answer prompt to the question-and-answer model for response. This is a fast processing path in the method of this invention. When the system determines that the scenario category of the input text exists in the set of scenarios exempt from security review, it means that the compliance risk of the input text is extremely low, and there is no need to enter the subsequent complex security review process. Therefore, the system will directly skip the input-side security review process for the input text and send it as a question-and-answer prompt to the backend question-and-answer model.

[0050] A question-answering big model refers to a generative large language model responsible for reasoning based on user-inputted question-answering prompts and generating the final response. It can be a well-known general-purpose large language model or a domain-specific large model obtained by further fine-tuning these general models on domain-specific data. Upon receiving question-answering prompts, this big model performs deep semantic understanding and logical reasoning to generate corresponding responses to the user's request.

[0051] As a standard review path in the method of this invention, if the scenario category of the input text does not belong to the set of scenarios exempt from security review, such as the input text belonging to a general knowledge Q&A scenario or a government affairs learning scenario, the system will initiate a hierarchical review process to ensure the compliance of the content, specifically including: First, determine the initial risk level of the input text. The initial risk level is a quick, preliminary risk assessment of the input text, primarily aimed at rapidly filtering out text with clearly defined risk levels. The initial risk level can be divided into multiple levels, such as high risk, medium risk, and low risk. Determining the initial risk level can be based on a computationally low-cost review mechanism, such as a rule-matching engine based on a security vocabulary database. This engine matches the input text against a pre-defined blacklist (containing a set of explicitly prohibited words) and a whitelist (containing a set of words allowed in specific scenarios), and provides an initial risk judgment based on the matching results. For example, input text containing high-risk politically sensitive terms can be initially classified as high-risk, while input text that does not match any blacklisted words can be initially classified as low-risk.

[0052] If the initial risk level is determined to be low, the input text is fed into the security audit model to determine the final risk level of the input text based on the output of the security audit model.

[0053] This step embodies the layered review concept of the present invention, which is to conduct a second in-depth review of texts that are initially determined to be low-risk, in order to prevent potential, obscure risk content that is difficult to identify through simple rules from being overlooked.

[0054] A security auditing model can be a model with deep semantic understanding capabilities. Its core function is to perform deeper semantic analysis on the input text to identify risks that are difficult to detect through literal matching, such as illegal intentions expressed through metaphors, variations, and homophones. This security auditing model can be one or more large language models trained with security-domain corpora.

[0055] The system sends input text initially deemed low-risk to the security audit model. Upon receiving the input text, the security audit model performs a comprehensive semantic security analysis and outputs a more accurate audit result. Based on this result, the system determines the final risk level of the input text. For example, a sentence initially judged as low-risk may have its final risk level revised to medium or high risk if the security audit model identifies hidden offensive or discriminatory semantics; if the security audit model finds no potential risks, its final risk level will be confirmed as low risk.

[0056] Finally, if the final risk level is determined to be low, the system will send the input text as a question-and-answer prompt to the question-and-answer model for response. Only when an input text passes both the initial review and the deep semantic review, and its final risk level is confirmed to be low, will the system determine that the input text is safe and can be processed. At this point, the system will send the input text as a question-and-answer prompt to the backend question-and-answer model for reasoning and response. For texts with a final risk level of medium or high risk, the system will trigger subsequent risk handling strategies, such as directly intercepting and returning a warning message to the user, or rewriting the input content for compliance, etc., which will be described in detail in subsequent embodiments.

[0057] This invention constructs a scenario-adaptive hierarchical content security review mechanism. On the one hand, by establishing a set of scenarios exempt from security review, it opens up an efficient processing path for clearly secure interaction requests, significantly improving review efficiency and system response speed. On the other hand, through a two-stage review process combining initial risk level determination and deep security model review, it conducts a detailed screening of input content in non-exempt scenarios from shallow to deep, ensuring the accuracy and comprehensiveness of the review, effectively making up for the shortcomings of traditional single review methods in semantic understanding, and effectively achieving a balance between review security, efficiency, and scenario-based user experience.

[0058] Figure 2 This is the second schematic diagram of the overall process of the content security review method provided by this invention, see reference. Figure 2 This demonstrates a more specific implementation process of the security review method provided by the present invention, which mainly includes four core steps: Step S1 involves a security terminology review of the input text. Its core objective is to conduct a rapid and efficient initial risk screening. In step S1, after receiving the input text to be reviewed, the system first utilizes a security terminology review engine to quickly match the text. The security terminology can include a blacklist (e.g., explicitly prohibited words like those related to terrorism and violence) and a whitelist (e.g., permitted proper nouns in specific business scenarios). Through this rule-based rapid screening, the system can quickly identify and filter out text containing obviously prohibited content.

[0059] Step S2: Security review of the input text by the security review model. For input text that has passed the security thesaurus review (e.g., text initially determined to be low-risk or potentially risky), the system will initiate a deep review, such as sending the input text to one or more security review models. These security review models possess deep semantic understanding capabilities, enabling them to identify potential risks that cannot be detected through literal matching, such as malicious intent expressed through homophones, metaphors, variations, or complex sentence structures. Through deep semantic analysis, the system can make more accurate judgments about the security of the input text.

[0060] Step S3: Dynamic Risk Classification and Response Strategy Formulation. After integrating the review results from Steps S1 and S2, the system will perform a final risk classification on the input text, such as dividing it into three levels: high risk, medium risk (suspected), and low risk (normal). Subsequently, the system will formulate and implement corresponding response strategies based on different risk levels. For example: for high-risk text, it will be directly intercepted and a compliance prompt will be returned to the user; for medium-risk text, an intelligent rewriting mechanism can be activated for harmless processing; and for text ultimately determined to be low-risk, it will be approved to enter the subsequent question-and-answer process, that is, it will be sent to the question-and-answer model as a question-and-answer prompt.

[0061] Step S4 involves security review of the question-and-answer model's generated and output content. This step establishes a second layer of security: after the question-and-answer model generates responses based on secure input text, these responses are not immediately returned to the user but undergo security review. The system can reuse the security lexicon review capabilities from the first stage and the security review model capabilities from the second stage to perform a comprehensive security check on the output content generated by the model. This aims to prevent potential "illusions" or unintentionally generated content that violates security standards during the content generation process. Only after the output content passes this stage of review will it be presented to the user.

[0062] The content security review method provided by this invention achieves differentiated review processing capabilities in high-concurrency scenarios, including strict interception of high-risk content based on scenarios and user profiles, intelligent rewriting of medium-risk content, and rapid release of low-risk content. It effectively balances review efficiency and security, and improves the accuracy and flexibility of review through a two-layer review of security terminology rules and security review models.

[0063] Based on the foregoing embodiments, in order to more clearly illustrate how the system determines the initial risk level of the input text, the present invention provides a specific implementation method.

[0064] Specifically, this invention provides a method for rapid and accurate preliminary risk assessment of input text during the security dictionary review stage. This method employs a weighted judgment scheme that integrates dictionary attributes and user profiles.

[0065] Figure 3 This is a schematic diagram of the overall process for determining the initial risk level provided by the present invention, as follows: Figure 3 As shown, the weighted judgment scheme may include, but is not limited to, the following steps: Step S11: Matching determination of the scene category of the input text; Step S12: Calculation of the weighted risk value for the blacklist and the weighted exemption value for the whitelist; Step S13: Initial risk level assessment based on user profile and security terminology; Step S14: Initial risk level determination and task classification processing.

[0066] Step S11 is a preprocessing step of classifying the input text into scenarios, while steps S12 and S13 are the core steps of calculating risk scores and making level determinations. Step S14 is to perform subsequent diversion operations based on the determination results.

[0067] As an optional embodiment, the present invention provides a specific implementation method for determining the initial risk level of input text, which mainly includes, but is not limited to, the following steps: First, determine the blacklist-weighted risk value and the whitelist-weighted exemption value of the input text. The calculation of both can be a parallel quantitative evaluation method.

[0068] A blacklist-weighted risk value is a numerical measure of the potential risk contained in input text. In one embodiment, the system can determine this value using a secure word auditing engine. This engine matches the input text against a pre-defined blacklist, identifying one or more blacklisted words. Specifically, each blacklisted word in the blacklist is assigned a pre-defined risk weight, reflecting its risk intensity. After matching, the system sums the risk weights of all matched blacklisted words; the total sum constitutes the blacklist-weighted risk value of the input text.

[0069] The whitelist-weighted exemption value is a numerical value used to measure the degree of compliance or exemption contained in the input text. Its determination method is similar to the calculation of the blacklist-weighted risk value. The system matches the input text against a pre-defined whitelist word library, identifying one or more whitelisted words in the text. Each whitelisted word in the library also has a pre-defined exemption weight value, which reflects the exemption capability of the whitelisted word in a specific context. The system sums the exemption weight values ​​of all matching whitelisted words, and the sum constitutes the whitelist-weighted exemption value of the input text.

[0070] Then, based on the blacklist-weighted risk value, the whitelist-weighted exemption value, and the security weight of the input user, the initial risk level of the input text is determined. The security weight of the input user is determined based on the user profile of the input user; specifically, it is a numerical coefficient used to characterize the individual user's risk profile, which can be understood as a quantitative representation of the user's trustworthiness or historical behavioral risk tendency.

[0071] A user profile is a structured dataset that records and describes a user's multi-dimensional characteristics. This can include, but is not limited to, basic user information (such as occupation), historical behavioral data (such as past questions and violation records), and content preferences. In one embodiment, the system can dynamically build and update the profile database for each user by analyzing their historical interaction records. For example, for a user who has frequently submitted compliant content historically and has never triggered a high-risk interception, the system can assign a higher security weight (indicating a lower risk propensity) based on their user profile. Conversely, for a user with multiple violation records, the system will assign a lower security weight (indicating a higher risk propensity). In this way, the security weight becomes a quantitative representation of the user profile in terms of its risk dimension, enabling the review strategy to dynamically adapt to the risk levels of different users.

[0072] Specifically, the system first performs a comprehensive numerical calculation by subtracting the weighted risk value from the blacklist and the weighted exemption value from the whitelist to obtain a base score that reflects only the risk level of the text content itself. Then, the system uses the input user's security weights to adjust or correct this base score, resulting in an initial risk score that considers both content risk and user risk. Finally, by comparing this initial risk score with one or more preset risk thresholds, the score is mapped to a corresponding risk level, such as high risk, medium risk, or low risk; this is the initial risk level of the input text.

[0073] This invention, through the introduction of weighted calculations using blacklists and whitelists, combined with security weights based on user profiles for comprehensive judgment, abandons the simple rule-matching logic of traditional review methods. It not only enables a more refined quantitative assessment of the risk inherent in the text content itself but also allows for differentiated review of the same content submitted by users with different risk profiles. This makes the initial risk level determination more accurate, flexible, and personalized, significantly improving the accuracy of the review process and providing a more reliable basis for subsequent decision-making. For example, for input containing sensitive words but with compliant intent, such as a staff member in an important department quoting a sensitive person in a work setting, this method can accurately classify it as low-risk through scene recognition and a high user security weight, thereby avoiding false blocking and greatly improving the user experience.

[0074] Based on the foregoing embodiments, in order to more clearly illustrate how the system calculates the blacklist weighted risk value and the whitelist weighted exemption value from the original input text through a standardized process, this embodiment of the invention provides a specific calculation method.

[0075] Figure 4This is a schematic diagram illustrating the calculation process of the blacklist weighted risk value and the whitelist weighted exemption value provided by the present invention, as follows: Figure 4 As shown, the blacklist-weighted risk value and the whitelist-weighted exemption value of the input text are determined based on the following steps: First, all the words that make up the input text are identified. This involves preprocessing the original input text to transform it into a sequence of basic units suitable for word matching. In one embodiment, standard Natural Language Processing (NLP) word segmentation tools, such as dictionary-based or statistical model-based segmenters, can be used to segment the input text (a string) into a sequence of multiple words or phrases. These segmented basic units are the words. In another embodiment, to achieve more comprehensive matching and avoid missed matches due to word segmentation errors, the system can also treat all possible substrings of the input text as word units to be matched.

[0076] Secondly, identify all blacklisted word segments that match any blacklisted word in the security lexicon, and assign the preset risk weight value of any blacklisted word to the matching blacklisted word segment.

[0077] This step aims to identify all risky words in the input text and obtain their corresponding risk levels. The safety vocabulary is a centralized, structured knowledge base that includes at least a blacklist and a whitelist of words.

[0078] Specifically, the system iterates through all the word segments obtained in the previous step and performs precise or fuzzy matching on each segment against a pre-defined blacklist of words in the security lexicon. All word segments that successfully match entries in the blacklist are identified as blacklisted words. Each blacklisted word in the blacklist is pre-configured with a numerically preset risk weight value, which quantifies the risk level of the word (e.g., ranging from 0 to 1). After identifying all blacklisted words, the system assigns the preset risk weight value of the corresponding blacklisted word to that blacklisted word segment.

[0079] Next, the preset risk weight values ​​of all blacklisted word segments are accumulated and used as the blacklisted weighted risk value of the input text.

[0080] The system will sum the preset risk weight values ​​assigned to all blacklisted words in the previous step, and the final sum is the blacklist-weighted risk value of the input text.

[0081] In a specific mathematical implementation, the blacklist weighted risk value The following formula can be used for calculation: ; The blacklist is weighted with a risk value; the higher the value, the more sensitive the input text is to be reviewed. T The higher the perceived level of risk, the better; Indicates input text T The first in k Each substring; It exists in the blacklist word database B The first in i One word (or phrase); It is a pre-set blacklist of words. B The first in i Numerical weights of each word; It is an implicit matching function used to compare input text. T a substring Blacklist of words B Blacklist words Whether they match. If they match, the value is 1; if they do not match, the value is 0.

[0082] Then, identify all whitelisted word segments that match any whitelisted word in the safe word library, and assign the preset exemption weight value of any whitelisted word to the matching whitelisted word segment.

[0083] Specifically, the system iterates through all word segments and matches each segment with a pre-defined whitelist of words in the security lexicon. All word segments that successfully match entries in the whitelist are identified as whitelisted word segments. Each whitelisted word in the whitelist is pre-configured with a numerical preset exemption weight value. This preset exemption weight value (e.g., ranging from 0 to 1) quantifies the compliance or exemption capability of the word in a specific context. After identifying all whitelisted word segments, the system assigns the preset exemption weight value of the corresponding whitelisted word to that whitelisted word segment.

[0084] Finally, the preset exemption weight values ​​of all the whitelisted word segments are summed to obtain the whitelisted weighted exemption value of the input text.

[0085] The system will sum the preset exemption weight values ​​assigned to all whitelisted word segments in the previous step, and the final sum will be used as the whitelist-weighted exemption value of the input text.

[0086] In a specific mathematical implementation, the whitelist weighted exemption value Sw can be calculated using the following formula: ; in, It refers to the white name word library W The first in m One word (or phrase); For a pre-defined whitelist of words W The first in m Numerical weights of each word; It is an implicit matching function used to compare input text. T A substring 𝑡𝑘 and a whitelist of words M Chinese and white noun words Whether they match. If they match, the value is 1; if they do not match, the value is 0.

[0087] Traditional review methods typically employ rule priority and Boolean logic, such as "if a blacklist is matched but no valid whitelist exemption is matched, it is considered a violation." This approach often struggles to make accurate judgments when dealing with complex text containing both blacklisted and whitelisted words. The proposed solution, which calculates risk and exemption values, upgrades the risk assessment of input text from traditional Boolean logic (i.e., match or miss) to a refined numerical calculation. This enables subsequent numerical risk balancing, resulting in more reasonable judgment logic, better judgment performance, and significantly improved accuracy and scientific rigor in initial risk level determination.

[0088] In the foregoing embodiments, the system has already been able to determine the blacklist-weighted risk value, the whitelist-weighted exemption value, and the security weight of the input text. Building upon this, this embodiment of the invention further provides a specific method for effectively integrating these three factors and ultimately determining the initial risk level.

[0089] Figure 5 This is a schematic diagram of the process for determining the initial risk level based on the user's security weight, as provided by the present invention. Figure 5 As shown, it mainly includes: Step S131: Determine the risk difference between the blacklist weighted risk value and the whitelist weighted exemption value.

[0090] The system directly performs a mathematical operation, typically a subtraction, between the determined blacklist weighted risk value and the whitelist weighted exemption value to obtain a value reflecting the net risk of the content; this is the risk difference. A positive risk difference indicates that the risk factors in the text exceed the exemption factors, while a negative risk difference indicates the opposite. This calculation method allows risky and exempted words that coexist in the text to cancel each other out and be weighed, which is more reasonable than simple Boolean logic judgments.

[0091] Step S132: Use safety weights to weight the risk difference to obtain an initial risk score.

[0092] The system uses the aforementioned security weights determined based on user profiles to weight the obtained risk differences. In one embodiment, this weighting process can be a simple multiplication operation. Through this step, a final quantitative score that combines content risk and user risk can be obtained, which is the initial risk score.

[0093] For example, for trusted users with low risk tolerance and high security weight, their input text containing slightly risky words may have a lower initial risk score after being weighted by their higher security weight. Conversely, for users with a history of violations and low security weight, even if they submit the same text, their initial risk score will be relatively high. This mechanism enables differentiated risk assessment of the same content submitted by different users.

[0094] Step S133: The initial risk score is compared with a preset risk threshold range to map the initial risk score to one of high risk, medium risk or low risk, as the initial risk level of the input text.

[0095] This step involves converting quantified risk scores into discrete risk levels that can be directly used by business logic for decision-making. The system pre-sets one or more risk thresholds, thus forming a risk threshold range. For example, two risk thresholds can be preset. and (in > These two risk thresholds can be set manually based on operational strategies, or they can be dynamically adjusted by algorithms based on historical data. These two risk thresholds can divide the entire score range into three parts.

[0096] The system will compare the initial risk score with the risk threshold range. If the initial risk score is greater than or equal to the higher risk threshold... If the initial risk score is less than the risk threshold, the system will classify the input text as high risk. But greater than or equal to the lower risk threshold If the initial risk score is less than the lower risk threshold, it is classified as medium risk (or suspected risk). If so, it is judged as low risk.

[0097] In a specific mathematical implementation, this determination process can be represented as: ; in, As the initial risk score, P For user security weight, Weighted risk values ​​for the blacklist. The weighted exemption value for the whitelist is determined by the following logic: like ≥ If the initial risk level of the input text is high, then the input text is determined to be high risk.

[0098] like ≤ < If the initial risk level of the input text is determined to be medium risk, then the input text is determined to be medium risk.

[0099] like < If the initial risk level of the input text is low, then the input text is determined to be low risk.

[0100] Through the above steps, this invention constructs a scientific and reasonable initial risk level determination model, which not only quantifies the inherent risk of text content, but also innovatively incorporates user-level risk assessment, realizing a dynamic review strategy tailored to each individual. This greatly improves the accuracy and flexibility of risk identification and effectively solves the problems of high misjudgment rate and poor scenario adaptability in traditional static review strategies.

[0101] In the aforementioned embodiments, for input text whose initial risk level is determined to be low, the system will send it to the security audit model for a second in-depth audit. Figure 6 This is a schematic diagram of the final risk level determination process provided by the present invention, such as... Figure 6 As shown, this embodiment of the invention further provides a specific method for determining the final risk level of the input text based on the output of a security audit model, including but not limited to the following steps: Step S21: Use the security audit model to perform semantic analysis on the input text to obtain supplementary blacklisted words and supplementary whitelisted words extracted from the input text.

[0102] When an initial low-risk input text is fed into a security auditing model, the model leverages its powerful language understanding capabilities to conduct a comprehensive analysis of the text's deep semantics, contextual relationships, metaphors, and homophonic variations. Based on this deep semantic analysis, the security auditing model can extract words or phrases from the input text that were missed in the initial review but actually possess risk or exemption attributes. These newly extracted words are known as supplementary blacklisted and supplementary whitelisted words. For example, for the input text "I want to plant you in a flowerpot," the initial review might classify it as low-risk because it didn't match any blacklisted words, but the security auditing model can understand the underlying semantic meaning of violence and threat, thus extracting "plant in a flowerpot" as a supplementary blacklisted word.

[0103] Step S22: Accumulate the preset exemption weight values ​​of the supplemented blacklist segmentation words to obtain the blacklist weighted risk supplement value, and use the blacklist weighted risk supplement value to update the blacklist weighted risk value.

[0104] The system assigns a pre-defined risk weight value to each supplementary blacklisted word segment. This pre-defined risk weight value can be derived from a broader risk knowledge base containing semantic information. Then, the system accumulates the risk weight values ​​of all supplementary blacklisted words to obtain a weighted risk supplement value for the blacklist. Finally, this weighted risk supplement value is used to update the weighted risk value obtained in the initial review stage, for example, by directly adding them together to obtain an updated weighted risk value for the blacklist.

[0105] Step S23: Accumulate the preset exemption weight values ​​of all supplementary whitelist word segments to obtain the whitelist weighted exemption supplementary value, and use the whitelist weighted exemption supplementary value to update the whitelist weighted exemption value.

[0106] Each supplementary whitelist word segment is pre-assigned an exemption weight value. By accumulating these exemption weight values, a whitelist weighted exemption supplementary value is obtained. Finally, the whitelist weighted exemption value obtained in the initial review stage is updated using this whitelist weighted exemption supplementary value, for example, by direct addition or weighted addition, to obtain an updated whitelist weighted exemption value.

[0107] Step S24: Determine the final risk score based on the updated blacklist weighted risk value, the updated whitelist weighted exemption value, and the security weight of the input user.

[0108] The system will use a method similar to that used to determine the initial risk score, but will use risk and exemption values ​​that have been corrected by the security audit model. Specifically, the system will first calculate the risk difference between the updated blacklist weighted risk value and the updated whitelist weighted exemption value, and then use the security weights of the input users to weight this difference, thereby obtaining a more accurate final risk score.

[0109] Step S25: Based on the final risk score, determine the final risk level of the input text. The system compares the final risk score obtained in the previous step with a preset risk threshold range. Through this comparison, the system maps the final risk score to one of high risk, medium risk, or low risk, which is the final risk level of the input text.

[0110] For example, a text that initially has a low risk score may have its final risk score exceed the threshold for medium or high risk after the security audit model adds new risk terms, thus its final risk level is revised.

[0111] This invention constructs a two-stage review process of "rapid initial review + in-depth verification." This process effectively supplements and corrects the initial review results through the deep semantic analysis capabilities of the security review model, accurately identifying subtle and deep-seated semantic risks and significantly reducing the false negative rate. This layered review mechanism, while ensuring efficiency, greatly improves the accuracy and reliability of content security review, ensuring that only truly safe content is allowed to proceed to the question-and-answer model.

[0112] Based on the aforementioned embodiments, in order to further enhance the dynamic response capability of the content security review system to sudden and time-sensitive risk events, this embodiment of the invention introduces a real-time risk compensation mechanism in the process of determining risk scores.

[0113] Figure 7 This is a schematic diagram of the process for final risk score compensation based on hot events provided by the present invention, as shown below. Figure 7 As shown, this document provides a process for dynamically adjusting risk scores by monitoring trending events, triggering compensation decisions, and utilizing compensation coefficients. This ensures the real-time nature of the review strategy and effectively addresses emerging sensitive terms arising from trending events. The process mainly includes, but is not limited to, the following steps: Step S51: Monitor hot events related to the input text to determine whether to trigger a decision on the outbreak of emerging sensitive words.

[0114] In one embodiment of the present invention, the system can continuously and automatically collect and analyze current hot topics from multiple information sources such as news websites, social media, and search engine hot lists through crawlers, API interfaces, etc.

[0115] The system analyzes these trending events to determine whether they contain or may generate emerging sensitive terms with risky connotations. For example, after a social hotspot occurs, related nicknames, slang, or derivative terms with negative or illegal connotations may quickly appear online. When the system detects that the frequency, scope of dissemination, or associated risk level of these emerging sensitive terms rises sharply in a short period of time and exceeds a preset outbreak threshold, it will make a decision to trigger an outbreak of emerging sensitive terms.

[0116] Step S52: If it is determined that a decision to trigger the outbreak of emerging sensitive words is made, then the temporary compensation coefficient corresponding to the decision to trigger the outbreak of emerging sensitive words is determined.

[0117] The temporary compensation coefficient is a numerical factor used to temporarily and globally enhance the sensitivity of risk assessment.

[0118] In one embodiment, the temporary compensation coefficient (denoted as Δ) can be a value greater than 0. Its specific magnitude can be dynamically determined based on factors such as the risk level and spread of emerging sensitive words. For example, a larger temporary compensation coefficient can be set for a sudden event with extremely high risk and a wide impact; conversely, a smaller temporary compensation coefficient can be set for a less risky event.

[0119] Optionally, this temporary compensation coefficient is time-sensitive. When the popularity of the relevant hot topic decreases or the risk is effectively controlled, the coefficient can be automatically or manually reset to 0.

[0120] Step S53: Use temporary compensation coefficients to compensate for the initial risk score and / or the final risk score.

[0121] This step is the process of translating temporary risk perception into actual adjustments to the audit strategy. During the calculation of the initial or final risk score, the system applies the temporary compensation coefficient determined in the previous step to dynamically compensate the original risk score calculation formula.

[0122] For example, when calculating the final risk score, the original formula might be: ; The new final risk score after introducing a temporary compensation coefficient Δ The calculation formula can be: Based on this, the final risk score The judgment logic is as follows: like ≥ If the initial risk level of the input text is high, then the input text is determined to be high risk.

[0123] like ≤ < If the initial risk level of the input text is determined to be medium risk, then the input text is determined to be medium risk.

[0124] like < If the initial risk level of the input text is low, then the input text is determined to be low risk.

[0125] By multiplying the risk score by a compensation factor (1+Δ) greater than 1, the system can temporarily amplify the risk assessment results of all input texts. This allows texts containing emerging sensitive words that may be on the edge of the risk threshold under normal review to have their risk scores effectively amplified, making them easier to be judged as medium or high risk, thereby triggering corresponding interception or handling mechanisms.

[0126] It should be noted that this compensation can be applied to the calculation of the initial risk score, the calculation of the final risk score, or both, to achieve the maximum risk control.

[0127] This invention, by introducing a real-time risk compensation mechanism based on hot topic monitoring, greatly enhances the dynamic adaptability and real-time performance of content security review strategies. This real-time risk compensation mechanism enables the system to move beyond relying solely on periodically updated static terminology databases and instead respond rapidly within minutes to rapidly changing public opinion risks on the internet. This effectively intercepts new types of illegal content arising from sudden events, significantly enhancing the risk prevention and control capabilities and robustness of the entire content security review system.

[0128] As an optional embodiment, after determining the initial or final risk level of the input text, the present invention further provides an implementation method for performing differentiated and refined processing on texts with different risk levels, so as to ensure content security, improve review efficiency, and take into account the user's interactive experience. The implementation method also includes the following processing steps for different risk levels: If the initial risk level is determined to be high or the final risk level is determined to be high, a preset input interception mechanism will be triggered.

[0129] When the system determines that the initial risk level of the input text is high during the initial review stage, or when it determines that the final risk level is high after a second in-depth review, it means that the input text contains explicit and unacceptable violations (for example, it hits the high-risk word list such as terrorism, or it is judged as a serious violation at the semantic level).

[0130] At this point, the system will immediately trigger a pre-set input interception mechanism. This mechanism will halt the current interaction process, preventing the high-risk input text from being sent to the backend question-and-answer model. Simultaneously, the system can return a pre-set, clear feedback message to the user, such as "Your input contains prohibited content and cannot be processed. Please re-enter according to the relevant rules."

[0131] Furthermore, as a dynamic risk management strategy, the system can also adjust the user profile based on the violation, for example, by increasing the violation count to dynamically reduce the corresponding security weight. P This allows for more stringent review of input during subsequent interactions.

[0132] In this embodiment of the invention, if the initial risk level is determined to be medium risk or the final risk level is medium risk, the input text is rewritten using a content rewriting model to remove blacklisted words and obtain the rewritten text.

[0133] Specifically, when the initial or final risk level of the input text is determined to be medium risk, it indicates that the text may contain some semantically ambiguous or inappropriate words in a specific context, but its overall risk has not yet reached the level that requires complete interception.

[0134] To mitigate potential risks without disrupting the user interaction flow, the system invokes a content rewriting model, which can be a generative language model finely tuned with specific instructions. Its core task is to perform compliance corrections on the input text.

[0135] Specifically, this content rewriting model analyzes the input text, identifies and removes or replaces blacklisted words or sensitive expressions, while preserving the user's original query intent as much as possible. This process is called blacklist removal and rewriting. After blacklist removal, the system obtains a rewritten text that is compliant with regulations and has reduced risk. For example, for a question containing mildly offensive words, "That product's design is terrible," the content rewriting model can rewrite it as "Please evaluate the design of that product and point out areas for improvement."

[0136] Finally, the rewritten text is sent as new input text to the large language model for response. After obtaining the compliant rewritten text, the system treats it as a completely new and safe input text, using it as a question-and-answer prompt and sending it to the backend question-and-answer model for normal reasoning and response. In this way, the user's query intent is satisfied after being rendered harmless, the interaction process continues, triggering a normal dialogue output process and avoiding a poor user experience caused by direct interception.

[0137] This invention, through its differentiated handling strategies targeting different risk levels, constructs a more intelligent and user-friendly risk response mechanism. For high-risk content, a decisive blocking strategy is adopted to ensure a safety baseline; for suspected medium-risk content, an innovative content rewriting mechanism is introduced for intelligent intervention, maximizing user satisfaction while ensuring compliance and enhancing user experience. This "strict blocking for high-risk content, intelligent rewriting for medium-risk content" approach significantly improves the system's flexibility and practicality compared to traditional "one-size-fits-all" blocking.

[0138] In the aforementioned embodiments, the system's auditing capabilities heavily rely on a high-quality, high-coverage security thesaurus. However, in practical applications, especially in the early stages of system deployment, the construction of a security thesaurus faces a severe cold-start problem. Traditional construction methods mainly rely on manual annotation and accumulation, or the expensive purchase of third-party thesauruses, which is not only costly and time-consuming, but also difficult to cope with emerging and ever-changing violations. To solve this technical challenge, this invention provides an innovative, data-driven cold-start construction scheme for a security thesaurus.

[0139] Figure 8 This is one of the schematic diagrams of the overall process of constructing the cold start security thesaurus provided by the present invention, such as... Figure 8 As shown, this embodiment of the invention provides a method to solve the cold start problem of building security capabilities from scratch through a dynamic, self-evolving closed-loop process. The core idea of ​​this closed-loop process is to cleverly utilize external general-purpose large-scale model resources when a professional security audit model is not yet available. This not only completes temporary audit tasks, but more importantly, simultaneously builds two core security assets: a security lexicon and a security model training corpus.

[0140] Specifically, the process begins with the cold start phase, during which the system performs the following steps: Step S61: Extraction and voting confirmation of the security lexicon based on a general large model. The system introduces multiple external general large models to jointly analyze and vote on the input text, thereby automatically extracting high-confidence sensitive words and writing them into the security lexicon, solving the problem of building the lexicon from scratch.

[0141] While S61 is being executed, step S62 can be initiated in parallel: the cold start phase of corpus construction and training of the security auditing model. This step fully utilizes the information from the multi-model voting process in S61, especially the disagreements among general large models, using it as valuable and high-value annotation signals to automatically generate massive amounts of supervised fine-tuning (SFT) corpus for training specialized security auditing models. This SFT corpus is continuously stored in the corpus. When the corpus accumulates to a certain scale, the system will initiate the training process of the specialized security auditing model.

[0142] As S62 progresses, once the professional security audit model is trained and reaches a usable standard, the entire cold start process can transition to step S63: auditing text security based on the generalization and language understanding capabilities of the security audit model. This means that the system will switch from a mode that relies on multiple, expensive, external general-purpose models for voting audits to a mode that uses its own, more cost-effective, and professional security audit model for independent audits.

[0143] In summary, the embodiments of the present invention provide a complete evolution path from "borrowing" to "self-reliance". Through two parallel construction tasks, steps S61 and S62, each real-time audit request is cleverly transformed into an enhancement of its own security capabilities. Ultimately, the automated and low-cost construction of the security thesaurus and professional security audit model is achieved, and the transition from the cold start stage to the normalized operation stage is smoothly completed.

[0144] Figure 9 This is the second schematic diagram of the overall process for constructing the cold start security thesaurus provided by this invention, as shown below. Figure 9 As shown, this invention provides a cold-start construction method for a secure thesaurus, specifically how to intelligently and automatically extract and confirm sensitive words through a multi-model voting mechanism. In a specific embodiment, the secure thesaurus is constructed based on the following cold-start method: Step S611: Send any text to multiple pre-defined general-purpose large language models. The system pre-integrates multiple (e.g., three or more) mainstream general-purpose large language models with strong general knowledge and secure understanding capabilities to form a model set. During system operation, for any received text (e.g., the user's historical query input), the system will distribute it in parallel and indiscriminately to each general-purpose large language model in the model set.

[0145] Step S612: Receive the candidate words and risk attributes of each candidate word from the output of each general large language model for any text.

[0146] Upon receiving input text, each general-purpose large language model performs a security analysis based on its own algorithm and training data, outputting words or phrases it identifies as potentially risky or exempt from liability. These words or phrases are called candidate words, which can include both blacklisted and whitelisted candidate words. Simultaneously, guided by prompts, the general-purpose large language model assigns a risk attribute to each candidate word. This risk attribute can include multiple dimensions of information, such as the type of risk (e.g., terrorism) and the confidence level in the output.

[0147] Step S613: If the proportion of the number of general large language models of the same candidate word reaches a preset first threshold, then the candidate word and its risk attributes are added to the safe word library.

[0148] After collecting the candidate word lists returned by all general-purpose language models, the system will perform aggregation and voting statistics. For a given candidate word (e.g., the words "ABC"), the system will count how many general-purpose language models have provided this candidate word.

[0149] If the proportion of models displaying a candidate word reaches or exceeds a preset first threshold (for example, in a configuration of three general-purpose language models, this threshold can be set to 2 / 3, meaning a majority of models reach a consensus), the system considers this a high-confidence, high-consensus result. In this case, the system will automatically add the candidate word and its corresponding risk attribute (for example, selecting the risk type recognized by the majority of models and using a weighted average of the confidence levels as its initial weight) directly to either the blacklist or whitelist of the safe word library. For example, if two or three of the three general-purpose language models determine a candidate word as a violation, that candidate word will be automatically extracted and added to the blacklist.

[0150] Step S614: If the proportion of the number of general large language models of the same candidate word is less than the first threshold and greater than or equal to the preset second threshold, then manual review is triggered to determine whether to add the candidate word to the security word library based on the manual review result.

[0151] Specifically, if the percentage of models with the same candidate word is less than a first threshold but reaches or exceeds a preset second threshold (for example, in a configuration of three general-purpose large language models, the first threshold is 2 / 3 and the second threshold can be set to 1 / 3), this usually corresponds to a situation where there is a large divergence among the general-purpose large language models (for example, the voting result is 1:2, i.e., complete divergence). In this case, it is generally difficult to make a reliable automated decision.

[0152] At this point, the system will trigger a manual review process. The candidate word, its related contextual information, and the judgment results of various models will be pushed to a manual review platform. Professional operations or review personnel will make the final judgment on the candidate word, determining whether it should be added to the safe word library and its inherent risk attributes. The results of the manual review will then be fed back to the system and written into the safe word library.

[0153] To more clearly illustrate the specific implementation details of the cold start construction scheme based on multi-model voting, especially the quantification method for handling different voting results and determining the risk attributes of words, this embodiment of the invention provides a more specific implementation.

[0154] In a specific embodiment, the process of adding candidate words to the security lexicon can be formally described by a voting function: ; in Representing three pre-defined generalized large language models, The voting process will comprehensively consider the consistency of risk type assessments and the confidence levels provided by each model. Specific voting procedures and handling methods may include: Scenario 1: All models consistently determine the risk type of candidate words.

[0155] When the voting result is 3:0, meaning all three general-purpose language models identify the same candidate word. These are prohibited keywords, and the risk type is... (Consistent): The system will automatically extract this candidate word. And it is directly added to the blacklist of the security terminology database. Its initial risk attribute weight. Based on the confidence level of all models and the preset basic weight of this risk type Perform calculations, for example, by using a weighted average: ; in, It is a candidate word The final risk attribute weights assigned; It is a collection of general large language models that participate in voting, such as those containing { }; M i It is a collection of general-purpose large language models The first in i One model; { M i | = } represents a subset containing all candidate words. Risk type determination Large language models; It is a large language model M i Regarding its judgment (i.e.) The risk type is The confidence level given is usually in the range of [0,1]. Ω It is a preset set of basic weights for risk types. Representative risk type (e.g., politically sensitive) corresponding basic weight values; |{ M i | = }| represents the number of models in the above subset.

[0156] When the voting result is 2:1 (i.e., two out of the three general language models determine it as a violation): the system will also automatically extract the candidate word and add it to the blacklist. Its initial risk attribute weight. ωj The calculation can be performed by weighted averaging based solely on the confidence levels of the two general large language models that provide violation judgments.

[0157] For example, if a general large language model and If a word is classified as a politically sensitive term with confidence levels of 0.8 and 0.9, its weight can be calculated as follows: ω j =(0.8* +0.9* ) / 2. Assume, If the value is 0.9, then the calculated initial risk attribute weight is... It is 0.855.

[0158] When the voting result is 1:2 (meaning only one general language model judges it as a violation, constituting a complete divergence): the system will forcibly trigger a manual review process. The candidate word, its context, and the judgment details of each general language model will be pushed to the manual review platform, where professionals will ultimately determine whether it should be added to the blacklist or whitelist and manually configure the corresponding risk attributes and weights for it.

[0159] Scenario 2: Different models make inconsistent judgments on the risk type of candidate words.

[0160] When the voting result is 3:0 (i.e., all three general language models determine it as a violation, but the risk types given are not exactly the same, for example, one determines it as politically related, and the other two determine it as pornographic): the system will still automatically extract the candidate word and add it to the blacklist. When determining its risk type, a "majority vote" principle can be used, that is, the risk type that is judged most frequently is selected as its final risk type. If each of the three risk types has one vote, then a confidence level can be selected. The risk type determined by the highest-ranking general language model is taken as the final risk type. Its weight is calculated similarly to the 3:0 situation in Scenario 1.

[0161] When the voting result is 2:1 (i.e., both general-purpose language models determine it as a violation, but with different risk types): the system will automatically extract the candidate word and add it to the blacklist. When determining its risk type, the risk type determined by the general-purpose language model with the higher confidence level among the two violation determinations can be selected. Its weight is calculated based on these two general-purpose language models that gave the violation determination.

[0162] When the voting result is 1:2: the same as the 1:2 situation in Scenario 1, manual review will be forcibly triggered.

[0163] This invention, through its multi-model voting-based cold-start construction scheme, innovatively solves the problems of difficult, manual, and time-consuming traditional security thesaurus construction. It cleverly utilizes the collective intelligence of existing general-purpose large models, automatically and efficiently building and expanding the security thesaurus from scratch through a voting mechanism. Furthermore, it handles disagreements through human-machine collaboration, ensuring the quality of the security thesaurus. This transforms the security thesaurus construction from "manual weekly updates" to "system minute-level response," achieving 100% automated cold-start construction of the security thesaurus and significantly reducing labor and time costs.

[0164] In the foregoing embodiments, the system can automatically construct a security lexicon during the cold start phase through a multi-model voting mechanism. To further reduce the high computational costs associated with long-term reliance on multiple external general-purpose models and to build a more professional and efficient proprietary security auditing capability, this embodiment of the invention also provides a method for automatically constructing training corpora and training a professional security auditing model during system operation, specifically including but not limited to the following steps: First, after determining that the number of training samples in the training corpus reaches a preset training quantity threshold, an initial general model is trained using the training corpus to obtain the security audit model.

[0165] The system will pre-define a training corpus, which will be used to continuously store high-quality data automatically generated during the cold start phase for model training. At the same time, the system will set a preset training quantity threshold (e.g., 1 million question-answer samples), which will serve as the trigger condition for starting model training.

[0166] During system operation, when the number of training samples accumulated in the training corpus reaches or exceeds the training quantity threshold, the model training process will be automatically triggered.

[0167] Generally, an initial general model is selected as the base model (or foundation model), and supervised fine-tuning is performed on this initial general model using all or part of the data in the training corpus. After sufficient supervised fine-tuning training, the trained model obtained is the security review model with professional content security review capabilities required by this invention.

[0168] The training samples in the training corpus are obtained by filtering candidate words output by each of the generalized large language models for the input text, and then generalizing the selected candidate words to obtain question-answer pairs.

[0169] This step details the source and generation method of high-quality training samples in the training corpus, and its core lies in the clever use of information from the multi-model voting process.

[0170] Specifically, during the cold start phase, for each input text, the system receives candidate words from the outputs of multiple general-purpose large language models. The system then filters and analyzes these candidate words from different models. This filtering is not merely a simple collection, but more importantly, an analysis of the consensus or disagreement among the models reflected in these candidate words.

[0171] After identifying valuable candidate words and their corresponding voting results (e.g., which candidate words are unanimously considered risk words, and which candidate words are disputed), the system uses one or more generalized large language models to generalize this information into question-answer pairs conforming to the SFT format. This generalization process is essentially a form of data augmentation and format conversion. For example, for an input text that is voted by multiple models to contain high-risk words, the system can automatically generate multiple question-answer pairs around the input text and the judgment results, such as "Input [original text], please determine its risk level and extract risk words" as the prompt, and "Risk level is high risk, risk words are [extracted risk words]" as the response.

[0172] In this way, the system cleverly transforms the intermediate process of multi-model voting into a large number of training samples with clear supervision signals, which are continuously stored in the training corpus. This process is fully automated, achieving zero manual annotation of the training corpus.

[0173] This invention innovatively constructs a closed-loop optimization process that accumulates corpora during operation and then autonomously trains the system after accumulation. This not only solves the industry challenges of constructing corpora required for training professional security models and the high cost of manual annotation, but also achieves automated and large-scale generation of high-quality training corpora by leveraging the generalization capabilities of general-purpose models. This enables the system to train a professional content security review model at low cost and high efficiency without affecting online services, laying a solid foundation for reducing reliance on multiple external general-purpose models, saving computing resources, and improving review efficiency.

[0174] In the foregoing embodiments, the system can construct a training corpus by generalizing candidate words selected during the multi-model voting process. To make the generation of training corpus more intelligent and efficient, and to specifically learn the divergence knowledge between models, this embodiment of the invention further provides a specific method for dynamically adjusting the corpus generalization strategy based on model consensus.

[0175] Figure 10 This is a schematic diagram of the overall process for constructing a training corpus provided by the present invention, as follows: Figure 10 As shown, the training corpus is constructed using the following steps: Step S621: Send any text to multiple general-purpose large language models respectively, and receive the candidate words output by each of the general-purpose large language models for any text.

[0176] This step is consistent with the initial steps of building a safe vocabulary during the cold start described above. It is the basis for obtaining the judgment criteria of multiple models. The system will distribute any text (which can be understood as a text sample used for model training) in parallel to multiple preset general-purpose large language models and collect the candidate word list returned by each general-purpose large language model.

[0177] Step S622: Based on the candidate words received from each general large language model, determine the consensus degree of the general large language model for the risk judgment of any text.

[0178] The consensus level can be defined by a specific mathematical formula. For example, when three general-purpose language models are configured, for an input text, if all three models give the same risk assessment (e.g., all consider it a violation, or all consider it compliant), the consensus level is considered the highest; if two models give the same assessment and the third model gives a different assessment, the consensus level is medium; if all three models give completely different assessments, or only one model gives a risk assessment, the consensus level is the lowest. In a specific implementation, the consensus level D can be defined as: D = {number of models with different voting results} / 3.

[0179] For example, a 3:0 vote (meaning all three general language models judge it as either illegal or compliant) has the highest consensus, and D may be quantified as 0 (indicating no disagreement); a 2:1 result has a medium consensus; and a 1:2 or 1:1:1 result has the lowest consensus, and D may be quantified as 1 (indicating high disagreement).

[0180] Step S623: Based on the consensus degree, determine a data generalization generation ratio, wherein the consensus degree is negatively correlated with the data generalization generation ratio.

[0181] This invention dynamically adjusts the data augmentation intensity based on the difficulty level determined by multiple generalized language models. The system pre-defines a mapping relationship, assigning different consensus levels to different data generalization generation ratios. This mapping relationship follows a core principle: the lower the consensus level (i.e., the greater the divergence among the generalized language models), the higher the data generalization generation ratio. The underlying logic is that samples with divergence among the generalized language models are typically boundary samples or difficult samples, possessing higher learning value for training more robust and accurate professional security auditing models. Therefore, these high-value samples require larger-scale data augmentation. For example, the following generalization strategy can be set: For unambiguous scenarios (high consensus), such as a voting result of 3:0, it indicates that the sample is simple and uncontroversial, and a lower generalization ratio of 1:30 can be used.

[0182] For scenarios with moderate ambiguity (moderate consensus), such as a voting result of 2:1, indicating some disagreement, a moderate generalization ratio of 1:100 can be adopted.

[0183] For highly ambiguous scenarios (low consensus), such as voting results of 1:2 or more dispersed, it indicates that the sample is a typical difficult sample and the highest generalization ratio of 1:500 should be used for large-scale data augmentation.

[0184] Step S624: Based on the data generalization generation ratio, use a general large language model to generalize any text to generate a corresponding number of question-answer pairs, and store them in the training corpus.

[0185] The system will, based on a determined data generalization ratio, invoke one or more generalized large language models to generate a corresponding number of formatted question-answer pairs around the input text and its corresponding multi-model voting results. For example, in a highly ambiguous scenario with a ratio of 1:500, the system will instruct the generalized large language model to generate 500 question-answer pairs that are semantically similar to the original input text but have different expressions, and store all these newly generated question-answer pairs in the training corpus (e.g., denoted as SafeFAQ).

[0186] This invention, through a consensus-based dynamic generalization strategy, significantly improves the intelligence and efficiency of training corpus construction. It automatically identifies and focuses on the most valuable "difficult" samples for data augmentation, thereby generating a training dataset with higher information density and enhanced model generalization ability within limited computing resources. This provides high-quality data assurance for training high-performance professional security auditing models.

[0187] In the foregoing embodiments, the system has already been able to train a professional security auditing model using an automatically constructed training corpus. To ensure that the newly trained security auditing model can truly replace the original voting auditing system composed of multiple general-purpose large language models, and to achieve the goals of saving computing resources and improving auditing efficiency, this embodiment of the invention further provides a closed-loop process for validating the effectiveness of the new model and deploying it for replacement.

[0188] In one specific embodiment, after training an initial general model using the training corpus, the method further includes the following steps: First, the effectiveness of the security audit model is verified. After the professional security audit model is trained, the system will not immediately deploy it to the production environment. Instead, a rigorous verification process will be initiated to objectively evaluate its performance.

[0189] In one embodiment, the verification process can employ parallel testing. The system simultaneously sends real-time audit data (or a reserved test dataset) to two audit systems: one is the existing voting audit system composed of multiple general-purpose large language models; the other is a newly trained, single-focused professional security audit model. The system collects and compares the audit results of these two systems on the same batch of data.

[0190] Evaluation metrics may include, but are not limited to, accuracy, precision, recall, F1 score, and error rate. For example, the system might compare the F1 score of the security audit model on the test set with the F1 score of a voting audit system composed of multiple general-purpose language models on the same test set, while also examining the error rate of the security audit model.

[0191] Then, after confirming that the obtained audit results meet the preset audit conditions, a secure audit model is deployed to replace the aforementioned general-purpose large language models used to build the training corpus. The system will only deploy and replace the model when the audit results of the newly trained secure audit model are confirmed to meet the preset audit conditions during the verification process.

[0192] The preset audit conditions are a series of quantitative standards used to determine whether a security audit model meets the requirements. These quantitative standards aim to ensure that the performance of the security audit model not only meets, but also exceeds, the original voting system. For example, the quantitative standard can be specifically defined as follows: the F1 score of the newly trained security audit model on the test set needs to be higher than the F1 score of the general voting model by a specified percentage (e.g., 10%), and its false positive rate must be lower than a very low value (e.g., 1%).

[0193] The system will deploy a security audit model only when the above-mentioned preset audit conditions are met. In the online audit process, this single, efficient, and professional security audit model will replace the original, costly voting audit system, which consists of multiple general-purpose language models.

[0194] If the verification results fail to meet the preset verification conditions, it indicates that the performance of the obtained security verification model has not yet met the standards. In this case, the system will not perform a replacement operation, but will continue to use the original voting verification system for online services. At the same time, the system will continue to use the divergence data from the voting process to generate new training data and conduct a new round of iterative training on the security verification model. This "training-verification-training" iterative process will continue until the performance of the newly obtained security verification model finally meets the standards.

[0195] This invention, through the aforementioned closed-loop process of verification and replacement, innovatively achieves an automated solution for the construction, effect evaluation, and deployment of a security audit model. This not only ensures that the final deployed security audit model has superior performance, but also achieves significant savings in computing resources and a substantial improvement in audit efficiency through a clear, data-driven replacement mechanism. It completes the strategic transformation from relying on external general-purpose large language models to using security audit models with proprietary professional capabilities.

[0196] In the foregoing embodiments, this invention primarily focuses on reviewing user input to ensure that the prompts sent to the question-and-answer model are safe and compliant. However, during the content generation process, the question-and-answer model may also produce non-compliant or inappropriate outputs due to its own "illusions" or knowledge boundary issues. To construct a more comprehensive content security protection system and form a closed loop of "head-to-tail" dual review, this invention further provides a method for secondary review and processing of the answers generated by the question-and-answer model. Figure 11 This is a flowchart illustrating the process of reviewing the answers in a large question-and-answer model, as provided by this invention. Figure 11 As shown, in this embodiment of the invention, after the question-and-answer model generates the answer content, it will perform a security review and execute differentiated output based on the review results, which mainly includes, but is not limited to, the following steps: First, obtain the response content generated by the large language model based on the question-answering prompts.

[0197] After the question-answering model receives secure question-answering prompts and completes reasoning, the system will intercept the generated, original answer content. This answer content will enter the subsequent review process before being sent to the user.

[0198] Then, the received responses undergo a security audit to determine their output risk level. The system can reuse the aforementioned security capabilities used to audit input text to perform a comprehensive security audit of the responses. In one embodiment, this audit process can also be a hierarchical workflow, that is, first, a fast matching is performed using a security thesaurus, then a deep semantic analysis is performed by a security audit model, and finally a comprehensive determination of an output risk level is made. This output risk level can also be divided into three levels: high risk, medium risk, and low risk.

[0199] (1) If the output risk level is high, a preset fallback response will be output to the user. Specifically, when the system determines that the output risk level of the response is high, it means that the content contains explicit illegal information. In this case, in order to avoid showing the user this harmful content, the system will directly discard the high-risk response and output a preset, universal, and absolutely safe fallback response to the user, such as "Sorry, I cannot answer this question at the moment. Please try another question."

[0200] (2) If the output risk level is medium risk, a content generation model is called to clean up the answer content, so as to generate a cleaned answer content and output it to the user.

[0201] This invention implements intelligent intervention measures for suspected non-compliant or inappropriate responses generated by the content generation model. When a medium-risk situation occurs, it indicates that the response may contain inappropriate words or semantic ambiguity. The system will then invoke a content generation model (which may be the same or a different model than the aforementioned content rewriting model used to rewrite the input text) to clean up the response. This cleanup aims to correct or remove inappropriate expressions while retaining its core, valuable information, ultimately generating a cleaned response that is compliant and appropriately expressed, which is then output to the user.

[0202] (3) If the output risk level is low risk, the answer content will be directly output to the user.

[0203] When the system determines that the output risk level of the response is low, it indicates that the content is safe, compliant, and reliable. At this point, the system will directly present the original response to the user, completing a full and secure interaction.

[0204] This invention, through its scheme of secondary review and tiered processing of the output content of the question-and-answer model, constructs a complete dual security barrier of "input review + output review." This not only prevents risky input from the user end but also effectively intercepts and corrects potential risky output from the question-and-answer model itself, thereby achieving rigorous monitoring of the entire human-computer interaction process. This dual review system significantly enhances the level of content security, ensuring that every piece of information ultimately delivered to the user is safe, compliant, and of high quality, thus providing solid technical support for building trustworthy artificial intelligence applications.

[0205] Figure 12 This is the third schematic diagram of the overall process of the content security review method provided by this invention. To more comprehensively illustrate the technical solution provided by this invention, the following will be combined with… Figure 12 The system flow shown, and Figure 13 The schematic diagram of the content security review system shown illustrates a comprehensive embodiment of the present invention in detail. For ease of description, the content security review system will be referred to as the system in this embodiment.

[0206] Next, refer to Figure 12 and Figure 13 As shown, the execution flow of the content security review method provided by this invention is fully demonstrated from the perspective of system operation. This execution flow is realized through a complex system that integrates scenario judgment, layered review, dynamic response and closed-loop optimization, ensuring the accuracy, efficiency and flexibility of content review.

[0207] In one specific embodiment, the content security review system adopted by this invention can employ a layered architecture design. This layered architecture encompasses an application layer, a service layer, and a storage layer, with each layer working collaboratively to ensure efficient and accurate content security review. The following section will discuss this system architecture in conjunction with... Figure 12 The process shown is explained in series.

[0208] I. The starting point of system architecture and process: input and configuration.

[0209] This process begins at the application layer, which serves as the interface between the system and external systems, as well as for manual configuration. This layer primarily includes, but is not limited to, the security audit access configuration module (F101), the security thesaurus configuration module (F102), the security corpus configuration module (F103), and the manual audit and security review strategy module (F104). F101 is responsible for receiving various types of content awaiting review. When a user enters a question, it receives the text and distributes it to the starting point of the review process, namely the input content scenario judgment stage. F102, F103, and F104 provide the basic rules, data, and policy configurations for the entire system. For example, operators can configure blacklists and whitelists and their weights through F102, configure corpus generalization strategies through F103, and formulate handling plans for different risk levels through F104. This configuration information is stored in databases such as the security policy library and security thesaurus in the storage layer (denoted as F301).

[0210] II. Process Branch 1: Fast Track (Exemption Scenarios).

[0211] The first step in process branch one is to determine the input content scenario. The system will call the scenario library in the storage layer to perform basic scenario matching and determination on the input text.

[0212] If the input text is determined to fall under a scenario that is exempt from security review, such as a recommended question scenario, the process will proceed through a fast track. The input text will skip subsequent input review stages and be directly sent to the question-answering model inference service. Finally, after output content review, the question-answering model output (including recommended questions) will be obtained.

[0213] III. Process Branch Two: Regular Review Channel (Non-Exempt Scenarios)

[0214] If the scenario is determined to be different, the process will proceed to the regular review channel. First, a security terminology review is performed, specifically by the security terminology review engine (denoted as F201) in the service layer. It loads the security terminology and user profile database in the storage layer F301 and performs a rapid weighted risk assessment on the input text.

[0215] If the review result determines that the violation is high-risk, the process will directly jump to the risk classification and response stage, and carry out interception and response (such as a fallback response).

[0216] If the review result is determined to be suspected (i.e., medium risk), the process will also jump to the risk classification and response stage, and take measures such as rewriting the input content in compliance with regulations using a large model.

[0217] If the audit result is passed (i.e., low risk), the process will proceed to the next level of audit: large model security audit.

[0218] IV. Differentiated in-depth auditing under cold start and non-cold start conditions.

[0219] During the large-scale model security audit process, the system will select a security audit model.

[0220] During the non-cold start phase, the system will directly call the pre-trained security audit model (denoted as F203) in the service layer to perform deep semantic auditing on the input text.

[0221] During the cold start phase, the system will call the model set of the general large language model in the service layer (denoted as F202, which includes large model 1, large model 2, and large model 3) to review the input text.

[0222] During the voting process, the system determines whether there is any ambiguity in the vote. If not (i.e., high consensus), the vote result is directly used as the result of in-depth review. Simultaneously, these high-consensus vote results are used for sensitive word writing into the thesaurus and security corpus generalization, updating the security thesaurus and security corpus in the storage layer, respectively. If so (i.e., there is disagreement), manual review of the vote result is triggered. The result of manual review not only determines the review result of the current text but is also used to update the security thesaurus and security corpus, forming a closed-loop optimization through human-machine collaboration. All intermediate voting process data can be stored in the voting information database.

[0223] V. Risk Management and Reasoning Generation.

[0224] After in-depth review, the process enters the risk classification and response phase. The system will dynamically classify the risk and formulate response strategies based on the final risk level. Text determined to be normal (low risk) will be sent to the question-answering big data model inference service. Text determined to be suspected (medium risk) will first undergo compliance rewriting by the big data model, and the rewritten compliant text will then be sent to the inference service. Text determined to be non-compliant (high risk) will trigger a fallback response.

[0225] VI. Output review and final delivery.

[0226] After the question-answering model completes its reasoning, its output answers are not directly delivered to the user. Instead, they first enter the output content review stage. This stage utilizes the capabilities of the security thesaurus and the large model's security review to further examine the model's output answers.

[0227] The system will determine whether the model's output is acceptable. If it is, the output of the large question-and-answer model will be sent directly to the user. If not (for example, if it is judged to be suspected or in violation), mechanisms such as removing blacklisted words and rewriting them will be triggered, or a fallback reply will be used to ensure the absolute safety of the final output.

[0228] This comprehensive embodiment demonstrates, through a complete, closed-loop system process and a layered system architecture, how the present invention achieves full-lifecycle content security management from input to output, and from cold start to normalization. This solution is not only technically sound but also possesses high flexibility, intelligence, and self-evolution capabilities, effectively addressing complex and ever-changing content security challenges.

[0229] Figure 14 This is a structural schematic diagram of the content security review device provided by the present invention, as shown below. Figure 14 As shown, it mainly includes, but is not limited to, scenario type determination unit 1, review and processing unit 2, and risk level determination unit 3, wherein: The scene type discrimination unit 1 is mainly used to determine the scene category of the input text to be reviewed.

[0230] The review and processing unit 2 is mainly used to skip the security review of the input text when it is determined that the scenario category belongs to the set of scenarios exempt from security review, and send the input text as a question and answer prompt to the question and answer model for answering.

[0231] Otherwise, the risk level determination unit 3 determines the initial risk level of the input text, and if the initial risk level is low, the input text is input to the security audit model to determine the final risk level of the input text based on the output of the security audit model; If the final risk level is low, the review and processing unit 2 will send the input text as a question-and-answer prompt to the question-and-answer model for answering.

[0232] It should be noted that the content security review device provided by the present invention can execute the content security review method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0233] The content security review device provided by this invention achieves differentiated review processing capabilities in high-concurrency scenarios, including strict interception of high-risk content based on scenarios and user profiles, intelligent rewriting of medium-risk content, and rapid release of low-risk content. It effectively balances review efficiency and security, and improves the accuracy and flexibility of review through a two-layer review of security terminology rules and security review models.

[0234] Figure 15 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 15As shown, the electronic device may include a processor 1510, a communications interface 1520, a memory 1530, and a communication bus 1540, wherein the processor 1510, communications interface 1520, and memory 1530 communicate with each other via the communication bus 1540. The processor 1510 can invoke logical instructions in the memory 1530 to execute a content security audit method. This method includes: determining the scenario category of the input text to be audited; if the scenario category belongs to an exemption from security audit scenario set, then skipping the security audit of the input text and sending the input text as a question-and-answer prompt to a question-and-answer model for answering; otherwise, determining the initial risk level of the input text, and if the initial risk level is low, inputting the input text into a security audit model to determine the final risk level of the input text based on the output of the security audit model; if the final risk level is low, then sending the input text as a question-and-answer prompt to the question-and-answer model for answering.

[0235] Furthermore, the logical instructions in the aforementioned memory 1530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0236] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, when the program instructions are executed by a computer, the computer is able to execute the content security auditing method provided in the above embodiments, the method including: determining the scenario category of the input text to be audited; if the scenario category is determined to belong to the set of scenarios exempt from security audit, then skipping the security audit of the input text and sending the input text as a question-and-answer prompt to the question-and-answer big model for answering; otherwise, determining the initial risk level of the input text, and if the initial risk level is low risk, inputting the input text into the security auditing model to determine the final risk level of the input text according to the output of the security auditing model; if the final risk level is low risk, then sending the input text as a question-and-answer prompt to the question-and-answer big model for answering.

[0237] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the content security auditing method provided in the above embodiments. The method includes: determining the scenario category of the input text to be audited; if the scenario category is determined to belong to a set of scenarios exempt from security audit, then skipping the security audit of the input text and sending the input text as a question-and-answer prompt to a question-and-answer model for answering; otherwise, determining the initial risk level of the input text, and if the initial risk level is low risk, inputting the input text into a security auditing model to determine the final risk level of the input text based on the output of the security auditing model; if the final risk level is low risk, then sending the input text as a question-and-answer prompt to a question-and-answer model for answering.

[0238] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0239] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A content security auditing method, characterized in that, include: Determine the scenario category of the input text to be reviewed; If it is determined that the scenario category belongs to the set of scenarios exempt from security review, then the security review of the input text is skipped, and the input text is sent as a question-and-answer prompt to the question-and-answer big model for answering; Otherwise, determine the initial risk level of the input text, and if the initial risk level is low, input the input text into the security audit model to determine the final risk level of the input text based on the output of the security audit model; If the final risk level is low, the input text will be sent as a question-and-answer prompt to the question-and-answer model for response.

2. The content security auditing method according to claim 1, characterized in that, Determining the initial risk level of the input text includes: Determine the blacklist-weighted risk value and the whitelist-weighted exemption value of the input text; The initial risk level of the input text is determined based on the blacklist weighted risk value, the whitelist weighted exemption value, and the security weight of the user who input the input text. The security weight is determined based on the user profile of the input user.

3. The content security auditing method according to claim 2, characterized in that, The blacklist-weighted risk value and the whitelist-weighted exemption value of the input text are determined based on the following steps: Determine all the words that make up the input text; Identify all blacklisted word segments that match any blacklisted word in the security lexicon among all the word segments, and assign the preset risk weight value of any blacklisted word to the blacklisted word segment that matches it; The preset risk weight values ​​of all the blacklisted words are summed to obtain the blacklisted weighted risk value of the input text. Determine all whitelisted word segments that match any whitelisted word in the security lexicon among all the word segments, and assign the preset exemption weight value of any whitelisted word to the whitelisted word segment that matches it; The preset exemption weight values ​​of all the whitelisted word segments are summed up to obtain the whitelist-weighted exemption value of the input text; The security terminology database includes at least a blacklist terminology database and a whitelist terminology database.

4. The content security auditing method according to claim 2, characterized in that, The step of determining the initial risk level of the input text based on the blacklist weighted risk value, the whitelist weighted exemption value, and the security weight of the user inputting the input text specifically includes: Determine the risk difference between the blacklist weighted risk value and the whitelist weighted exemption value; The initial risk score is obtained by weighting the risk difference using the safety weights. The initial risk score is compared with a preset risk threshold range to map the initial risk score to one of high risk, medium risk, or low risk, which serves as the initial risk level of the input text.

5. The content security auditing method according to claim 4, characterized in that, The step of inputting the input text into a security audit model to determine the final risk level of the input text based on the output of the security audit model includes: The security audit model is used to perform semantic analysis on the input text to obtain supplementary blacklist and supplementary whitelist words extracted from the input text. The preset exemption weight values ​​of all the supplementary blacklist words are summed to obtain the blacklist weighted risk supplement value, and the blacklist weighted risk value is updated using the blacklist weighted risk supplement value; The preset exemption weight values ​​of all the supplementary whitelist word segments are summed to obtain the whitelist weighted exemption supplementary value, and the whitelist weighted exemption value is updated using the whitelist weighted exemption supplementary value; The final risk score is determined based on the updated weighted risk value of the blacklist, the updated weighted exemption value of the whitelist, and the security weight of the input user. Based on the final risk score, the final risk level of the input text is determined.

6. The content security auditing method according to claim 5, characterized in that, Before obtaining the initial risk score, the process also includes: Monitor hot events related to the input text to determine whether to trigger a decision on the outbreak of emerging sensitive words; If it is determined that the emerging sensitive word outbreak decision is triggered, then a temporary compensation coefficient corresponding to the emerging sensitive word outbreak decision is determined; The temporary compensation coefficient is used to compensate for the initial risk score and / or the final risk score.

7. The content security auditing method according to claim 1, characterized in that, If the initial risk level is determined to be high risk or the final risk level is determined to be high risk, a preset input interception mechanism is triggered; If the initial risk level is determined to be medium risk or the final risk level is determined to be medium risk, then the input text is rewritten by removing blacklisted words using a content rewriting model to obtain the rewritten text. The rewritten text is used as new input text and sent to the large language model for response.

8. The content security auditing method according to claim 3, characterized in that, The security thesaurus was built using the following cold start method: Send any text to multiple pre-defined general-purpose large language models; Receive candidate words and risk attributes of each candidate word from the output of the general large language model for any text; the candidate words include candidate blacklisted words and candidate whitelisted words; If the proportion of the same candidate word in the general large language model reaches a preset first threshold, then the candidate word and its risk attributes are added to the safe word library. If the proportion of the same candidate word in the general language model is less than the first threshold and greater than or equal to the preset second threshold, then manual review is triggered to determine whether the candidate word should be added to the secure word library based on the results of the manual review.

9. The content security auditing method according to claim 8, characterized in that, The security audit model is trained based on the following steps: If the number of training samples in the training corpus reaches a preset training quantity threshold, the initial general model is trained using the training corpus to obtain the security audit model. The training samples in the training corpus are obtained by filtering candidate words output by each of the generalized large language models for the input text, and then generalizing the filtered candidate words to obtain question-answer pairs.

10. The content security auditing method according to claim 9, characterized in that, The training corpus is constructed using the following steps: Send any text to multiple general-purpose large language models respectively, and receive the candidate words output by each of the general-purpose large language models for any text; Based on the candidate words received from each of the general large language models, the consensus degree of the general large language model for the risk judgment of any text is determined; Based on the consensus level, a data generalization generation ratio is determined, wherein the consensus level and the data generalization generation ratio are negatively correlated. Based on the data generalization generation ratio, a generalized large language model is used to generalize any text to generate a corresponding number of question-answer pairs, which are then stored in the training corpus.

11. The content security auditing method according to claim 9, characterized in that, After training an initial general model using the training corpus, the method further includes: Verify the audit effectiveness of the security audit model; If the audit effect is determined to meet the preset audit conditions, the security audit model is deployed to replace the multiple general-purpose large language models used to construct the training corpus.

12. The content security auditing method according to any one of claims 1-11, characterized in that, After sending the input text as question-answering prompts to the question-answering model for response, the process also includes: Obtain the response content generated by the large language model in response to the question-answering prompt words; The responses are subject to security review to determine the risk level of the output. If the output risk level is high risk, then a preset fallback response will be output to the user. If the output risk level is medium risk, a content generation model is invoked to clean up the answer content, so as to generate a cleaned answer content and output it to the user. If the output risk level is low, the answer will be directly output to the user.

13. A content security review device, characterized in that, include: Scenario type identification unit, review and processing unit, and risk level identification unit; The scene type discrimination unit is used to determine the scene category of the input text to be reviewed; The review and processing unit is used to skip the security review of the input text and send the input text as a question-and-answer prompt to the question-and-answer big model for answering when it is determined that the scenario category belongs to the set of scenarios exempt from security review. Otherwise, the risk level determination unit determines the initial risk level of the input text, and if the initial risk level is low, the input text is input into the security audit model to determine the final risk level of the input text based on the output of the security audit model; If the final risk level is low, the review and processing unit will send the input text as a question-and-answer prompt to the question-and-answer model for response.

14. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the content security auditing method as described in any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the content security auditing method as described in any one of claims 1 to 12.

Images