Risk perception based content security detection system self-optimization method

Abstract

The application discloses a risk perception-based content security detection system self-optimization method, and relates to the field of artificial intelligence. The steps include: calculating endogenous indexes generated in the running process of the system per unit time, and performing normalization processing on the endogenous indexes, wherein the endogenous indexes at least include a rule invalidation index and a variant confrontation entropy; calculating a risk score based on the normalized endogenous indexes and preset weight parameters; if the risk score is greater than a first preset threshold and the rule invalidation index is greater than a second preset threshold, starting a rule base updating process; if the risk score is greater than the first preset threshold and the rule invalidation index is less than or equal to the second preset threshold, starting a model training process. The scheme dynamically evaluates the system risk by real-time calculation of endogenous indexes and triggers a double-channel incremental update, significantly shortens the update delay of rules and models, and realizes the rapid evolution of defense knowledge.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a self-optimization method for a content security detection system based on risk perception. Background Technology

[0002] Existing content security detection systems based on deep learning and rule matching generally adopt a static maintenance model. Knowledge updates heavily rely on manual intervention and fixed-cycle maintenance. The iteration of rule bases and models requires a long process, including false negative detection, manual analysis, rule writing, and offline training. The average response time exceeds 48 hours, resulting in significant blind spots in the system's protection against rapidly evolving new attacks.

[0003] Meanwhile, existing systems lack the ability to perceive their own operational status in real time, such as the degree of rule failure, model confidence drift, and attack variant complexity. They cannot proactively trigger defense upgrades at the initial stage of risk and can only respond passively after the attack has caused substantial harm.

[0004] Therefore, there is an urgent need to build a self-optimization method for a content security detection system that can perceive risks in real time based on the system's own operating status and adaptively update the knowledge base. Summary of the Invention

[0005] This invention provides a self-optimization method for a risk-aware content security detection system. It aims to address the technical problem that existing content security detection systems in business scenarios such as social media platforms and government systems, due to their static maintenance model, cannot perceive risks in real time based on the system's own operational status and adaptively update the knowledge base, resulting in delayed responses to new attacks and the existence of protection blind spots.

[0006] To achieve the above objectives, this application adopts the following technical solution: Firstly, a self-optimization method for a content security detection system based on risk perception is provided, the method comprising: Calculate the endogenous indicators generated during the operation of the system per unit time, and normalize the endogenous indicators. The endogenous indicators include at least the rule failure index and the variant adversarial entropy. The risk score is calculated based on the normalized endogenous indicators and preset weight parameters. If the risk score is greater than the first preset threshold and the rule failure index is greater than the second preset threshold, then the rule base update process is initiated. If the risk score is greater than the first preset threshold and the rule failure index is less than or equal to the second preset threshold, then the model training process is started.

[0007] Secondly, a content security detection system is provided, which adopts a layered processing architecture, including: The rules engine layer is used to quickly screen input content and block explicit violations; The semantic awareness layer is used to perform deep semantic analysis on suspected fragments output by the rule engine layer and to triage them according to the risk confidence level. The full-model decision layer is used to make decisions on the fuzzy fragments output by the semantic perception layer through multi-model fusion.

[0008] The third aspect provides a method for content security detection, including: The text to be detected is input into the rule engine layer for rapid screening to obtain the first processing result, which includes allowing, blocking or suspected fragments. The suspected fragment is input into the semantic perception layer for semantic analysis, the risk confidence is calculated, and a second processing result is obtained based on the risk confidence. The second processing result includes releasing, intercepting, or pending fragments. The undetermined segment is input into the full model decision layer for comprehensive analysis, and the final decision on whether to intercept or allow passage is output.

[0009] The fourth aspect provides a self-optimizing device for a risk-aware content security detection system, comprising: An endogenous index calculation unit is used to calculate the endogenous indexes generated during the operation of the system per unit time, and to normalize the endogenous indexes. The endogenous indexes include at least the rule failure index and the variant adversarial entropy. The risk scoring unit is used to calculate the risk score based on the normalized endogenous indicators and preset weight parameters. The decision-making and update unit is used to initiate a rule base update process when the risk score is greater than a first preset threshold and the rule failure index is greater than a second preset threshold; and to initiate a model training process when the risk score is greater than the first preset threshold and the rule failure index is less than or equal to the second preset threshold.

[0010] The self-optimization method provided in this application dynamically assesses system risk and triggers dual-channel incremental updates by real-time calculation of four endogenous indicators: rule failure index, variant adversarial entropy, behavioral anomaly density, and confidence drift. This significantly shortens the update latency of rules and models, enabling rapid evolution of defensive knowledge. Simultaneously, the provided content security detection system adopts a three-tiered hierarchical architecture: the rule engine layer intercepts explicit violations, while the semantic perception layer and the full-model decision layer process content according to confidence levels, effectively reducing the overall computational load. This invention, with its self-optimization method at its core, drives the detection system's adaptive iteration, significantly improving the real-time performance, resource efficiency, and dynamic evolution capabilities of content security protection. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a self-optimization method for a content security detection system based on risk perception, provided by an embodiment of the present invention. Figure 2 This is a schematic diagram of a content security detection system structure provided by an embodiment of the present invention; Figure 3 This is a flowchart of a content security detection method provided according to an embodiment of the present invention; Figure 4 This is a schematic diagram of a self-optimizing device for a content security detection system based on risk perception, provided by an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Example 1

[0016] Figure 1This is a flowchart illustrating a risk-aware self-optimization method for a content security detection system according to an embodiment of the present invention. This method is applicable to social media platforms and government systems, especially for high-risk scenarios characterized by frequent new variant attacks, complex contextual semantic ambiguity, and the need for real-time responses. This method can be executed by a risk-aware self-optimization device for the content security detection system. This device can be implemented in hardware and / or software and can be integrated into the content security detection system. Figure 1 As shown, the method includes: S110. Calculate the endogenous indicators generated during the operation of the system per unit time, and normalize the endogenous indicators. The endogenous indicators include at least the rule failure index and the variant adversarial entropy.

[0017] To enable self-optimization of the content security detection system, the system randomly selects a unit time period t (e.g., a random day, hour, or week) during operation according to a preset random sampling strategy (e.g., based on uniform random distribution, Poisson process, or dynamic triggering by system load status). Endogenous metrics generated during system operation are collected only within this selected unit time period t. No metrics are calculated for non-selected time periods to reduce system computational overhead and prevent attackers from predicting fixed calculation cycles.

[0018] The endogenous indicators refer to quantitative parameters calculated based on internal data such as the system's own operation logs, model outputs, and manual review results. They originate from internal access records, model confidence outputs, rule engine interception records, and manual review logs, and can be used to characterize the system's operational health status, external risk pressures, and the degree of degradation in its defense capabilities.

[0019] In this embodiment, the endogenous metrics include at least the rule failure index and the variant adversarial entropy. The rule failure index is used to quantify the proportion of erroneous decisions made by the rule engine, reflecting the timeliness of the rule base and providing guidance for the optimization of the detection system. The variant adversarial entropy is used to measure the complexity of attack variants, reflecting the diversity of attackers' strategies to bypass rules.

[0020] Furthermore, the rule failure index is calculated based on the interception records and manual review logs of the rule engine layer within the content security detection system. It is the ratio of erroneous decisions to the total processing volume, where erroneous decisions include erroneous interception and missed release.

[0021] The specific statistical method is as follows: Total processing volume: The total number of all items that trigger rule judgments within a unit of time (such as daily, weekly or monthly) is automatically collected by the system backend; Error interception: By combining the interception logs recorded by the system with manual spot checks or subsequent audits, the number of cases that were incorrectly judged is confirmed; Missed releases: These are usually identified through post-event spot checks, customer feedback, anomaly monitoring, or special audits, which count the number of items that should have been intercepted but were released.

[0022] Specifically, the formula for calculating the rule failure index (REI) is as follows:

[0023] Here, the subscript t represents the t-th randomly selected time period (e.g., the t-th selected hour or day), and is not a consecutive time sequence. The system only counts the values ​​of each indicator within the randomly selected time period. The statistical method for other endogenous indicators is the same as described above, and is also performed only within the randomly selected time period.

[0024] This indicates the number of incorrectly blocked items per unit of time, i.e., the number of items that should have passed but were mistakenly blocked by the rules; This indicates the number of items that were missed within a unit of time, i.e., the number of items that should have been blocked but were mistakenly judged as passed by the rules; This represents the total processing volume per unit of time, which is the total number of all items that have been judged by the rules within that time period.

[0025] The rule failure index is calculated by summing the number of erroneous interceptions and the number of missed passes within a randomly selected unit of time, and then dividing by the total processing volume during the same period. The lower the index, the more accurate and reliable the rule execution; the higher the index, the greater the risk of rule failure, requiring timely optimization and adjustment.

[0026] For example, in the context of AI dialogue business in the security field, when REI>0.3, it indicates insufficient rule coverage, and when REI>0.4, it triggers an emergency rule update.

[0027] Furthermore, the variant's adversarial entropy is determined in the following way: Obtain the variant word set and the base word corresponding to each variant word, calculate the edit distance from each variant word to the corresponding base word, determine the frequency of each edit distance in the variant word set, and determine the variant anti-entropy based on the frequency of occurrence according to the entropy calculation formula.

[0028] Variant entropy is a measure of variant attack complexity based on edit distance distribution. It reflects the diversity of attackers' strategies for bypassing the system and is used to guide the optimization of defense strategies. The higher the entropy value, the more dispersed the variant forms are, and the greater the risk of failure of traditional rules.

[0029] The formula for calculating variant adversarial entropy (VAE) is as follows:

[0030] in, It is the edit distance (Lewinstein distance) from the variant word to the base word. To edit distance Frequency of occurrence in the variant set.

[0031] In the calculation of variant adversarial entropy, this application introduces the concept of entropy from information theory into the quantitative evaluation of attack strategy complexity. By statistically analyzing the probability distribution of the edit distance (Lewinstein distance) between variant words and the base word, the magnitude of the entropy value reflects the degree of dispersion of variant morphologies. When variant morphologies are concentrated in a few edit distances, the entropy value is low, indicating a singular attack strategy. For example, for "transfer," only "zhu" appears. When a transliteration variant of "n zhàng" appears, the rule engine can quickly cover it by supplementing the phonetic-shape matching rules. Conversely, when the variant forms are widely distributed across multiple edit distances, the entropy value is high, indicating that the attacker has used diverse transformation methods. For example, when "murder" has multiple variant forms and homophones such as "open melon", "open claw", and "open quack", traditional rules are difficult to exhaustively list them. In this case, the system determines that the semantic model needs to be activated based on the high entropy value to ensure the accuracy of recognition.

[0032] For example, in AI dialogue business scenarios in the security field, when A value greater than 2.0 indicates the presence of highly complex variant attacks, requiring the introduction of a model to ensure accurate identification.

[0033] Furthermore, to enhance the accuracy of risk perception and the robustness of model health assessment, the endogenous indicators also incorporate behavioral anomaly density and confidence drift as core monitoring dimensions: Behavioral anomaly density is determined based on the ratio of the number of abnormal user sessions detected per unit time to the total number of sessions. It is used to characterize the abnormal access pressure faced by the system. Abnormal sessions meet any of the following conditions: high-frequency submission or access during abnormal time periods. Confidence drift, determined based on the degree of deviation between the distribution of the model's output confidence on a given day and the baseline during the stable period, is used to characterize the change in the model's ability to understand the current input.

[0034] Specifically, User Activity Density (UAD) is defined as the proportion of non-standard user sessions to the total number of sessions per unit of time. It is used to quantify the abnormal access pressure the system experiences per unit of time. This metric relies solely on system access logs for real-time calculation, without the need for external threat intelligence or additional probes, and can be used for early risk warning.

[0035] The calculation formula is as follows:

[0036] in: This represents the number of sessions that are judged as non-routine within a randomly selected time window t; This indicates the total number of sessions within the same time window.

[0037] This calculation process relies solely on the session identifier, request timestamp, and request frequency fields in the system access log, and can complete the statistics within seconds.

[0038] Unregular sessions meet any of the following conditions: High-frequency submissions: The frequency of single-session requests exceeds the threshold N (e.g., default N=5 times / minute), mainly used to identify automated scripts, web crawlers, or probing attack behaviors; Unscheduled access: Sessions initiated during off-peak business hours (e.g., 02:00-08:00 local time). Used to identify malicious behavior that attempts to circumvent normal monitoring periods.

[0039] The two types of behaviors mentioned above occur with a low probability in normal user interactions, and their increased density often indicates that the system faces potential risks.

[0040] Monitoring the density of abnormal behaviors aims to provide early warnings of external threats and improve the timeliness of risk perception. The goal is to directly capture abnormal signals at the behavioral level, detecting probing attacks before violations occur, shifting the identification window from "post-incident discovery" to "in-process perception." Furthermore, by combining the criteria of "high-frequency submissions" and "access during unusual periods," it effectively distinguishes between normal business fluctuations and potential attacks, reducing false alarm rates and enhancing the accuracy of perception.

[0041] The confidence drift (CID) is defined as the degree of deviation between the model prediction confidence distribution of the released text on the current day and the confidence baseline during the system's stable period. It uses only the model's own output distribution characteristics for health monitoring, thus achieving "unsupervised perception" of the model's state.

[0042] This degree of offset includes a two-dimensional measure: Mean deviation: Reflects the systematic drift of the model's overall confidence level. When the model's ability to understand the current batch of text changes, the average level of its output confidence level will fluctuate accordingly.

[0043] Volatility measure: Reflects changes in the stability of model predictions. When the model makes uncertain judgments about parts of the text, the dispersion of the confidence level will increase.

[0044] The calculation formula is as follows:

[0045] in, The average confidence level of the texts released on that day is calculated using the following formula:

[0046] l : Indicates the text that is allowed, that is, the text that is ultimately determined by the system to be "allowed". Only the confidence scores of these texts are used when calculating the confidence drift.

[0047] t : Indicates a randomly selected time period.

[0048] For the first The confidence level of the model prediction for each release text. This represents the total number of documents released that day.

[0049] in, This is the probability value output by the semantic awareness layer model for a given text, indicating how confident the model is that the text is normal / risk-free. The model is typically a binary classification or threshold-based scoring model, outputting a value between 0 and 1. The closer to 1, the more confident the model is that the text is safe; the closer to 0, the more confident the model is that the text is illegal.

[0050] The baseline confidence level during the system's stable period is defined as the average of historical confidence levels within a sliding window T (e.g., the past 7 days), reflecting the expected performance of the system under normal business conditions. The calculation method is the same as above.

[0051] The standard deviation of the confidence level for the day reflects the dispersion of the confidence level distribution.

[0052] in, Indicates the semantic perception layer's number of days. The confidence level of the released text (a value between 0 and 1 from the output of the semantic awareness layer).

[0053] This represents the average confidence level of all released texts on that day. See [link to calculation method] for details. The calculation method.

[0054] This reflects the mean deviation of the confidence level for the day. This reflects the volatility of the confidence level on that day.

[0055] This calculation process relies solely on the confidence values ​​output by the model inference, requiring no additional labeled data, and can complete statistical updates in real time within a streaming processing architecture.

[0056] Confidence drift enables real-time monitoring and adaptive early warning of model health. When a model faces new semantic expressions or data distribution shifts, the output confidence level may systematically decrease or fluctuate more significantly. By acquiring this change data, the system can be provided with a quantitative indicator of the model's adaptability. An increase in the drift indicates a decline in understanding capabilities, requiring timely intervention. Traditional solutions rely on manual annotation or A / B testing, which has a lag of several hours. In contrast, confidence drift can detect changes in business data within a minute-level window, avoiding widespread false positives and false negatives caused by model rigidity. When the confidence drift remains low, the system maintains efficient processing; when it increases significantly, a risk warning is automatically triggered.

[0057] S120. The risk score is calculated based on the normalized endogenous indicators and preset weight parameters.

[0058] Since the four endogenous indicators have different dimensions and distribution characteristics, direct weighting and fusion would lead to the dominant indicator in terms of dimension dominating the risk score, failing to truly reflect the system's risk status. Therefore, this invention designs a differentiated normalization strategy for each indicator, mapping them uniformly to the [0,1] interval, providing comparable and stable input for subsequent dynamic risk scoring.

[0059] 1. Density of abnormal behavior

[0060] The density of behavioral anomalies is itself a proportional value with a clearly defined physical range of [0,1]. Its risk contribution is approximately linearly correlated with the proportion of abnormal sessions; for every percentage point increase in the anomaly proportion, the system risk should increase linearly accordingly. Therefore, linear proportional normalization is adopted.

[0061] in The maximum tolerance threshold preset for the system. In one specific embodiment, based on the statistical distribution of business scenarios, the threshold is set to... The value is set to 0.3, meaning that when the proportion of abnormal sessions exceeds 30%, the abnormal behavior density is considered to have reached its limit. The value is 1. This threshold can be dynamically adjusted based on the historical baseline of different business scenarios (such as government systems, social platforms, and financial transactions).

[0062] Linear normalization preserves the original proportional relationship of abnormal behavior density, enabling changes in early abnormal behavior to be reflected linearly and smoothly in the risk score, which is convenient for operation and maintenance personnel to understand and debug.

[0063] 2. Confidence drift

[0064] The confidence drift consists of two parts: mean deviation and standard deviation. Its physical range is relatively controllable, typically between 0 and 0.5. The change in confidence drift is approximately linearly related to the degree of model drift; the more severe the drift, the larger the confidence drift value, and the greater its contribution to the system's risk. Therefore, linear proportional normalization is also used.

[0065] in The preset threshold is determined based on historical statistics of the confidence distribution during the system's stable period. In one specific embodiment, confidence data is collected over 30 days of normal system operation to calculate... CID The 95th percentile as The value is approximately 0.4. When CID When this threshold is exceeded, the model is considered to have drifted significantly. Approaching 1.

[0066] Linear normalization guarantees CID The changes are synchronized with the degree of decline in model health, when CID When the threshold is exceeded, the subsequent incremental training mechanism of the model can be smoothly triggered, avoiding false triggering or missed triggering caused by nonlinear jumps.

[0067] 3. Variants against entropy

[0068] Variant entropy is an entropy value, theoretically ranging from [0, +∞). However, in practical applications, once the complexity of the variant's morphology reaches a certain level, further increases in entropy tend to saturate the guiding significance for defense strategies. Using linear normalization makes it difficult to determine the upper limit. VAEmax On the other hand, extremely large values ​​can dominate risk scores and distort the true level of risk.

[0069] Therefore, upper bound truncation normalization is adopted:

[0070] The truncation threshold of 3.0 was determined based on the following: through statistical analysis of historical variant attack samples, when... VAE When the value reaches 3.0, the edit distance distribution of variant words is highly dispersed, covering the vast majority of variant types; beyond this value, the gain of new variants on risk assessment is less than 5%, and the business significance tends to saturate. Therefore, 3.0 is set as the upper limit of risk complexity, and all values ​​exceeding this value are normalized to 1.

[0071] Truncated normalization avoids the excessive influence of extreme entropy values ​​on risk scores, while treating "high complexity" and "extremely high complexity" as the same risk level, which is highly consistent with the business defense strategy (unified model intervention when VAE is too high).

[0072] 4. Rule Failure Index (REI)

[0073] The range of REI is [0,1], but the sensitivity of business to REI is non-linear: when REI is low (e.g., less than 0.3), the impact of rule failure on the system is small, and the risk contribution should increase slowly; when REI approaches the trigger threshold (0.4) and continues to rise, it is necessary to quickly amplify its risk signal in order to push the system to enter the rule base update path in a timely manner.

[0074] Therefore, a natural exponential transformation is employed:

[0075] The properties of this transformation are as follows: when = At 0 o'clock, = ≈ 0.368, retaining the baseline weight to avoid the risk of rule failure being completely ignored; when When =0.3, = ≈ 0.496, growth is slow; when When =0.4, = ≈ 0.549, still in the low to medium range; when When =0.8, = ≈ 0.819, acceleration begins; when When =1.0, = = 1.0, reaching the upper limit.

[0076] The exponential transformation achieves a non-linear amplification effect of "smooth early stage and accelerated later stage", which precisely matches the business requirement of "high tolerance in the early stage of rule failure and rapid response after approaching the threshold", ensuring that risk scoring can drive the rule base update mechanism in a timely manner.

[0077] After the above differential normalization process, the four endogenous indicators are uniformly mapped to the [0,1] interval, eliminating the differences in dimensions and distributions, and can be directly input into the risk scoring formula for weighted fusion.

[0078] For example, the risk scoring formula is:

[0079] Based on the design principle of "prevention first, remediation second", behavioral anomalies (UAD) and model drift (CID) can capture risk signs from different dimensions in advance before security threats cause substantial harm. Therefore, they are defined as early risk signals and each accounts for 30% of the weight allocation. Variants complexity (VAE) and rule failure index (REI) mainly reflect the exposed defense gaps and are lagging assessment indicators. Therefore, each accounts for 20% of the weight allocation.

[0080] The weight coefficients (0.3, 0.3, 0.2, 0.2) in this embodiment are not preset initial values. Instead, they are the optimal weight combination obtained after parameter tuning and A / B comparison experiments, based on the AI ​​dialogue business scenario in the social domain. This was achieved by constructing 100,000 labeled samples, using a linear regression algorithm to establish a quantitative correlation between risk scores and real risk events. This weight reflects the statistical regularity that the contribution of early risk signals (UAD, CID) is higher than that of defense gap indicators (VAE, REI) in this dataset. After deployment, these weights can remain unchanged as a fixed configuration of the system. However, if switching to other business scenarios (such as finance or government affairs) or if the data distribution changes significantly, it is recommended to re-collect data and train new weights using the same method to ensure the accuracy of risk scoring. It should be noted that the above weight allocation is only an exemplary configuration in this embodiment. In actual applications, the weights can be adjusted according to the risk preferences of specific business scenarios (such as a greater emphasis on false negative rate control, false positive rate tolerance, or response timeliness). This invention is not limited to a fixed weight ratio.

[0081] S130. If the risk score is greater than the first preset threshold and the rule failure index is greater than the second preset threshold, then the rule base update process is initiated.

[0082] If the risk score is greater than the first preset threshold and the rule failure index is less than or equal to the second preset threshold, then the model training process is started.

[0083] Since endogenous indicators are calculated only within a randomly selected time period, risk scoring and subsequent rule base updates or model training decisions are only executed after the sampling period ends. During non-sampling periods, the system maintains its existing strategy.

[0084] The core of the decision-making process lies in automatically selecting one of two paths—short-term rapid defense or long-term semantic optimization—based on the dominant type of risk.

[0085] Based on the comparison results of the risk score and the first preset threshold, and the comparison results of the rule failure index and the second preset threshold, the system determines whether to perform rule base update and model training, or maintain regular monitoring.

[0086] In one case, if the risk score exceeds the first preset threshold and the rule failure index is greater than the second preset threshold, it indicates that the existing rules have failed on a large scale and cannot effectively cover the current risk. The system will initiate the rule library update process.

[0087] Specifically, when the following conditions are met, the system determines that the current risk mainly stems from insufficient rule coverage (such as explicit violation content like variant words and jargon), and thus initiates the rule library update process: > and

[0088] Where: is the first preset threshold (risk score threshold), which takes the value of 0.8 in a specific embodiment. This threshold is determined based on the ROC curve analysis of historical risk events. On the premise of ensuring that the false negative rate is lower than 5%, the system initiates defense upgrade when the risk reaches a medium level.

[0089] <0OO0310> is the second preset threshold (rule failure index threshold), which takes the value of 0.4 in a specific embodiment. This threshold is determined based on the misinterception and false negative statistics of the rule engine: when the REI exceeds 0.4, it indicates that the coverage ability of the rule library for the current input text has significantly declined, and more than 40% of the rules make incorrect decisions (misinterception or false negative), and an update is urgently needed.

[0090] When the risk score is greater than it indicates that the overall risk of the system is at a high level and active intervention is required; the rule failure index is greater than then it indicates that the main cause of the current risk is the lack of timeliness of the rule library, rather than the lack of semantic understanding ability of the model; When both are satisfied, it means that the problem lies in the rule layer, and rapid defense should be achieved by updating the rule library first, rather than initiating model incremental training with higher computational costs.

[0091] Further, the specific steps of the rule library update process: a1: Extract the false negative variants from the manual review logs or rule engine interception records, and screen out the variant words that are not covered by the rules but are determined to be violations.

[0092] The system automatically collects the decision records of the rule engine and the manual review results every day. For samples that are determined by the rule engine to be "released" but are later manually marked as "violation", the system extracts the keywords or phrases therein as false negative variants. These variants are usually deformations of sensitive words not included in the rule library. For example: Shape variants: "opening the melon" (replacing the benchmark word "killing"); Pinyin / pronunciation variant: "zhu n zhàng" (replacing the base word "transfer"); Mixed variant: Replace "gun" with an emoji made of a gun pattern.

[0093] The purpose of this step is to identify the coverage blind spots in the existing rule base, provide raw materials for generating new rules, and achieve targeted rule supplementation.

[0094] a2: Based on the grapheme similarity and / or phonetic similarity between the variant word and the base word, generate corresponding fuzzy matching rules as multi-hop matching rules.

[0095] Based on the extracted variant words and their corresponding base words (original sensitive words), use grapheme similarity (such as similar characters, radical replacement, structure decomposition) and phonetic similarity (such as homophones, near-homophones, fuzzy pinyin matching) to automatically generate a set of fuzzy matching rules, that is, multi-hop matching rules.

[0096] The system uses a predefined transformation rule library and an edit distance algorithm to generate regular expressions or wildcard patterns that can cover multiple variant forms for each missed report variant. The purpose is to generalize the matching rules for similar variants from a single missed report variant, improve the coverage and robustness of the rules, and avoid the inefficient way of manually writing rules for each variant.

[0097] Exemplarily, for the grapheme variant "开瓜": Base word: "杀人" Generated rule: 开[瓜爪呱] Can match: grapheme variants such as "开瓜", "开爪", "开呱", etc.

[0098] Exemplarily, for the phonetic variant "zhu n zhàng": Base word: "转账" Generated rule: (转|zhu?an)[账帐|zhang] Can match: pinyin input variants such as "转账", "转帐", "zhuanzhang", etc.

[0099] a3: Push the generated fuzzy matching rules to the rule engine to complete the rule deployment and update.

[0100] The rules engine supports a dynamic rule loading interface. After generating a new rule, the system sends it to the rules engine's management endpoint in JSON or DSL (Domain-Specific Language) format. Upon receiving the new rule, the rules engine adds it to the active rules table and completes compilation and activation within one minute. Throughout this process, the system continuously processes online requests without affecting the matching performance of existing rules. This step enables minute-level deployment of rule updates (compared to several hours of service restarts using traditional methods), ensuring rapid response to new variant attacks and meeting real-time defense requirements.

[0101] This decision path enables rapid short-term defense. Tests show that it takes only 5 to 10 minutes from the discovery of a missed detection to the defense taking effect. Since it only involves the generation and loading of string matching rules, no GPU resources are required, and the computational cost of rule updates is much lower than that of model training, thus controlling system overhead while ensuring real-time performance.

[0102] In another scenario, if the risk score is greater than a first preset threshold and the rule failure index is less than or equal to a second preset threshold, then the model training process is initiated.

[0103] Based on the comparison results of the risk score and the first preset threshold, and the comparison results of the rule failure index and the second preset threshold, the system determines whether to perform rule base update and model training, or maintain regular monitoring (if the risk score does not exceed the first preset threshold).

[0104] If the risk score exceeds the first preset threshold, but the rule failure index does not exceed the second preset threshold, it indicates that the existing rules are still effective overall, but the risk score is abnormally high, which may indicate the presence of new or variant attacks. In this case, the system will trigger model training to improve the model's ability to identify unknown patterns. At this point, the system will start the model training process.

[0105] Furthermore, the model training process includes: b1: Extract text segments from the text to be optimized whose confidence level is within a preset fuzzy range.

[0106] The system first receives the text after it has been processed by the rule engine and the semantic awareness layer. The semantic awareness layer outputs a risk confidence score for each text based on a deep learning model (e.g., mapping the model output to a probability value in the range [0,1] using the Softmax function, expressed as a percentage, with a value ranging from 0% to 100%): Confidence level ≥ 90%: Directly block; Confidence level ≤ 70%: Release directly; Confidence ∈ (70%, 90%): Considered as “fuzzy fragments”, transferred to the full model decision layer or used for incremental training.

[0107] This step involves extracting the text content requiring optimization from these blurred fragments, which then serves as input for subsequent adversarial example generation and model training. This avoids the waste of resources from full text scanning, focusing instead on areas where the model's current recognition capabilities are insufficient. By accurately locating the model's knowledge blind spots, high-quality, targeted training samples are provided for model training.

[0108] b2: Based on the quantification result of the variant adversarial entropy, generate semantically preserved adversarial samples for the high-risk segments.

[0109] Using the variant adversarial entropy (VAE) value calculated in step S110, the complexity and diversity of the current variant attack are evaluated, and then adversarial samples that retain the original semantics but change the expression form are automatically generated for high-risk segments.

[0110] Variant adversarial entropy (VAE) reflects the diversity and randomness of variant words used by attackers; the higher the entropy value, the more dispersed the variants. Based on the quantification results of VAE (e.g., VAE > 2.0 indicates a high-complexity variant attack), the system generates adversarial examples using preset transformation strategies such as homophonic substitution, glyph substitution, insertion of interfering characters, and pinyin transliteration.

[0111] For example, adversarial examples can be generated for the high-risk segment "open melon": "open claw", "open croak", "open melon" etc.

[0112] For the medical / drug ambiguous term "alprazolam", generate adversarial examples such as contextual variants like "alprazolam tablets" and "alprazolam capsules".

[0113] These adversarial examples are semantically consistent with the original violation content, but differ in their surface form. They are used to train the model to learn the essential features of the variants rather than mechanically matching them. This is to augment the training data, enhance the model's robustness to attacks with unknown variants, and prevent overfitting to fixed representations.

[0114] b3: Use the adversarial example to update some parameters of the semantic model.

[0115] Use lightweight semantic models (such as LSTM after BERT distillation or small Transformers) to update parameters only on the last fully connected layer (or the last few layers) of the model (incremental training) without retraining the entire model.

[0116] Traditional full-scale model training requires significant computational resources and time (several hours on GPUs). This step employs a partial-layer freeze incremental training strategy: keeping the underlying model parameters unchanged, as these layers have already learned general semantic knowledge; only the top layers (usually the last two fully connected layers) are updated to adapt to newly generated adversarial examples, improving the ability to discriminate variant words and ambiguous contexts. The training process uses mini-batch adversarial examples (e.g., a few hundred) and a small number of original samples, converging after several iterations. The computational cost of this method is only 1 / 10 of full-scale training, with GPU time less than 10 minutes. It achieves the goal of rapidly adapting the model to new attack patterns with extremely low resource cost, while avoiding catastrophic forgetting (preserving existing knowledge).

[0117] b4: Deploy the updated model to the inference service.

[0118] The model parameters obtained from incremental training are pushed to the online inference service, and the model replacement is completed within 5 minutes through a hot-swap mechanism, without affecting the business traffic being processed. The system adopts a blue-green deployment or version hot-loading approach: the model inference service maintains the current active version and the newly loaded version. After the parameters are updated, the new model is uploaded to the server and its integrity is verified. Then, traffic is gradually or all at once switched to the new model through the configuration center or service registry. At the same time, the old model is retained as a rollback option. If the new model has an anomaly, it can be rolled back in seconds. This process does not require restarting the service or interrupting request processing. The typical effective time is within 5 minutes, thus realizing a minute-level closed loop from model training to deployment. This solves the problem of "rigid knowledge update" that traditional model updates require hours or even days, ensuring the real-time performance of the defense system.

[0119] Example 2

[0120] Figure 2 This is a schematic diagram of the architecture of a content security detection system provided according to an embodiment of the present invention. The content security detection system adopts a layered processing architecture, including: Rule engine layer 210 is used to quickly screen input content and block explicit violations; Semantic Awareness 220 is used to perform deep semantic analysis on suspected fragments output by the rule engine layer and to perform traffic splitting based on risk confidence. The full-model decision layer 230 is used to make multi-model fusion decisions on the fuzzy segments output by the semantic perception layer.

[0121] Specifically, the rule engine layer 210 uses a preset sensitive word library and simple matching rules to quickly match keywords in the input text with extremely low computational overhead, directly blocking explicit violations containing clear sensitive words, thereby filtering out most simple threats at an early stage and reducing the load on subsequent semantic analysis.

[0122] The semantic awareness layer 220 uses a lightweight deep learning model to perform contextual semantic understanding on the suspected fragments returned by the rule engine layer, outputs risk confidence, and implements three-level diversion based on preset thresholds to ensure accurate interception of high-risk content, rapid release of low-risk content, and in-depth analysis of ambiguous fragments.

[0123] The full-model decision layer 230 calls multiple models of different dimensions, such as semantic understanding models, knowledge graph models, and behavior analysis models, to perform comprehensive reasoning for ambiguous segments that the semantic perception layer cannot clearly determine. It also integrates the output results of each model through dynamic weights to make the final decision on interception or release, ensuring the accuracy of the judgment of high-difficulty samples.

[0124] Figure 3 This is a flowchart of a content security detection method according to an embodiment of the present invention. This method is applicable to social media platforms and government systems. The method can be executed by a content security detection device, which can be implemented in hardware and / or software. Figure 3 As shown, the method includes: S310 performs a rapid screening of the text input rule engine layer to be detected, and obtains a first processing result, which includes allowing, blocking or suspected fragments.

[0125] When performing content security detection using the content security detection method described in this embodiment, the text to be detected is first input into the rule engine layer. This layer has a built-in basic sensitive word library and simple string matching rules, such as keyword exact matching and regular expression matching, which can quickly screen the input text with extremely low computational overhead.

[0126] The rule engine layer compares the text content line by line: If the text contains sensitive words explicitly listed in the dictionary or matches a preset violation rule pattern, it is judged as explicit violation content, and "blocked" is directly output as the first processing result; If the text does not contain any sensitive words and does not match any rules, it is judged as normal content and "allow" is output directly as the first processing result; If a text does not match a specific interception rule, but cannot be definitively determined by the rule engine layer due to the presence of suspected variants (such as pinyin substitution, character shape deformation, or insertion of special symbols), the text is marked as a "suspected fragment" and output to the subsequent semantic perception layer for further in-depth analysis.

[0127] Through this rapid screening mechanism, the rule engine layer can block approximately 90% of explicit violations, thereby significantly reducing the computational load on subsequent processing layers.

[0128] S320 inputs the suspected fragment into the semantic perception layer for semantic analysis, calculates the risk confidence level, and obtains a second processing result based on the risk confidence level. The second processing result includes releasing, intercepting, or pending fragments.

[0129] The suspected segments output by the rule engine layer are input into the semantic perception layer. This layer uses a lightweight deep learning model (such as a distilled LSTM or a small BERT) to perform contextual semantic understanding and risk assessment on each suspected segment, and outputs a risk confidence level between 0% and 100%.

[0130] After completing the confidence calculation, the system performs three-level traffic splitting based on preset thresholds: If the confidence level is ≥90%, the segment is determined to be high-risk content, and "Blocked" is directly output as the second processing result; If the confidence level is ≤70%, it is judged as low-risk content, and "release" is directly output as the second processing result; If the confidence level is in the fuzzy range between 70% and 90%, the semantics of the segment are not clear enough and the semantic perception layer cannot make a final judgment independently. At this time, the segment is marked as a "pending segment" and output to the full model decision layer for further comprehensive judgment.

[0131] Through the above mechanism, the semantic perception layer can pass only about 5% to 10% of the most ambiguous samples to the subsequent full model decision layer while ensuring a high recognition accuracy, thereby achieving an effective balance between accuracy and computational cost.

[0132] S330 inputs the pending fragment into the full model decision layer for comprehensive analysis and outputs the final decision on interception or release.

[0133] The undetermined segments (i.e. fuzzy samples with confidence levels between 70% and 90%) output by the semantic perception layer are input into the full model decision layer, which uses a multi-model fusion strategy for comprehensive evaluation.

[0134] Specifically, the system simultaneously invokes multiple deep learning models with different dimensions, including: The models include a Transformer-based semantic understanding model (used to capture deep meanings in context), a knowledge graph-based associative reasoning model (used to identify ambiguities in domain-specific terms, such as the different meanings of "alprazolam" in medical and drug use scenarios), and a behavior sequence-based anomaly detection model (used to associate user historical behavior patterns).

[0135] After each model independently outputs a violation probability or risk score for the segment, the overall model decision layer finally outputs a definite decision to block or allow the passage. Since only about 5% to 10% of the most ambiguous samples enter this layer for processing, the overall model decision layer can achieve accurate judgment on highly difficult samples with acceptable computational overhead, thereby effectively reducing the false positive rate and false negative rate while ensuring the overall system throughput.

[0136] The content security detection system and method provided in this embodiment form a closed-loop collaborative relationship with the content security detection system self-optimization method provided in Embodiment 1: The rule engine interception records, semantic perception layer confidence output, full model decision layer adjudication results, and manual review logs generated during the operation of the content security detection system provide calculation data for endogenous indicators such as rule failure index, confidence drift, variant adversarial entropy, and behavioral anomaly density for the self-optimization method. The self-optimization method calculates risk scores based on these indicators and automatically selects two paths—rule base update or incremental model training—based on a comparison of the score results with the rule failure index. The updated fuzzy matching rules are deployed to the rule engine layer in real time, or the incrementally trained lightweight model is hot-deployed to the semantic perception layer and the full model decision layer. Thus, the content security detection system provides the optimization method with a data source and execution platform, while the optimization method endows the content security detection system with dynamic evolution capabilities, jointly achieving adaptive security protection through real-time perception, hierarchical decision-making, and on-demand optimization.

[0137] Example 3

[0138] Figure 4 This is a self-optimizing device for a risk-aware content security detection system provided according to an embodiment of the present invention.

[0139] Structural diagram. This embodiment is applicable to social media platforms and government systems. The device can be implemented in hardware and / or software, such as... Figure 4 As shown, the device includes: Endogenous index calculation unit 410 is used to calculate the endogenous index generated during the operation of the system per unit time, and to normalize the endogenous index 420. The endogenous index includes at least rule failure index and variant adversarial entropy. The risk scoring unit is used to calculate the risk score based on the normalized endogenous indicators and preset weight parameters. The decision-making and update unit 430 is used to initiate a rule base update process when the risk score is greater than a first preset threshold and the rule failure index is greater than a second preset threshold; and to initiate a model training process when the risk score is greater than the first preset threshold and the rule failure index is less than or equal to the second preset threshold.

[0140] The self-optimization device for a content security detection system based on risk perception provided in this embodiment of the invention can execute the self-optimization method for a content security detection system based on risk perception provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0141] Those skilled in the art should understand that the technical features of the above embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as the combination of these technical features does not contradict each other, it should be considered within the scope of this specification. The above embodiments only illustrate several implementations of the present invention, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make several modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of the present invention should be determined by the appended claims. It should be understood that various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of the present invention can be achieved, and this is not limited herein.

Claims

1. A self-optimization method for a content security detection system based on risk perception, characterized in that, Includes the following steps: Calculate the endogenous indicators generated during the operation of the system per unit time, and normalize the endogenous indicators. The endogenous indicators include at least the rule failure index and the variant adversarial entropy. The risk score is calculated based on the normalized endogenous indicators and preset weight parameters. If the risk score is greater than the first preset threshold and the rule failure index is greater than the second preset threshold, then the rule base update process is initiated. If the risk score is greater than the first preset threshold and the rule failure index is less than or equal to the second preset threshold, then the model training process is started.

2. The method according to claim 1, characterized in that, The rule failure index is calculated based on the rule engine's interception records and manual review logs. The rule failure index is the ratio of erroneous decisions to the total processing volume. Erroneous decisions include: erroneous interception and missed release.

3. The method according to claim 1, characterized in that, The variant adversarial entropy is determined in the following way: Obtain the variant word set and the base word corresponding to each variant word, calculate the edit distance from each variant word to the corresponding base word, determine the frequency of each edit distance in the variant word set, and determine the variant anti-entropy based on the frequency of occurrence according to the entropy calculation formula.

4. The method according to claim 1, characterized in that, The endogenous indicators also include: Behavioral anomaly density is determined based on the ratio of the number of abnormal user sessions detected per unit time to the total number of sessions. It is used to characterize the abnormal access pressure faced by the system. Abnormal sessions meet any of the following conditions: high-frequency submission or access during abnormal time periods. Confidence drift, determined based on the degree of deviation between the distribution of the model's output confidence on a given day and the baseline during the stable period, is used to characterize the change in the model's ability to understand the current input.

5. The method according to claim 1, characterized in that, The rule base update process includes: Extract missed variants from manual review logs or rule engine interception records, and filter out variant words that are not covered by the rules but are judged to be in violation; Based on the similarity in shape and / or sound between the variant word and the base word, a corresponding fuzzy matching rule is generated as a multi-hop matching rule; The generated fuzzy matching rules are pushed to the rule engine to complete the rule deployment and update.

6. The method according to claim 1, characterized in that, The model training process includes: Extract text segments whose confidence levels fall within a preset fuzzy range from the text to be optimized; Based on the quantification results of the variant adversarial entropy, semantically preserved adversarial samples are generated for the high-risk segments; The adversarial examples are used to update some parameters of the semantic model; Deploy the updated model to the inference service.

7. The method according to claim 1, characterized in that, The content security detection system adopts a layered processing architecture, including: The rules engine layer is used to quickly screen input content and block explicit violations; The semantic awareness layer is used to perform deep semantic analysis on suspected fragments output by the rule engine layer and to triage them according to the risk confidence level. The full-model decision layer is used to make decisions on the fuzzy fragments output by the semantic perception layer through multi-model fusion.

8. The method according to claim 7, characterized in that, The method by which the content security detection system performs security detection on content includes: The text to be detected is input into the rule engine layer for rapid screening to obtain the first processing result, which includes allowing, blocking or suspected fragments. The suspected fragment is input into the semantic perception layer for semantic analysis, the risk confidence is calculated, and a second processing result is obtained based on the risk confidence. The second processing result includes releasing, intercepting, or pending fragments. The undetermined segment is input into the full model decision layer for comprehensive analysis, and the final decision on whether to intercept or allow passage is output.

9. A self-optimizing device for a content security detection system based on risk perception, characterized in that, include: An endogenous index calculation unit is used to calculate the endogenous indexes generated during the operation of the system per unit time, and to normalize the endogenous indexes. The endogenous indexes include at least the rule failure index and the variant adversarial entropy. The risk scoring unit is used to calculate the risk score based on the normalized endogenous indicators and preset weight parameters. The decision-making and updating unit is used to initiate a rule base update process when the risk score is greater than a first preset threshold and the rule failure index is greater than a second preset threshold. When the risk score is greater than the first preset threshold and the rule failure index is less than or equal to the second preset threshold, the model training process is started.

Images