A content dynamic review method, system, device and medium
By constructing a dynamic threshold mapping model of severity index and dual-channel intelligent review, the problem that existing content review systems cannot respond to changes in review requirements in real time has been solved, achieving a balance between system adaptability and user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- 湖南工商大学
- Filing Date
- 2026-06-23
- Publication Date
- 2026-07-21
AI Technical Summary
Existing content moderation systems are unable to adapt to changes in moderation requirements in real time, resulting in a high false positive rate and low efficiency, and failing to achieve precise quantitative alignment between moderation standards and policy requirements.
By constructing a dynamic threshold mapping model based on severity index, combined with keyword density statistics and high-dimensional semantic vector similarity calculation, a quantitative perception of review requirements is achieved. Furthermore, through a dual-channel intelligent review and negative feedback correction mechanism, the review strategy is adaptively adjusted.
The content moderation system has achieved real-time adaptive response to moderation requirements, reducing the false positive rate and ensuring a dynamic balance between pursuing high compliance and maintaining user experience.
Smart Images
Figure CN122432324A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of natural language processing technology, and in particular to a method, system, device and medium for dynamic content review. Background Technology
[0002] With the explosive growth of UGC (user-generated content) platforms on the internet, the governance of online content compliance faces unprecedented challenges. This necessitates content moderation systems to possess extremely high sensitivity and adaptability, capable of rapidly responding to changes in moderation requirements. However, faced with massive and real-time influxes of data, purely manual moderation can no longer meet the demands, making automated and intelligent content moderation technologies a necessity for the industry. The development of content moderation technology is essentially a process of evolution from feature engineering to deep semantic understanding, and also a transition from static rules to dynamic adaptation.
[0003] The first stage (keyword matching technology): Early content moderation mainly relied on a blacklist for mechanical matching, blocking texts containing sensitive words. This method was simple and efficient, but it was easily bypassed by variations (such as pinyin, homophones, and word splitting), had a very high false positive rate, and could not understand the context.
[0004] The second stage (traditional machine learning): This was followed by classification methods based on traditional machine learning models such as SVM and Bayesian methods. Researchers trained classifiers by extracting features from text, such as TF-IDF and N-gram. Although this improved generalization ability, feature engineering still relied on manual design and struggled to capture deep semantics.
[0005] The third stage (deep learning and pre-trained models): With the emergence of pre-trained language models such as BERT and RoBERTa, the mainstream approach shifted to sentiment analysis or text classification based on deep neural networks. This method utilizes massive amounts of labeled data to train deep models, outputting a probability score for content violations, and making judgments based on a preset fixed threshold. For example, using bidirectional LSTM combined with an attention mechanism to capture subtle offensive semantics in text significantly improves recognition accuracy.
[0006] While existing deep learning-based auditing systems have made significant progress in recognition accuracy, they still exhibit considerable lag and rigidity in engineering applications. The vast majority of existing systems still employ an "offline training + static threshold" model, meaning that once the model is deployed, its judgment criteria (thresholds) remain fixed. When auditing requirements change abruptly, these systems cannot automatically detect and adjust their scale, often requiring manual intervention from operations personnel to manually modify configurations or retrain the model with new data. This manual operation and maintenance approach is not only inefficient but also fails to achieve precise quantitative alignment between auditing standards and policy requirements, consistently failing to resolve the disconnect between "machine algorithms" and "dynamic regulations."
[0007] Therefore, it is necessary to provide a content moderation method with the ability to provide real-time feedback and adaptive correction. Summary of the Invention
[0008] To address the aforementioned technical problems, this invention proposes a method, system, device, and medium for dynamic content review.
[0009] In a first aspect, the present invention provides a method for dynamic content review, comprising: S100, Perception of Audit Requirements: Quantify the perception of audit requirements to obtain a comprehensive severity index; S200, Threshold Dynamic Mapping: Converts the comprehensive severity index into content review interception thresholds; S300, Dual-channel Intelligent Review: The content to be reviewed is initially screened and divided into high-confidence certainty content and semantically ambiguous content. High-confidence certainty content is reviewed and blocked by the content review and blocking threshold. Semantically ambiguous content is adaptively reviewed and blocked based on a large language model with an injected comprehensive severity index, and the final review conclusion is output.
[0010] Optionally, in S100, the process of obtaining the comprehensive severity index includes: S101. Data Preprocessing: Clean the data according to the audit requirements, removing irrelevant noise. The expression is as follows: ; in, This indicates the text of the review requirements after cleaning. This represents the original review request text. This indicates the removal of HTML tags. This indicates the removal of special characters and formatting noise; S102. Explicit risk calculation based on keyword density: Pre-set a high-risk sensitive word library. ,statistics The weighted frequency of occurrence of medium- and high-risk sensitive words is used to calculate the explicit risk score. : ; in, For indicator functions, Keywords Preset weights, This represents the total length of the text. The function is used to normalize numerical values; S103. Implicit semantic computation based on vector similarity: Utilizing a pre-trained language model as an encoder, Encode as a high-dimensional dense vector Collect a sample set of historical review requirements, calculate their semantic vectors, and obtain a strict control semantic center vector by performing cluster analysis or weighted averaging on these vectors. ;calculate and The cosine of the angle between them is expressed as follows: ; The calculated cosine of the included angle is used as the implicit semantic score. ; S104. Calculate the overall severity index: Linearly weight the explicit keyword scores and implicit semantic scores to obtain the final overall severity index. The expression is as follows: ; in, For adjustment coefficients, .
[0011] Optionally, in S200, the process of obtaining the content moderation interception threshold includes: Construct a linear inverse mapping function, with the following expression: ; in, This indicates the content moderation and blocking threshold. Indicates the baseline threshold. Indicates the sensitivity coefficient; set up The legal physical range is If the calculated If the value exceeds this range, a truncation process will be performed, meaning the threshold value of 0 or 1 will be used as the cutoff value. This ensures the feasibility of the project under any audit requirements for dynamic thresholds.
[0012] Optionally, in S300, the initial screening process for content to be reviewed includes: Input the content to be reviewed into a lightweight classification model, and output the probability score of violation for the content to be reviewed. ,in, Real-time retrieval of content review and blocking thresholds Introduce a preset fuzzy bias factor ,if If so, the content to be reviewed is determined to be semantically ambiguous. If so, the content to be reviewed is determined to be of high confidence certainty.
[0013] Optionally, in S300, the process of reviewing and blocking high-confidence-certainty content based on content moderation blocking thresholds includes: For content with high confidence and certainty, if If so, the interception action will be executed directly and the violation label will be recorded; if If so, then they will be allowed to pass.
[0014] Optionally, in S300, the process of adaptively reviewing and blocking semantically ambiguous content based on the large language model with an injection content review and blocking threshold includes: By dynamically injecting the comprehensive severity index into the prompt word template of the large language model, the large language model has the ability to adaptively adjust the judgment scale. The large language model performs content review on semantically ambiguous content and outputs a judgment conclusion.
[0015] Optionally, after outputting the final review conclusion, a negative feedback correction process is also included, the process of which includes: Real-time statistics of user appeal approval rate within the most recent time window ; like Exceeding the preset safety warning line This indicates the current situation. The overly strict approach has resulted in numerous misjudgments. Therefore, a rollback correction is necessary, expressed as follows: ; in, To adjust the step size; Corrected threshold Will replace This achieves a dynamic balance between review requirements and user experience.
[0016] In a second aspect, the present invention also provides a dynamic content review system for implementing the dynamic content review method described above, the system comprising: Audit Requirement Perception Module: Used to quantitatively perceive audit requirements and obtain a comprehensive severity index; Threshold dynamic mapping module: used to convert the comprehensive severity index into content moderation interception thresholds; Dual-channel intelligent review module: used to sort and filter the content to be reviewed, and output the final review conclusion; Negative Feedback Correction Module: Used to correct the content moderation interception threshold using negative feedback.
[0017] In a third aspect, the present invention also provides a computer device, including a memory and a processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the dynamic content review method described above.
[0018] In a fourth aspect, the present invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the content dynamic review method described above.
[0019] The application of the technical solution of the present invention has at least the following beneficial effects: This invention provides a dynamic content review method that transforms unstructured review requirements into a structured comprehensive severity index, thus addressing the deficiency of existing content review systems in their inability to quantify changes in review requirements. Furthermore, this invention combines keyword density statistics with high-dimensional semantic vector similarity calculation to accurately quantify the comprehensive severity index reflecting changes in review requirements, overcoming the shortcomings of traditional methods that rely on subjective human experience to adjust thresholds and lack scientific quantitative basis.
[0020] This invention constructs a dynamic mapping model for review thresholds based on a severity index driven by a reverse approach. Unlike existing technologies that use fixed static thresholds, this invention establishes a linear inverse mapping function that changes with review requirements. This mechanism can automatically generate and issue new content review interception thresholds, enabling the review system to adapt to changes in review requirements in real time and significantly shortening the compliance risk window.
[0021] This invention employs a robust execution architecture that combines dual-channel intelligent review with negative feedback correction. At the execution level, the invention dynamically injects a quantified comprehensive severity index into the prompt word template of the Large Language Model (LLM), enabling the LLM to perform adaptive secondary judgment on ambiguous content. Simultaneously, the negative feedback correction mechanism introduces automatic rollback based on user complaint rates to prevent false negatives caused by overly aggressive algorithm adjustments, ensuring a dynamic balance between high compliance and user experience. Attached Figure Description
[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.
[0023] Figure 1 This is a flowchart illustrating the steps of the content dynamic review method in a preferred embodiment of the present invention. Detailed Implementation
[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.
[0025] Example 1: like Figure 1 As shown, this embodiment provides a method for dynamic content review, including (steps S100 to S400): S100, Perception of Audit Requirements: Quantify the perception of audit requirements to obtain a comprehensive severity index.
[0026] The process of obtaining the overall severity index includes: S101. Data Preprocessing: Clean the data according to the audit requirements, removing irrelevant noise. The expression is as follows: ; in, This indicates the text of the review requirements after cleaning. This represents the original review request text. This indicates the removal of HTML tags. This indicates the removal of special characters and formatting noise.
[0027] It should be noted that the review requirements text after cleaning... In this embodiment, word segmentation is also required using a Chinese word segmentation tool (such as Jieba).
[0028] S102. Explicit risk calculation based on keyword density: Pre-set a high-risk sensitive word library. (High-risk sensitive words are strong-sounding words in the review requirements, such as "severely punish," "resist," and "prohibit," which can be set according to the actual review requirements.) Statistics The weighted frequency of occurrence of medium- and high-risk sensitive words is used to calculate the explicit risk score. : ; in, For indicator functions, Keywords Preset weights, This represents the total length of the text. The function is used to normalize numerical values.
[0029] S103, Implicit Semantic Computation Based on Vector Similarity: Utilizing a pre-trained language model as an encoder, Encode as a high-dimensional dense vector Collect a sample set of historical review requirements, calculate their semantic vectors, and obtain a strict control semantic center vector by performing cluster analysis or weighted averaging on these vectors. ;calculate and The cosine of the angle between them is expressed as follows: ; The calculated cosine of the included angle is used as the implicit semantic score. .
[0030] It should be noted that the implicit semantic score in this embodiment... This reflects how closely the current review requirements are semantically similar to the strict control standards at a deeper level. The higher the score, the more stringent the current review requirements are.
[0031] S104. Calculate the overall severity index: Linearly weight the explicit keyword scores and implicit semantic scores to obtain the final overall severity index. The expression is as follows: ; in, This is the adjustment coefficient (0.6 in this embodiment). .
[0032] It should be noted that the overall severity index The higher the value, the lower the tolerance for the content to be reviewed, meaning the review is more stringent.
[0033] S200, Threshold Dynamic Mapping: Converts the comprehensive severity index into a content moderation interception threshold.
[0034] In this embodiment, the process of obtaining the content moderation interception threshold includes: Construct a linear inverse mapping function, with the following expression: ; in, This indicates the content moderation and blocking threshold. Indicates the baseline threshold. Indicates the sensitivity coefficient; set up The legal physical range is If the calculated If the value exceeds this range, a truncation process will be performed, meaning the threshold value of 0 or 1 will be used as the cutoff value. This ensures the feasibility of the project under any audit requirements for dynamic thresholds.
[0035] It should be noted that the baseline threshold represents the default lenient review requirements, and in this embodiment, it is set to 0.85.
[0036] S300, Dual-channel Intelligent Review: The content to be reviewed is initially screened and divided into high-confidence certainty content and semantically ambiguous content. High-confidence certainty content is reviewed and blocked by the content review and blocking threshold. Semantically ambiguous content is adaptively reviewed and blocked based on a large language model with an injected comprehensive severity index, and the final review conclusion is output.
[0037] In this embodiment, the process of initial screening of the content to be reviewed includes: The content to be reviewed is input into a lightweight classification model (TextCNN or DistilBERT in this example), which outputs a violation probability score for the content to be reviewed. ,in, Real-time retrieval of content review and blocking thresholds Introduce a preset fuzzy bias factor ,if If so, the content to be reviewed is determined to be semantically ambiguous. If so, the content to be reviewed is determined to be of high confidence certainty.
[0038] Furthermore, the process of reviewing and blocking content with high confidence and certainty based on content review and blocking thresholds includes: For content with high confidence and certainty, if If so, the interception action will be executed directly and the violation label will be recorded; if If so, then they will be allowed to pass.
[0039] Furthermore, the process of adaptive content moderation and interception for semantically ambiguous content based on the large language model with an injection content moderation and interception threshold includes: By dynamically injecting the comprehensive severity index into the prompt word template of the large language model, the large language model has the ability to adaptively adjust the judgment scale. The large language model performs content review on semantically ambiguous content and outputs a judgment conclusion.
[0040] Specifically, the design scheme of the prompt word template in this embodiment is as follows: {"role": "system", "instruction": "You are a senior content security reviewer. We are currently in a special control period, and the policy severity index is < " (Maximum score 1.0) "constraints": [ "1. If the overall severity index is greater than 0.7, please 'block' any content containing metaphors, satire, or borderline content." 2. If the overall severity index is less than 0.3, please only 'block' content containing clearly prohibited words. 3. Please output the final judgment result (intercept / allow) and a brief reason. "input_content": "<User content pending review>"}.
[0041] Optionally, after outputting the final review conclusion, a negative feedback correction process is also included, the process of which includes: Real-time statistics of user appeal approval rate within the most recent time window ; like Exceeding the preset safety warning line This indicates the current situation. The overly strict approach has resulted in numerous misjudgments. Therefore, a rollback correction is necessary, expressed as follows: ; in, To adjust the step size; Corrected threshold Will replace This achieves a dynamic balance between review requirements and user experience.
[0042] In addition, this embodiment also provides a dynamic content review system for implementing the dynamic content review method described above. The system includes: Audit Requirement Perception Module: Used to quantitatively perceive audit requirements and obtain a comprehensive severity index; Threshold dynamic mapping module: used to convert the comprehensive severity index into content moderation interception thresholds; Dual-channel intelligent review module: used to sort and filter the content to be reviewed, and output the final review conclusion; Negative Feedback Correction Module: Used to correct the content moderation interception threshold using negative feedback.
[0043] To ensure the feasibility and optimal balance of the technical solution in actual engineering, this invention sets a clear applicable range for the core parameters of system operation, and the selection logic and boundary effects are shown in Table 1 below.
[0044] Table 1 Parameter Selection Logic and Boundary Effects
[0045] The beneficial effects of the method in this embodiment are verified through simulation experiments below: We selected 100,000 pieces of UGC (user-generated content) text from a certain content social platform as the test set, with the following preset system parameters: The simulation triggered a change in review requirements (a sudden surge in the severity of review requirements), and the processing performance of the method in this embodiment was compared with that of existing technologies, as shown in Table 2.
[0046] Table 2 Simulation Experiment Results
[0047] In addition, this embodiment also provides a computer device, including a memory and a processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the dynamic content review method described above.
[0048] It should be noted that computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof. These programming languages include object-oriented programming languages—such as Java, Smalltalk, and C++—as well as conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0049] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0050] In addition, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the content dynamic review method described above.
[0051] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., computer programs) for executing the above-described dynamic content review method. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the dynamic content review method provided in the above embodiments, and will not be repeated here.
[0052] The above description is only a preferred embodiment of the present invention and does not limit the scope of the present invention. All equivalent structural transformations made under the inventive concept of the present invention using the contents of the present invention specification and drawings, or direct / indirect applications in other related technical fields, are included within the protection scope of the present invention.
Claims
1. A method for dynamic content review, characterized in that, include: S100, Perception of Audit Requirements: Quantify the perception of audit requirements to obtain a comprehensive severity index; S200, Threshold Dynamic Mapping: Converts the comprehensive severity index into content review interception thresholds; S300, Dual-channel Intelligent Review: The content to be reviewed is initially screened and divided into high-confidence certainty content and semantically ambiguous content. High-confidence certainty content is reviewed and blocked by the content review and blocking threshold. Semantically ambiguous content is adaptively reviewed and blocked based on a large language model with an injected comprehensive severity index, and the final review conclusion is output.
2. The content dynamic review method according to claim 1, characterized in that, In S100, the process of obtaining the comprehensive severity index includes: S101. Data Preprocessing: Clean the data according to the audit requirements, removing irrelevant noise. The expression is as follows: ； in, This indicates the text of the review requirements after cleaning. This represents the original review request text. This indicates the removal of HTML tags. This indicates the removal of special characters and formatting noise; S102. Explicit risk calculation based on keyword density: Pre-set a high-risk sensitive word library. ,statistics The weighted frequency of occurrence of medium- and high-risk sensitive words is used to calculate the explicit risk score. : ； in, For indicator functions, Keywords Preset weights, This represents the total length of the text. The function is used to normalize numerical values; S103, Implicit Semantic Computation Based on Vector Similarity: Utilizing a pre-trained language model as an encoder, Encode as a high-dimensional dense vector Collect a sample set of historical review requirements, calculate their semantic vectors, and obtain a strict control semantic center vector by performing cluster analysis or weighted averaging on these vectors. ;calculate and The cosine of the angle between them is expressed as follows: ； The calculated cosine of the included angle is used as the implicit semantic score. ; S104. Calculate the overall severity index: Linearly weight the explicit keyword scores and implicit semantic scores to obtain the final overall severity index. The expression is as follows: ； in, For adjustment coefficients, .
3. The content dynamic review method according to claim 2, characterized in that, In S200, the process of obtaining the content moderation interception threshold includes: Construct a linear inverse mapping function, with the following expression: ； in, This indicates the content moderation and blocking threshold. Indicates the baseline threshold. Indicates the sensitivity coefficient; set up The legal physical range is If the calculated If the value exceeds this range, a truncation process will be performed, meaning the threshold value of 0 or 1 will be used as the cutoff value. This ensures the feasibility of the project under any audit requirements for dynamic thresholds.
4. The content dynamic review method according to claim 3, characterized in that, In S300, the initial screening process for content to be reviewed includes: Input the content to be reviewed into a lightweight classification model, and output the probability score of violation for the content to be reviewed. ,in, Real-time retrieval of content review and blocking thresholds Introduce a preset fuzzy bias factor ,if If so, the content to be reviewed is determined to be semantically ambiguous. If so, the content to be reviewed is determined to be of high confidence certainty.
5. The content dynamic review method according to claim 4, characterized in that, In S300, the process of reviewing and blocking high-confidence-certainty content based on content moderation blocking thresholds includes: For content with high confidence and certainty, if If so, the interception action will be executed directly and the violation label will be recorded; if If so, then they will be allowed to pass.
6. The content dynamic review method according to claim 5, characterized in that, In S300, the process of adaptive content moderation and interception for semantically ambiguous content based on the large language model with an injection content moderation and interception threshold includes: By dynamically injecting the comprehensive severity index into the prompt word template of the large language model, the large language model has the ability to adaptively adjust the judgment scale. The large language model performs content review on semantically ambiguous content and outputs a judgment conclusion.
7. The content dynamic review method according to claim 6, characterized in that, After outputting the final review conclusion, the process also includes negative feedback correction, which includes: Real-time statistics of user appeal approval rate within the most recent time window ; like Exceeding the preset safety warning line This indicates the current situation. The overly strict approach has resulted in numerous misjudgments. Therefore, a rollback correction is necessary, expressed as follows: ； in, To adjust the step size; Corrected threshold Will replace This achieves a dynamic balance between review requirements and user experience.
8. A dynamic content review system, characterized in that, The system for implementing the content dynamic review method as described in any one of claims 1 to 7, the system comprising: Audit Requirement Perception Module: Used to quantitatively perceive audit requirements and obtain a comprehensive severity index; Threshold dynamic mapping module: used to convert the comprehensive severity index into content moderation interception thresholds; Dual-channel intelligent review module: used to sort and filter the content to be reviewed, and output the final review conclusion; Negative Feedback Correction Module: Used to correct the content moderation interception threshold using negative feedback.
9. A computer device, characterized in that, Including memory and processor; The memory is used to store computer programs that can run on the processor; When the processor executes the computer program, it implements the steps of the content dynamic review method as described in any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the content dynamic review method as described in any one of claims 1 to 7.