Content auditing method and device, equipment, storage medium and computer program product

Abstract

The invention discloses a content auditing method, device and equipment, a storage medium and a computer program product, and the method comprises the steps: screening suspicious content from to-be-audited content, and confirming first content and second content; the first content is confirmed violation content, and the second content is to-be-confirmed violation content; inputting the second content into a plurality of language models for analysis; and processing the second content according to analysis results of the plurality of language models. According to the method, the suspicious content is quickly filtered out through preliminary screening, so that the first content and the second content are screened out according to the suspicious content, then the second content is input into the multiple language models for analysis, and finally the suspicious content is researched and judged according to the analysis results of the multiple language models. In conclusion, at least three-layer processing can be performed on the to-be-audited content, the content auditing quality can be effectively improved through a multi-layer processing mechanism, the manual auditing amount can be effectively reduced by introducing multiple language models, and therefore the auditing effect of improving the quality and improving the efficiency is achieved.

Description

Technical Field

[0001] This invention relates to the field of cloud technology, and in particular to a content moderation method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] With the explosive growth of online services, the scale of data has expanded dramatically, and the types and styles of media content have become increasingly diverse. At the same time, the surge in data volume has led to a corresponding increase in the amount of illegal content, with over ten million new illegal content generated daily. This has increased the risk of violations, and relying on traditional manual methods for interpreting illegal data is costly, inefficient, and lacks standardized procedures.

[0003] Existing technologies use full-text keyword matching for content review, but this method is slow, time-consuming, and resource-intensive, negatively impacting review speed. Furthermore, it often results in misjudgments during the review process, requiring subsequent manual review, which wastes resources and reduces efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a content review method, apparatus, device, storage medium, and computer program product that can effectively reduce the amount of manual review and significantly improve review efficiency.

[0005] To achieve the above objectives, embodiments of the present invention provide a content moderation method, including: Filter out suspicious content from the pending review materials; Based on the suspicious content, a first piece of content and a second piece of content are confirmed; wherein, the first piece of content is content that has been confirmed to be in violation, and the second piece of content is content that needs to be confirmed to be in violation; The second content is input into multiple language models for analysis; The second content is processed based on the analysis results of the multiple language models.

[0006] As an improvement to the above solution, the step of filtering out suspicious content from the content to be reviewed includes: Suspicious content is filtered out from the content to be reviewed by at least one of the following methods: blacklist / whitelist matching, keyword frequency calculation, and risk control algorithm model.

[0007] As an improvement to the above scheme, the blacklist and whitelist matching uses multiple preset first keywords to perform keyword matching on the content to be reviewed; the keyword frequency calculation uses the frequency of occurrence of second keywords in the content to be reviewed to filter the suspicious content.

[0008] As an improvement to the above solution, the risk control algorithm model is trained based on historical data as samples, and the risk control algorithm model is used to output the probability of violation risk of the suspicious content.

[0009] As an improvement to the above solution, the step of confirming the first content and the second content based on the suspicious content includes: Extract the semantic feature vector of the suspicious content, and calculate the similarity between the semantic feature vector and the sample feature vector; The suspicious content is classified into the first content and the second content based on the similarity.

[0010] As an improvement to the above solution, classifying the suspicious content into the first content and the second content based on the similarity includes: In response to the similarity being greater than or equal to the typification threshold, the suspicious content is classified as the first content; In response to the similarity being less than the typification threshold, the suspicious content is classified as the second content.

[0011] As an improvement to the above scheme, the sample feature vector is extracted from multiple sub-databases in the violation database.

[0012] As an improvement to the above scheme, the multiple sub-libraries correspond to different violation types, and the violation samples in each sub-library are periodically and dynamically updated.

[0013] As an improvement to the above scheme, the violation samples in each of the sub-libraries are periodically and dynamically updated, including: Remove non-compliant samples from the database within the first time limit; and / or Low-scoring samples are removed based on a value scoring model, which is calculated based on the sample's practical value, scarcity, redundancy, and timeliness.

[0014] As an improvement to the above scheme, the violation samples in each of the sub-libraries are periodically and dynamically updated, further including: The aforementioned non-compliant samples shall be retained in the database according to the second time limit; and / or The violation samples are retained based on their scarcity, where scarcity represents the distribution density of the violation samples.

[0015] As an improvement to the above scheme, the step of extracting the semantic feature vector of the suspicious content and calculating the similarity between the semantic feature vector and the sample feature vector includes: The semantic feature vector of the suspicious content is compared with the sample feature vector of each of the sub-libraries. The similarity between the semantic feature vector and the sample feature vector of each of the sub-libraries is calculated, and a corresponding similarity threshold is set for each of the sub-libraries. Based on the relationship between the similarity and the similarity threshold, the suspicious content is classified into the first content and the second content.

[0016] As an improvement to the above scheme, the number of the multiple language models is at least three heterogeneous language models.

[0017] As an improvement to the above scheme, each of the heterogeneous language models uses violation type, judgment criteria details, and typical violation examples as prompt words to review the second content for violations.

[0018] As an improvement to the above solution, the step of processing the second content based on the analysis results of the multiple language models includes: In response to the number of language models that determine the second content to be normal reaching a threshold, the second content is allowed; or In response to the fact that the number of language models that determine the second content to be normal has not reached a threshold, the second content is sent to the manual review queue.

[0019] As an improvement to the above solution, the content review method further includes: Construct a contradiction matrix between the results of manual review and the predictions of each of the stated language models; Perform three-dimensional contradiction detection based on the contradiction matrix; Based on the type of contradiction, generate prompt words to optimize instructions and update the prompt word library.

[0020] As an improvement to the above scheme, the three-dimensional contradictions include contradictions regarding validity, standard consistency, and counterexample coverage.

[0021] As an improvement to the above solution, the prompt word optimization instructions include: To address the contradiction of missing standards, optimization is achieved through feature induction. To address the ambiguity and contradictions in the standards, optimization is achieved through the ambiguity constraint method. To address the conflict and contradiction in the standards, optimization is performed using the counterexample boundary method.

[0022] This invention also provides a content moderation device, comprising: The filtering module is configured to filter out suspicious content from the content to be reviewed; The confirmation module is configured to: confirm first content and second content based on the suspicious content; wherein, the first content is content that has been confirmed to be in violation, and the second content is content that needs to be confirmed to be in violation; The analysis module is configured to input the second content into multiple language models for analysis. The processing module is configured to process the second content based on the analysis results of the multiple language models.

[0023] This invention also provides a terminal device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement any of the content moderation methods described above.

[0024] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute any of the above-described content moderation methods.

[0025] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the content moderation method described above.

[0026] Compared to existing technologies, the beneficial effects of the content review method, apparatus, device, storage medium, and computer program product provided by this invention are as follows: First, suspicious content is quickly filtered out through initial screening, thereby selecting first and second content based on the suspicious content. Then, the second content is input into multiple language models for analysis, and finally, the suspicious content is judged based on the analysis results of the multiple language models. In summary, this invention can perform at least three layers of processing on the content to be reviewed, effectively improving the quality of content review. The introduction of multiple language models also effectively reduces the amount of manual review, thus achieving an optimized effect of improving quality and efficiency. Attached Figure Description

[0027] Figure 1 This is a flowchart illustrating a preferred embodiment of a content moderation method provided by the present invention; Figure 2 This is a flowchart illustrating another preferred embodiment of a content moderation method provided by the present invention; Figure 3 This is a schematic diagram of a preferred embodiment of a content moderation device provided by the present invention; Figure 4 This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Please see Figure 1 , Figure 1 This is a flowchart illustrating a preferred embodiment of a content moderation method provided by the present invention. The content moderation method includes: Step S100: Filter out suspicious content from the content to be reviewed; Step S200: Confirm the first content and the second content based on the suspicious content, wherein the first content is the content that has been confirmed to be in violation, and the second content is the content that is yet to be confirmed to be in violation; Step S300: Input the second content into multiple language models for analysis; Step S400: Process the second content based on the analysis results of multiple language models.

[0030] Specifically, in step S100 of this invention, the input content, i.e., the content to be reviewed, is subjected to preliminary risk control screening to determine and filter out clearly illegal content, safe content, and suspicious content. For example, suspicious content in the content to be reviewed can be directly identified using the first keyword in the blacklist or whitelist. Clearly illegal content is handled through measures such as deletion, blocking, or warnings; safe content is allowed to proceed; and suspicious content is then processed through the subsequent steps of this invention.

[0031] In step S100, the input set of text to be reviewed is, for example, T = {t1, t2, ..., tn}. Then, the text to be reviewed is screened using a second keyword and related frequency strategy. For example, using a preset second keyword library K = {k1, k2, ..., km} and a frequency strategy, texts not matched by the blacklist or whitelist are scanned. For text ti∈T, its matching degree with the second keyword library and frequency is calculated. If the matching degree exceeds a set threshold Th1, it is directly determined as suspicious content and sent to the subsequent processing flow. It is understood that in this step, the frequency of occurrence of the second keyword can confirm whether the suspicious content violates regulations. In addition, this invention can also evaluate the text to be reviewed using a risk control algorithm model. For example, texts not matched by the blacklist or whitelist are input into the risk control algorithm model, which is trained based on historical data and can output the probability of violation Prf(ti). If Prf(ti)≥Trf (preset risk probability threshold), then mark it as "uncertain content" (i.e. suspicious content) and send it to the subsequent processing step S200.

[0032] In one embodiment, in step S200, the semantic feature vector of the suspicious content is extracted, and the similarity between the semantic feature vector and the sample feature vector is calculated; the suspicious content is then classified into first content and second content based on the similarity. Specifically, the suspicious content is classified as first content if the similarity is greater than or equal to a typification threshold; or it is classified as second content if the similarity is less than the typification threshold. It should be noted that the second content is content that needs further confirmation regarding its violation, while the first content is content that has been confirmed to be in violation.

[0033] Furthermore, the sample feature vector is extracted from multiple sub-libraries in the violation database. These sub-libraries correspond to different violation types, and the violation samples in each sub-library are periodically and dynamically updated.

[0034] It should be noted that step S200 is actually based on similarity matching of a dynamic historical violation database. Specifically, the construction and management of the dynamic violation content database includes: initial database cold start, value scoring initialization, and dynamic update engine (executed automatically daily).

[0035] For example, the data sources for the initial database, i.e. the newly created violation database, include: importing violation samples that have been manually reviewed and confirmed in the past, and pre-classifying them into the sub-database Hc according to type (political / terrorism / pornography, etc.).

[0036] Next, the data is structured: a quadruple {cnt, vt, ttype, tid} is generated for each sample.

[0037] Wherein, cnt is the original text, vt is a 512-dimensional feature vector generated by a semantic model (such as BERT), ttype is the violation type label (manually labeled), and tid is the current timestamp.

[0038] The above value scoring initialization includes: The new sample protection mechanism automatically grants newly added samples a 7-day protection period during which they are exempt from being discarded. Initial value assignment: Redundancy Rj=0 (no similar samples for comparison); Hit rate Hj=1 (initial value); Scarcity Cj is calculated based on sub-library density: if the sub-library sample size is <100, Cj=0.9 (forced high scarcity); otherwise, it is calculated according to the density formula.

[0039] The aforementioned dynamic update engines include: The value score for all samples is calculated using the following formula. Eliminate low-value samples: ; Among them, practical value (Hj): calculated based on the number of times the sample has been successfully matched recently, i.e., Hj=1+ln(1+xj); practical value (Hj) not only reflects the practical utility of the sample, but also suppresses the monopolistic influence of high-frequency samples through natural logarithm, ensuring that new samples have a basic survival value.

[0040] Scarcity (Cj): Calculated based on the distribution density of the sample within the same category of violating content, i.e., Cj = 1 - tanh(ρlj / ρmax_c); where ρlj represents the local density of sample j within category l of the same type of violating content, and ρmax_c represents the maximum density value within category c of the same type of violating content, used to normalize ρlj. Then, it is exponentially amplified ( Strengthen the weight of scarce samples to protect new and niche violation patterns.

[0041] Redundancy (Rj): The percentage of samples with semantic similarity ≥ 80% in the statistical database (Rj = Nsim / Ncj). By applying a power-square penalty, the score of duplicate samples is automatically reduced to optimize storage efficiency.

[0042] Timeliness (Agej): Directly taking the number of days since the sample was entered into the database, after exponential decay ( Reduce the value of outdated samples.

[0043] The final value score Vj achieves dynamic selection of the best and elimination of the worst. The numerator amplifies high-value and scarce samples (such as a surge in scores for new fraudulent tactics), while the denominator suppresses redundant and outdated samples (such as a sharp decrease in scores for duplicate advertising texts). This enables the violation database to autonomously retain high-value samples and eliminate inefficient data, continuously improving review efficiency and coverage.

[0044] For example, the phase-out strategy includes: Remove samples that have been stored for more than N=90 days (i.e., hard expired); Sort by Vj in ascending order and discard the last 10% of low-scoring samples (K=10%). Flexible adjustment: If the recall rate from manual review decreases on the same day, K will be reduced to 5%; High-value samples were retained, including: Permanent exemption: scarce samples with Cj > 0.8 (exemption can be obtained even if the expiration period is over); Extended protection: Samples will not be eliminated during the protection period; Add new samples: Inject the newly added violation samples from manual review on the same day into the corresponding sub-database, automatically generate the feature vector vf and initialize the value parameters.

[0045] It should be noted that the update process for the dynamic violation knowledge base (i.e., the violation database) also includes: performance monitoring and optimization. Specifically, when the sample size of a single sub-database is less than 500, the elimination ratio is appropriately relaxed; when the recall rate of manual review decreases, K is reduced; and long-tail violation cases are proactively mined and injected based on the proportion of scarce samples.

[0046] In one embodiment, classifying the first content and the second content further includes: Text semantic feature extraction: For the text to be inspected ti (i.e. suspicious content), a pre-trained language model (e.g. BERT) is used to extract its semantic feature vector vti.

[0047] Similarity calculation: For ti, traverse all sub-libraries of the historical violation library H. In each sub-library, calculate the cosine similarity Sij between the feature vector vti of ti and the feature vectors of all samples in the library.

[0048] Threshold determination: A similarity threshold T1 is preset for each violation type sub-library. For example, T1 can be dynamically adjusted according to the sub-library type. If Sij ≥ T1 exists in any sub-library, then ti is determined to be highly suspected of being a violation and is sent to the manual review queue.

[0049] Step S300 above is the collaborative voting and decision-making step of the large model. Before fine screening through the language model, it is necessary to complete the review standard prompting project, such as building a structured review prompt lexicon P={p1, p2, ...,p k Each prompt word p contains three elements: ① violation type, ② judgment criteria details, and ③ typical violation example. During the review process, the prompt words are formatted according to a preset template and input into the large model to accurately guide the violation judgment.

[0050] In this embodiment, the second round of screening is completed through a multi-model collaborative voting mechanism. Specifically, step S300 includes: a heterogeneous model jury formation step, which includes: selecting M (M≥3) heterogeneous large language models to form a voting committee LLM (Large Language Model) = {llm1, llm2, ..., llm m For example: general models (which can be large language models) and domain-specific fine-tuning models (which can be specialized models).

[0051] The voting and adjudication process includes: reviewing low-risk text samples. i Perform the following operations: Parallel inference: t i Synchronized input of each llm with the prompt word library P m ∈LLM; Output Analysis: Each model returns a predicted classification C. m (t i ) and according to E m (t i (The trigger word ID and specific rules need to be associated.) Voting decision: If the number of models judged as normal is greater than or equal to the confidence threshold T2, then automatically allow t. i Otherwise (if the threshold is not reached or the classification discrepancies are significant), it will be sent to the manual review queue, and the review results will feed back into the iterative optimization of the prompt word library P.

[0052] Please refer to Figure 2 , Figure 2 This is a flowchart illustrating another preferred embodiment of a content moderation method provided by the present invention. In this embodiment, the present invention further includes: Step S501: Construct a contradiction matrix between the manual review results and the predictions of each language model; Step S502: Perform three-dimensional contradiction detection based on the contradiction matrix; Step S503: Generate prompt word optimization instructions based on the contradiction type and update the prompt word library.

[0053] Steps S501 to S503 actually involve manually reviewing and optimizing the prompt word standards.

[0054] In one embodiment, in step S501, after reviewing the content submitted for manual review, the final content classification Ch is obtained. See Table 1 below. The predicted classification Ci of the sample and its prediction basis Ei (associated with specific prompt word terms Pi) of multiple LLM instances (LLM1, LLM2, ..., LLMm) are compared to construct a contradiction matrix.

[0055] Table 1

[0056] Where Ci = Ch ( ): This indicates that the LLM's predicted classification is consistent with the manual review result (correct).

[0057] Ci ≠ Ch ( ): This indicates that the predicted classification of the LLM is inconsistent with the results of manual review (error).

[0058] Furthermore, the calculation of basis-clause similarity is performed: the semantic similarity sim(Ei,PCh) is calculated between the basis text Ei output by the model and the prompt word PCh corresponding to the correct classification Ch.

[0059] The threshold determination rule is as follows: ; Where R = 1, it indicates that Ei is highly consistent with the prompt word PCh, which shows that it effectively supports the classification Ch; R = 0 indicates insufficient support for the prompt word PCh based on Ei, and the following situations exist: When the predicted classification Ci=Ch, invalid reasoning problems are exposed; When the predicted classification Ci≠Ch, the incorrect association becomes apparent.

[0060] In step S501, based on the contradiction matrix, the prediction results of LLM are analyzed in depth from three dimensions: contradiction of validity, contradiction of standard consistency, and contradiction of counterexample coverage. By using quantitative indicators, various potential problems in the prediction process of LLM are identified, providing a basis and decision guidance for subsequent targeted optimization of prompt word standards.

[0061] Dimension 1: Based on the contradiction of validity (whether the reasons are fabricated) Indicator: Percentage of invalid evidence for erroneous LLMs ; This metric measures the proportion of instances where LLM predictions are incorrect within a statistical period, based on invalid evidence. Specifically, it's the proportion of instances that not only predict incorrectly (Ci ≠ Ch) but also whose predictions are based on evidence that doesn't effectively support a correct classification (R(Ei,Ch)=0), out of all incorrectly predicted instances. If this proportion exceeds 0.7, a missing criteria alert is triggered, indicating that LLM lacks valid evidence among a large number of incorrect predictions.

[0062] Dimension 2: Inconsistent Standards (Is the wording inaccurate?) Metric: Conflict degree between the same prompt word category Input: Prompt word Pj, and its corresponding multi-model prediction classification set Cj={C1,C2,...,CN} (N≥2) Testing process: 1. Existence test of divergence: When the dispersion |Cj| ≥ 2 and |unique(Cj)| > 1, it is determined that there is a classification divergence between the models. Here, unique(Cj) is the set of classes after deduplication.

[0063] 2. Conflict score calculation: conflict_scorej = 1 - (max_class_countj / len(Cj)), where max_class_countj is the number of occurrences of the class that appears most frequently in Cj (denoted as max_classj), and len(Cj) is the length of the class set Cj.

[0064] 3. Conditions for triggering fuzzy alarms (must be met simultaneously): conflict_scorej > 0.3 and max_classj is not equal to the correct classification Chj.

[0065] When an alarm is triggered, it indicates that the prompt word Pj has a problem with ambiguity in its standard expression: A high degree of conflict (>0.3) indicates inconsistency in the model's understanding; High-frequency classification errors indicate that the standard has failed to guide correct judgment; The logic framework and examples for handling actions need to be reconstructed.

[0066] For example, the standard fuzzy formula for prompt words is expressed as:

[0067] Dimension 3: Counterexamples Covering Contradictions (Is there a lack of constraints?) Metric: Proportion of incorrect associations with prompt words Input: Historical references of the prompt word Pk Testing process: 1. Statistical calculation: Tktotal: The total number of times Pk is referenced by the LLM; Tkerr: The number of times referencing Pk has caused misclassification.

[0068] 2. Risk Assessment: ; Here, 0.4 is the configurable risk threshold.

[0069] 3. Handling actions: If RiskFlagk=1, it indicates that the prompt word has a standard conflict, which is likely to mislead LLM to produce incorrect classification. Pk can be added to the risk prompt word set Rrisk. At the same time, a negative example missing alarm is triggered, and the prompt word optimization process (i.e., supplementing negative example constraints) is started.

[0070] In step S503, the optimized instruction generation rules include: Input: Contradiction type + historical sample set + original prompt words; Processing engine: Large Language Model (LLM) optimization suggestion generator; Output: Structured optimization instructions; Specifically, the conflict type-oriented optimization rules include: To address the contradiction of missing standards, optimization is achieved through feature induction. For example, the instruction template is: when text simultaneously satisfies both key semantic features (feature clustering results) and discrimination patterns (high-frequency pattern descriptions), it should be classified as Ch to ensure that the features cover more than 85% of positive samples.

[0071] To address the ambiguity and contradictions in the standards, optimization is achieved through ambiguity constraints. For example, the instruction template is: Supplementary decision constraint: If feature F (conflicting feature) exists, it is preferentially classified as Ch, because F and Ch are strongly correlated, ensuring that F's occurrence rate in conflict cases is >70%.

[0072] To address the conflict and contradiction in the standards, optimization is achieved using the counterexample boundary method. For example, the instruction template is: Add an exception clause: When an erroneous associated feature X occurs, it does not belong to Ch even if the main condition is met, i.e., a counterexample sample {neg}, to ensure that its occurrence rate in erroneous cases is greater than 90%.

[0073] Based on the above rules, a new prompt word judgment standard P_candidate is obtained.

[0074] Furthermore, this invention can also verify its effectiveness by calculating the accuracy (≥95%) and coverage (≥5%) of the candidate prompt word P_candidate in the historical case library. If both indicators are met, the standard library is updated; otherwise, feature optimization iteration is initiated, forming a closed-loop feedback system of generation and verification, thus completing effective backtracking verification.

[0075] In summary, this invention constructs a dynamic three-dimensional value assessment and intelligent update mechanism for a violation database. It also establishes the computational logic of a value scoring model based on redundancy, historical hit rate, relative sparsity, and aging factors, along with its driving dynamic update strategy (including a new sample protection period, permanent exemption for highly scarce samples, and dynamic adjustment of elimination ratio rules). Furthermore, this invention provides a multi-LLM conflict arbitration and basis association analysis review mechanism, including a forced association prediction basis and a conflict matrix structure between specific prompt terms, as well as a quantitative calculation method for basis association (judging basis validity based on semantic similarity thresholds). This is the core basis for achieving in-depth error diagnosis. After review, this invention can also perform prompt word self-optimization based on three-dimensional conflict detection, automatically triggering a conflict detection algorithm for prompt word optimization (including quantitative indicators and threshold judgment rules for basis validity conflicts, standard consistency conflicts, and counterexample coverage conflicts) and differentiated optimization instruction generation rules (instruction logic for generating new rule templates when standards are missing, supplementing feature constraints when standards are ambiguous, and adding counterexamples when standards conflict). Therefore, this invention can achieve a progressive risk filtering process (initial screening layer interception, fine screening layer I similarity matching, fine screening layer II multi-LLM voting threshold release) and its resource scheduling principle (only "uncertain content" enters the high-cost layer) through a four-level hierarchical processing architecture and resource allocation logic, thus achieving a balanced architecture between cost and effectiveness. In other words, this invention provides a self-evolving system workflow and method with a closed-loop system interaction throughout the entire process: "dynamic library update → multi-model arbitration → manual feedback → contradiction detection → prompt word optimization → backtracking verification → library / model enhancement".

[0076] Accordingly, the present invention also provides a content moderation device capable of implementing all processes of the content moderation method in the above embodiments.

[0077] Please see Figure 3 , Figure 3 This is a schematic diagram of a preferred embodiment of a content moderation device provided by the present invention. The content moderation device includes: The filtering module 301 is configured to filter out suspicious content from the content to be reviewed; The confirmation module 302 is configured to confirm first content and second content based on suspicious content; wherein, the first content is content that has been confirmed to be in violation, and the second content is content that is yet to be confirmed to be in violation; Analysis module 303 is configured to input the second content into multiple language models for analysis; Processing module 304 is configured to process the second content based on the analysis results of multiple language models.

[0078] Preferably, the step of filtering out suspicious content from the content to be reviewed includes: Suspicious content is filtered out from the content to be reviewed by at least one of the following methods: blacklist / whitelist matching, keyword frequency calculation, and risk control algorithm model.

[0079] Preferably, the blacklist / whitelist matching uses multiple preset first keywords to perform keyword matching on the content to be reviewed; the keyword frequency calculation uses the frequency of occurrence of second keywords in the content to be reviewed to filter the suspicious content.

[0080] Preferably, the risk control algorithm model is trained based on historical data as samples, and the risk control algorithm model is used to output the probability of violation risk of the suspicious content.

[0081] Preferably, confirming the first content and the second content based on the suspicious content includes: Extract the semantic feature vector of the suspicious content, and calculate the similarity between the semantic feature vector and the sample feature vector; The suspicious content is classified into the first content and the second content based on the similarity.

[0082] Preferably, classifying the suspicious content into the first content and the second content based on the similarity includes: In response to the similarity being greater than or equal to the typification threshold, the suspicious content is classified as the first content; In response to the similarity being less than the typification threshold, the suspicious content is classified as the second content.

[0083] Preferably, the sample feature vector is extracted from multiple sub-databases in the violation database.

[0084] Preferably, the multiple sub-libraries correspond to different violation types, and the violation samples in each sub-library are periodically and dynamically updated.

[0085] Preferably, the violation samples in each of the sub-databases are periodically and dynamically updated, including: Remove non-compliant samples from the database within the first time limit; and / or Low-scoring samples are removed based on a value scoring model, which is calculated based on the sample's practical value, scarcity, redundancy, and timeliness.

[0086] Preferably, the violation samples in each of the sub-databases are periodically and dynamically updated, further comprising: The aforementioned non-compliant samples shall be retained in the database according to the second time limit; and / or The violation samples are retained based on their scarcity, where scarcity represents the distribution density of the violation samples.

[0087] Preferably, the step of extracting the semantic feature vector of the suspicious content and calculating the similarity between the semantic feature vector and the sample feature vector includes: The semantic feature vector of the suspicious content is compared with the sample feature vector of each of the sub-libraries. The similarity between the semantic feature vector and the sample feature vector of each of the sub-libraries is calculated, and a corresponding similarity threshold is set for each of the sub-libraries. Based on the relationship between the similarity and the similarity threshold, the suspicious content is classified into the first content and the second content.

[0088] Preferably, the number of the plurality of language models is at least three heterogeneous language models.

[0089] Preferably, each of the heterogeneous language models performs violation review on the second content based on violation type, judgment criteria details, and typical violation examples as prompt words.

[0090] Preferably, processing the second content based on the analysis results of the multiple language models includes: In response to the number of language models that determine the second content to be normal reaching a threshold, the second content is allowed; or In response to the fact that the number of language models that determine the second content to be normal has not reached a threshold, the second content is sent to the manual review queue.

[0091] Preferably, the content review device further includes an update module, used for: Construct a contradiction matrix between the results of manual review and the predictions of each of the stated language models; Perform three-dimensional contradiction detection based on the contradiction matrix; Based on the type of contradiction, generate prompt words to optimize instructions and update the prompt word library.

[0092] Preferably, the three-dimensional contradictions include contradictions regarding validity, standard consistency, and counterexample coverage.

[0093] Preferably, the prompt word optimization instruction includes: To address the contradiction of missing standards, optimization is achieved through feature induction. To address the ambiguity and contradictions in the standards, optimization is achieved through the ambiguity constraint method. To address the conflict and contradiction in the standards, optimization is performed using the counterexample boundary method.

[0094] In specific implementation, the working principle, control process and technical effects of the content moderation device provided in the embodiments of the present invention are the same as those of the content moderation method in the above embodiments, and will not be repeated here.

[0095] Please see Figure 4 , Figure 4This is a schematic diagram of a preferred embodiment of a terminal device provided by the present invention. The terminal device includes a processor 401, a memory 402, and a computer program stored in the memory 402 and configured to be executed by the processor 401. When the processor 401 executes the computer program, it implements the content moderation method described in any of the above embodiments.

[0096] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory 402 and executed by the processor 401 to complete the present invention. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, and the instruction segments are used to describe the execution process of the computer program in the terminal device.

[0097] The processor 401 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor, or the processor 401 can be any conventional processor. The processor 401 is the control center of the terminal device, connecting various parts of the terminal device through various interfaces and lines.

[0098] The memory 402 mainly includes a program storage area and a data storage area. The program storage area can store the operating system, applications required for at least one function, etc., while the data storage area can store related data, etc. Furthermore, the memory 402 can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart media card (SMC), a secure digital card (SD), and a flash card, or it can be other volatile solid-state storage devices.

[0099] It should be noted that the aforementioned terminal devices may include, but are not limited to, processors and memory, as will be understood by those skilled in the art. Figure 4The structural diagram is merely an example of the terminal device described above and does not constitute a limitation on the terminal device described above. It may include more or fewer components than shown in the diagram, or combine certain components, or use different components.

[0100] This invention also provides a computer-readable storage medium, which includes a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to execute the content moderation method described in any of the above embodiments.

[0101] This invention also provides a computer program product, which includes a computer program or computer instructions. When the computer program or computer instructions are executed by a processor, they implement the content moderation method described in any of the above embodiments.

[0102] This invention provides a content moderation method, apparatus, device, storage medium, and computer program product, which achieves core optimization of content moderation through "dynamic intelligent management of a violation database + multi-LLM collaborative conflict arbitration + self-optimization closed loop of prompt words". Specifically, it can achieve the following advantages: 1. Significantly reduced operating costs: Through four-level tiered filtering (initial screening to intercept obvious violations, fine screening layer I to process known patterns, and fine screening layer II to automatically adjudicate compliant content) and a dynamic violation database to intelligently eliminate redundant samples, the amount of manual review is significantly reduced and storage matching costs are lowered, while optimizing the frequency of high-performance LLM calls.

[0103] 2. Significant improvement in review quality and efficiency: Accurate matching of historical databases ensures extremely high accuracy in identifying known violations; multi-LLM heterogeneous voting effectively reduces the false positive rate and covers new types of ambiguous risks; automatic optimization and backtracking verification of prompts based on contradiction detection (invalid evidence / ambiguous standards / missing counterexamples) will enable coverage of new violations within 24 hours, resulting in rapid policy response.

[0104] 3. Enhanced Compliance Resilience: A layered risk control system (covering explicit, historical, and emerging risks) and a detailed contradiction matrix record (LLM decision-making basis vs. manual results) build a comprehensive risk defense and strong audit traceability capability, while value-driven violation database management ensures compliance with regulatory sample retention requirements.

[0105] 4. Faster technology iteration: The dynamic violation database's three-dimensional value assessment and intelligent update strategy, along with the automated prompt word diagnosis and optimization closed loop (contradiction detection -> precise optimization -> effect verification), form a self-iteratory improvement and optimization system, significantly reducing manual rule maintenance.

[0106] 5. Flexible application scenarios: This invention can be flexibly applied to scenarios such as Internet and social text moderation (to deal with emerging content), financial compliance quality inspection (accurate matching of text messages), cross-border content review (rapid localization of prompt words), and AIGC risk control (identifying logical contradictions). It can enter emerging markets through cloud APIs and private deployment solutions, effectively addressing the challenges of complex content security and regulation.

[0107] In summary, compared to existing technologies, this invention pioneers a three-dimensional value assessment system and implements an update strategy that combines a new sample protection period, permanent exemption for highly scarce samples, and a dynamic elimination ratio. This addresses the pain point of existing technologies in balancing sample freshness, practical value, and scarcity, significantly improving library utilization efficiency and ensuring that the review effect does not diminish. Compared to simple multi-model voting that relies on a single strong model or only statistically analyzes classification results, this invention also emphasizes recording the basis for each model's predictions and associating them with specific prompt terms, constructing a contradiction matrix, and introducing an assessment of the correlation between the basis and the predictions. This not only avoids single-model bias but also deeply diagnoses the root causes of errors (invalid basis / ambiguous standards / missing counterexamples), providing direction for subsequent precise optimization, reducing the false positive rate, and enabling the system to self-evolve. Furthermore, compared to existing technologies that rely entirely on manual experience to maintain review rules, this invention automatically performs three-dimensional contradiction detection based on the contradiction matrix and precisely triggers differentiated optimization actions (generating new rules / rewriting prompt terms / adding counterexamples) according to the contradiction type (missing standards / ambiguity / conflict), and finally achieves continuous autonomous optimization of review standards through rigorous backtracking verification. Therefore, this invention deeply integrates three core technologies: dynamic library management (ensuring knowledge quality), multi-model arbitration (enhancing complex judgment capabilities), and prompt word self-optimization (achieving rule evolution). These three technologies reinforce each other to form a closed loop (optimized rules improve model judgment capabilities, and high-quality judgment feedback drives library and rule updates), systematically solving the industry problem of balancing effectiveness, cost, and adaptability, and creating an overall advantage that is difficult to replicate.

[0108] It should be noted that the system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the system embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0109] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A content moderation method, characterized in that, include: Filter out suspicious content from the pending review materials; Based on the suspicious content, a first piece of content and a second piece of content are confirmed; wherein, the first piece of content is content that has been confirmed to be in violation, and the second piece of content is content that needs to be confirmed to be in violation; The second content is input into multiple language models for analysis; The second content is processed based on the analysis results of the multiple language models.

2. The content moderation method as described in claim 1, characterized in that, The process of filtering out suspicious content from the content to be reviewed includes: Suspicious content is filtered out from the content to be reviewed by at least one of the following methods: blacklist / whitelist matching, keyword frequency calculation, and risk control algorithm model.

3. The content moderation method as described in claim 2, characterized in that, The blacklist and whitelist matching uses multiple preset first keywords to perform keyword matching on the content to be reviewed; the keyword frequency calculation uses the frequency of occurrence of second keywords in the content to be reviewed to filter the suspicious content.

4. The content review method as described in claim 2, characterized in that, The risk control algorithm model is trained based on historical data as samples, and is used to output the probability of violation risk of the suspicious content.

5. The content moderation method as described in claim 1, characterized in that, The step of confirming the first content and the second content based on the suspicious content includes: Extract the semantic feature vector of the suspicious content, and calculate the similarity between the semantic feature vector and the sample feature vector; The suspicious content is classified into the first content and the second content based on the similarity.

6. The content moderation method as described in claim 5, characterized in that, The step of classifying the suspicious content into the first content and the second content based on the similarity includes: In response to the similarity being greater than or equal to the typification threshold, the suspicious content is classified as the first content; In response to the similarity being less than the typification threshold, the suspicious content is classified as the second content.

7. The content moderation method as described in claim 5, characterized in that, The sample feature vector is extracted from multiple sub-databases in the violation database.

8. The content moderation method as described in claim 7, characterized in that, The multiple sub-libraries correspond to different violation types, and the violation samples in each sub-library are dynamically updated periodically.

9. The content moderation method as described in claim 8, characterized in that, The violation samples in each of the aforementioned sub-databases are periodically and dynamically updated, including: Remove non-compliant samples from the database within the first time limit; and / or Low-scoring samples are removed based on a value scoring model, which is calculated based on the sample's practical value, scarcity, redundancy, and timeliness.

10. The content moderation method as described in claim 8, characterized in that, The violation samples in each of the aforementioned sub-libraries are periodically and dynamically updated, and the system also includes: The aforementioned non-compliant samples shall be retained in the database according to the second time limit; and / or The violation samples are retained based on their scarcity, where scarcity represents the distribution density of the violation samples.

11. The content moderation method as described in claim 7, characterized in that, The step of extracting the semantic feature vector of the suspicious content and calculating the similarity between the semantic feature vector and the sample feature vector includes: The semantic feature vector of the suspicious content is compared with the sample feature vector of each of the sub-libraries. The similarity between the semantic feature vector and the sample feature vector of each of the sub-libraries is calculated, and a corresponding similarity threshold is set for each of the sub-libraries. Based on the relationship between the similarity and the similarity threshold, the suspicious content is classified into the first content and the second content.

12. The content moderation method as described in claim 1, characterized in that, The number of the multiple language models is at least three heterogeneous language models.

13. The content moderation method as described in claim 12, characterized in that, Each of the heterogeneous language models uses violation type, judgment criteria, and typical violation examples as prompts to review the second content for violations.

14. The content moderation method as described in claim 12, characterized in that, The process of processing the second content based on the analysis results of the multiple language models includes: In response to the number of language models that determine the second content to be normal reaching a threshold, the second content is allowed; or In response to the fact that the number of language models that determine the second content to be normal has not reached a threshold, the second content is sent to the manual review queue.

15. The content moderation method as described in claim 14, characterized in that, The content review methods also include: Construct a contradiction matrix between the results of manual review and the predictions of each of the stated language models; Perform three-dimensional contradiction detection based on the contradiction matrix; Based on the type of contradiction, generate prompt words to optimize instructions and update the prompt word library.

16. The content moderation method as described in claim 15, characterized in that, The three-dimensional contradictions include contradictions regarding validity, standard consistency, and counterexample coverage.

17. The content moderation method as described in claim 15, characterized in that, The prompt word optimization instructions include: To address the contradiction of missing standards, optimization is achieved through feature induction. To address the ambiguity and contradictions in the standards, optimization is achieved through the ambiguity constraint method. To address the conflict and contradiction in the standards, optimization is performed using the counterexample boundary method.

18. A content moderation device, characterized in that, include: The filtering module is configured to filter out suspicious content from the content to be reviewed; The confirmation module is configured to confirm first content and second content based on the suspicious content; wherein, the first content is content that has been confirmed to be in violation, and the second content is content that needs to be confirmed to be in violation; The analysis module is configured to input the second content into multiple language models for analysis; The processing module is configured to process the second content based on the analysis results of the multiple language models.

19. A terminal device, characterized in that, The device includes a processor and a memory, the memory storing a computer program configured to be executed by the processor, wherein the processor, when executing the computer program, implements the content moderation method as described in any one of claims 1 to 17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein when the device containing the computer-readable storage medium executes the computer program, it implements the content moderation method as described in any one of claims 1 to 17.

21. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which, when executed by a processor, implement the content moderation method as described in any one of claims 1 to 17.

Images