Large model generation content auditing method based on retrieval enhancement, medium and equipment

Abstract

The invention provides a retrieval enhancement-based large model generation content auditing method, medium and equipment, and belongs to the technical field of large language models. The method comprises the following steps: firstly, vectorizing a text through a pre-training embedding model to capture deep semantics; then, carrying out multi-label security classification by utilizing the classification model subjected to large-scale content security data fine adjustment; preliminarily screening a candidate set from a case library by taking a classification result as metadata; dynamically determining a case set which is finally used for analogy reasoning by calculating information gains of candidate cases relative to a selected case set; a to-be-audited text and selected positive and negative cases are jointly constructed into a reasoning enhancement prompt, the reasoning enhancement prompt is input into an instruction fine-tuning large model for deep analogy reasoning, finally, a quantitative security score is output, and grading decision and automatic manual intervention routing are achieved according to a dynamic threshold value. According to the method, the dependence on high-cost and high-intensity manual auditing is effectively reduced, and efficient, extensible and high-robustness content security guarantee is realized.

Description

Technical Field

[0001] This invention belongs to the field of large language model technology, specifically relating to a method, medium, and device for content review based on retrieval enhancement in large model generation. Background Technology

[0002] In today's era, the rapid development of large language model technology has profoundly impacted and permeated numerous industries, fostering deep integration between them. In this process, the security of content generated by large language models has become a focal point of concern. Faced with this challenge, ensuring the compliance and security of the model's output content is of paramount importance.

[0003] Currently, content moderation methods for large language model-generated content mainly include manual moderation, rule-driven moderation, and machine learning model moderation. Manual moderation: Performed by professionally trained content moderators, but it relies on a large amount of human resources and is inefficient, making it difficult to match the high-speed generation of large-scale content. Furthermore, inconsistent standards among moderators can easily lead to biased results. Rule-driven moderation: Relies on preset rules and keywords, lacking flexibility, requiring frequent updates and maintenance, and exhibiting a high error rate when dealing with puns, metaphors, or complex contexts. Machine learning model moderation: While utilizing advanced algorithms to identify content safety, it is highly dependent on training data, and the model's generalization ability is limited, making it difficult to handle unknown content types or styles. Crucially, the model's decision-making process lacks transparency, and the reasons for content judgments are difficult to trace.

[0004] In summary, content moderation plays a crucial role in ensuring the safety, compliance, and ethical alignment of the outputs of generative artificial intelligence (AI) models. However, existing technological solutions face several significant challenges in accurately and efficiently defining content safety boundaries: First, the semantic complexity and diversity of content generated by large language models make traditional filtering methods based on keywords or static rules prone to misjudgment and ineffective in dealing with novel and subtle violations; second, with the widespread application and iteration of generative AI technology, continuous and large-scale manual review of its output content is not only extremely costly, but also makes it difficult to maintain consistent review standards, leading to fluctuations in review quality. Summary of the Invention

[0005] This invention addresses the shortcomings of existing technologies by providing a method, medium, and device for content review based on a large model generated through retrieval enhancement. It reviews model-generated content using retrieval enhancement techniques. Compared to traditional review methods, this invention offers greater flexibility, significantly improving review accuracy while effectively reducing review costs. More importantly, this invention transforms the review process from a traditional black-box model to a transparent white-box model, ensuring the openness and traceability of the detection process and greatly enhancing the transparency and reliability of the entire review system.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a method for content moderation based on a large model generated by retrieval enhancement, characterized by comprising the following steps: Step 1: Convert the original text generated by the large language model to be detected into a high-dimensional semantic vector; Step 2: Use a classification model to perform a safe classification of the original text and output the safe classification results; Step 3: Select candidate cases from the pre-built labeled case knowledge base based on the security classification results to form an initial screening set; Step 4: Calculate the semantic similarity between candidate cases and the original text, and rank the candidate cases according to the semantic similarity. Further filter the ranked candidate cases and dynamically retrieve the case set for reasoning by calculating the information gain of the candidate cases relative to the selected cases. Step 5: Perform deep semantic re-ranking on the case set retrieved in Step 4, and select the top-ranked cases from the re-ranked list as the selected case set; Step 6: Based on the original text, combined with the selected case set from Step 5, incorporate reasoning step prompts to construct enhanced reasoning prompts; Step 7: Input the reasoning enhancement hints constructed in Step 6 into the large language model and output a quantified safety score; Step 8: Determine the security level of the corresponding original text by comparing the security score with a preset threshold.

[0007] Optionally, in step 1, the original text generated by the detected large language model is converted into a high-dimensional semantic vector using the Embedding model.

[0008] Optionally, in step 2, the classification model adopts the Qwen2.5-7B model.

[0009] Optionally, in step 3, the candidate cases include positive cases and negative cases. Negative cases refer to cases in which the content fully complies with security standards in the corresponding category, while positive cases refer to cases in which the content is judged to have security risks or violations in the same category.

[0010] Optionally, step 4 includes the following sub-steps: Step 4.1: Calculate the macro-semantic similarity and micro-semantic similarity between the high-dimensional semantic vector and the vectors in the initial screening set. Combine the macro-semantic similarity and micro-semantic similarity to calculate the semantic similarity between the candidate cases and the original text, and sort the candidate cases accordingly to recall a list of candidate cases sorted in descending order of semantic similarity. Here, macro-semantic similarity is used to calculate the semantic similarity of documents or paragraphs, and micro-semantic similarity is used to calculate the semantic similarity of sentences or phrases. Step 4.2: Initialize the final case set S_final, starting with the candidate case with the highest semantic similarity, and iteratively process each case candidate_i in the candidate case list in order; Step 4.3: Start the retrieval process. Calculate the information gain of each case candidate_i relative to all cases currently selected into S_final. If its semantic similarity is higher than the preset minimum threshold and its information gain is higher than the preset dynamic threshold, then candidate_i will be selected into S_final. Step 4.4: When the candidate case list has been traversed or the information gain of multiple consecutive cases is lower than the dynamic threshold, the retrieval process ends. The number of cases in S_final is used as the dynamically determined K value, representing the size of the case set used for reasoning.

[0011] Optionally, in step 5, the cases in S_final after the search process ends are subjected to deep semantic reordering, and the top-ranked cases are selected from the reordered list as the selected case set.

[0012] Optionally, in step 6, an inference enhancement hint is constructed using a CoT template.

[0013] Optionally, in step 8, by comparing the security score with a preset threshold, the security scores are divided into a high-score security range, a medium-score review range, and a low-score risk range from high to low. The original text corresponding to the high-score security range is determined to be safe and automatically approved by the system. The original text corresponding to the medium-score review range is determined to have potential risks, marked, and routed to the manual review queue. The original text corresponding to the low-score risk range is determined to be highly likely to be unsafe, and is directly intercepted or marked as high-risk.

[0014] In a second aspect, the present invention provides a computer-readable storage medium storing a computer program that causes a computer to perform the content moderation method for generating large models based on retrieval enhancement as described in the first aspect.

[0015] Thirdly, the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the content review method for generating large models based on retrieval enhancement as described in the first aspect.

[0016] The beneficial effects of this invention are: (1) Use search enhancement to review the security of content generated by large models. This makes the review process of content generated by large models more flexible, consistent and transparent, and reduces the review cost while greatly improving the accuracy of the review.

[0017] (2) Enhancing the relevance and information diversity of case retrieval: The innovative dynamic case retrieval mechanism, through a dual screening of semantic similarity and information gain, ensures that the final retrieved case set is not only highly relevant to the text to be retrieved, but also contains the maximum amount of unique and complementary information. This breaks through the limitations of information redundancy that may be caused by traditional retrieval relying solely on similarity, and provides richer and more discriminative contextual references for subsequent reasoning.

[0018] (3) Optimize system efficiency and resource utilization: The initial screening based on metadata quickly narrows the search scope; the mechanism of "dynamically determining the K value" avoids the waste of resources or insufficient information caused by a fixed number of cases, enabling the system to adaptively adjust the search scale according to the complexity of the input text, thereby improving processing efficiency while ensuring effectiveness.

[0019] (4) Enhancing the interpretability and guidance of the decision-making process: By introducing carefully selected Few-Shot positive and negative cases and structured CoT inference templates, clear and high-quality discriminative references and inference paths are provided for large models. This not only enhances the credibility and interpretability of the final safety score decision, but also effectively guides the model to establish a more stable and reliable inference paradigm. Attached Figure Description

[0020] Figure 1 This is a flowchart of the content review method for generating large models based on retrieval enhancement according to the present invention.

[0021] Figure 2 This is a flowchart of the recall strategy of the present invention.

[0022] Figure 3 This is a flowchart of the preprocessing for data entry in this invention.

[0023] Figure 4 This is a schematic diagram of the database entry prompt words in the case of this invention. Detailed Implementation

[0024] The technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings.

[0025] In one embodiment, the present invention proposes a method for content moderation based on a large model generated by retrieval enhancement, the process of which is as follows: Figure 1 As shown, the specific steps include the following: 1. Input Acquisition and Vectorization: The text content generated by the large language model to be detected is used as the system input. The Embedding model is used to convert the text into a high-dimensional semantic vector (i.e., the input vector) to capture deep semantic information.

[0026] Embedding is a technique that maps words or phrases in text to continuous vector representations in a mathematical space. These vectors can effectively capture the semantic relationships and syntactic features between words, thereby converting natural language into a numerical form that can be processed by computers, providing basic support for downstream tasks such as text classification and semantic retrieval.

[0027] 2. Text Content Security Classification: Based on the Qwen2.5 (7B) model, which has been fine-tuned with large-scale content security data, multi-label security classification is performed on the input text. This model can accurately identify whether the text involves multiple violation categories such as sensitive topics and hate speech, and output the confidence score of each category, providing a classification basis for subsequent case retrieval.

[0028] 3. Initial screening of the case pool based on metadata: Based on the content security classification results output in step 2, candidate positive and negative case sets are efficiently screened from the pre-built labeled case knowledge base. Negative cases refer to cases in this category whose content fully complies with security standards; while positive cases are cases in the same category whose content is judged to have security risks or violations.

[0029] 4. Dynamic case retrieval mechanism based on dual-granularity similarity combined with information gain maximization: This step is the core innovation of this invention, aiming to retrieve the optimal set of positive and negative cases for the input text, the size of which, K, is dynamically determined by the algorithm. This process is applied independently and in parallel to the positive and negative candidate sets.

[0030] 4.1 Preliminary Recall: Based on micro-query (precise matching of key entities and phrases) and macro-query (topic summary matching), a large list of candidate cases, sorted in descending order of semantic similarity, is recalled from the initial screening set.

[0031] 4.2 Iterative Dynamic Set Construction: Initialize an empty final case set S_final. Starting with the case with the highest similarity, iteratively process each case in the candidate list in order.

[0032] 4.3 Information Gain Calculation and Selection Judgment: For each candidate case (candidate_i), calculate its information gain (IG) relative to all cases currently selected into the S_final set. IG is used to quantify the amount of new and unique information that candidate_i can provide. A candidate case is selected into S_final if it simultaneously satisfies the following conditions: its semantic similarity is higher than the preset minimum threshold θ_sim, and its information gain (IG) is higher than the preset dynamic threshold θ_IG.

[0033] Information gain is a concept based on entropy (uncertainty), which quantifies the reduction in uncertainty that can be achieved by using a certain feature to partition data. It is a fundamental and important tool in feature selection and decision tree model construction, helping to find the key features that best "distinguish" the data.

[0034] 4.4 Dynamically Determining the K Value: The retrieval process continues until the candidate list is exhausted or the IG of multiple consecutive candidate cases is lower than θ_IG (indicating that information addition has reached saturation). Finally, the number of cases in S_final is the dynamically determined K value. This strategy ensures that the final set possesses both high relevance and high information diversity.

[0035] 5. Case Refinement Based on Reranking Model: A finely tuned Rerank model is used to perform deep semantic reranking on the positive and negative case sets dynamically retrieved in step 4. Subsequently, the top 5 positive cases and 5 negative cases are selected from the reranked list to form the "essential case set" for final reasoning.

[0036] Rerank models are typically used in scenarios such as information retrieval and recommender systems where refined ranking of candidate results is required. The core idea is to use more complex models or strategies to perform a secondary ranking of results based on the initial retrieval or ranking, thereby improving the accuracy of the final result and user satisfaction.

[0037] 6. Construct reasoning-enhancing hints: 6.1 Text to be detected: Directly import the original text content obtained in step 1.

[0038] 6.2. Selected Case Set: Employing a Few-Shot (few-shot learning) mechanism, five positive examples and five negative examples are carefully selected from the output of step 5 to form a highly discriminative case set. These cases have undergone multiple rounds of evaluation and screening in the early stages, and are representative in dimensions such as logical consistency, clarity of intent, and semantic rationality. They can provide high-quality discriminative references for the model and effectively guide it to establish correct reasoning paradigms.

[0039] Few-shot refers to using only a small number of examples during model training or inference to guide the model to understand the task and generate the expected results. It is often used in scenarios where resources are limited or labeled data is scarce to improve the model's generalization ability and adaptation efficiency.

[0040] 6.3 CoT (Chain of Thought) Template: A structured instruction that guides the model through step-by-step reasoning.

[0041] CoT is a concept originating from cognitive psychology, referring to the complete cognitive path by which an individual gradually forms an understanding of a problem and a solution through a series of logical reasoning, association, and judgment steps during the problem-solving process.

[0042] 7. Security score of the instruction fine-tuning model: The final prompt word generated in step 6 is input into a large language model that has been fine-tuned by instructions. This model performs deep analogical reasoning based on rich context and outputs a quantitative security score.

[0043] 8. Tiered Decision-Making and Manual Intervention: The system compares the security score output by the model with preset thresholds and automatically executes tiered decisions: high scores are automatically approved, medium scores trigger manual review, and low scores are automatically blocked or trigger an alarm. High-score security range (e.g., ≥80 points): Content is deemed safe and automatically approved by the system without manual intervention. Medium-score review range (e.g., 60 to 80 points): Content is deemed to have potential risks; the system marks it and routes it to the manual review queue for final correction and judgment by reviewers. Low-score risk range (e.g., <60 points): Content is deemed highly potentially unsafe; the system can directly block it or mark it as high-risk.

[0044] The core innovation of this method lies in the introduction of a dynamic case retrieval mechanism based on dual-granularity similarity combined with information gain maximization. This mechanism can dynamically and adaptively retrieve the positive examples (violations of the security type) and negative examples (non-violations of the security type) with the highest information content and strongest similarity from a pre-built, finely annotated case knowledge base, rather than simply relying on static similarity matching. Specifically, this method first vectorizes the text through a pre-trained embedding model to capture deep semantics; then, it uses a classification model finely tuned with large-scale content security data to perform multi-label security classification; subsequently, it uses the aforementioned classification results as metadata to initially screen the candidate set from the case library; the key step is to first calculate the macro (document-level / paragraph-level) and micro (sentence-level / phrase-level) semantic similarity between the original text vector and the case library vector, respectively, and initially capture semantic associations through dual-granularity matching. Based on preliminary similarity results, an information gain evaluation mechanism is used to dynamically determine the candidate set size K. The dynamically selected Top-K candidate texts are then refined using a Rerank model to ensure that the selected cases have both high relevance and high information diversity, effectively avoiding the interference of redundant cases on the reasoning process. Finally, based on the original text to be reviewed, combined with a carefully selected set of positive and negative cases, and incorporating clear reasoning step prompts, an efficient reasoning enhancement prompt is constructed. This prompt is input into a large model for instruction fine-tuning for deep analogical reasoning, ultimately outputting a quantified safety score. Based on a dynamic threshold, hierarchical decision-making and automated manual intervention routing are implemented. Compared to traditional methods that rely on single-model classification or purely manual review, this method exhibits significant advantages in multiple dimensions: its flexibility lies in its ability to dynamically adjust the retrieval strategy and the number of cases based on the characteristics of the input content; its consistency benefits from case-driven analogical reasoning, reducing the uncertainty of the model itself; its high accuracy stems from the precise selection of discriminative cases by the information gain mechanism; and finally, by significantly improving the accuracy of automated review, it effectively reduces the reliance on high-cost, high-intensity manual review, achieving efficient, scalable, and highly robust content security assurance, and providing a crucial security infrastructure for the large-scale commercial application of generative AI.

[0045] In another embodiment, the present invention provides a computer-readable storage medium storing a computer program that causes a computer to execute the retrieval-enhanced large model generation content moderation method of the foregoing embodiments.

[0046] In another embodiment, the present invention proposes an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the content moderation method for generating large models based on retrieval enhancement as described in the foregoing embodiments.

[0047] In the embodiments disclosed in this application, a computer storage medium may be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CDROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0048] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed in this application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0049] The above are merely preferred embodiments of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should be considered within the scope of protection of the present invention.

Claims

1. A method for content moderation based on a large model generated by retrieval enhancement, characterized in that, Includes the following steps: Step 1: Convert the original text generated by the large language model to be detected into a high-dimensional semantic vector; Step 2: Use a classification model to perform a safe classification of the original text and output the safe classification results; Step 3: Select candidate cases from the pre-built labeled case knowledge base based on the security classification results to form an initial screening set; Step 4: Calculate the semantic similarity between candidate cases and the original text, and rank the candidate cases according to the semantic similarity. Further filter the ranked candidate cases and dynamically retrieve the case set for reasoning by calculating the information gain of the candidate cases relative to the selected cases. Step 5: Perform deep semantic re-ranking on the case set retrieved in Step 4, and select the top-ranked cases from the re-ranked list as the selected case set; Step 6: Based on the original text, combined with the selected case set from Step 5, incorporate reasoning step prompts to construct enhanced reasoning prompts; Step 7: Input the reasoning enhancement hints constructed in Step 6 into the large language model and output a quantified safety score; Step 8: Determine the security level of the corresponding original text by comparing the security score with a preset threshold.

2. The content review method based on retrieval enhancement for large model generation as described in claim 1, characterized in that: In step 1, the Embedding model is used to convert the original text generated by the detected large language model into a high-dimensional semantic vector.

3. The content review method based on retrieval enhancement for large model generation as described in claim 1, characterized in that: In step 2, the classification model used is the Qwen2.5-7B model.

4. The content review method based on retrieval enhancement for large model generation as described in claim 1, characterized in that: In step 3, the candidate cases include positive cases and negative cases. Negative cases refer to cases in which the content fully complies with the security standards in the corresponding category, while positive cases refer to cases in which the content is judged to have security risks or violations in the same category.

5. The content review method based on retrieval enhancement for large model generation as described in claim 1, characterized in that: Step 4 includes the following sub-steps: Step 4.1: Calculate the macro-semantic similarity and micro-semantic similarity between the high-dimensional semantic vector and the vectors in the initial screening set. Combine the macro-semantic similarity and micro-semantic similarity to calculate the semantic similarity between the candidate cases and the original text, and sort the candidate cases accordingly to recall a list of candidate cases sorted in descending order of semantic similarity. Here, macro-semantic similarity is used to calculate the semantic similarity of documents or paragraphs, and micro-semantic similarity is used to calculate the semantic similarity of sentences or phrases. Step 4.2: Initialize the final case set S_final, starting with the candidate case with the highest semantic similarity, and iteratively process each case candidate_i in the candidate case list in order; Step 4.3: Start the retrieval process. Calculate the information gain of each case candidate_i relative to all cases currently selected into S_final. If its semantic similarity is higher than the preset minimum threshold and its information gain is higher than the preset dynamic threshold, then candidate_i will be selected into S_final. Step 4.4: When the candidate case list has been traversed or the information gain of multiple consecutive cases is lower than the dynamic threshold, the retrieval process ends. The number of cases in S_final is used as the dynamically determined K value, representing the size of the case set used for reasoning.

6. The content review method based on retrieval enhancement for large model generation as described in claim 5, characterized in that: In step 5, deep semantic re-ranking is performed on the cases in S_final after the retrieval process ends, and the top-ranked cases are selected as the selected case set from the re-ranked list.

7. The content review method based on retrieval enhancement for large model generation as described in claim 1, characterized in that: In step 6, CoT templates are used to construct reasoning enhancement hints.

8. The content review method based on retrieval enhancement for large model generation as described in claim 1, characterized in that: In step 8, the security score is compared with a preset threshold, and the security score is divided into a high-score security range, a medium-score review range, and a low-score risk range from high to low. The original text corresponding to the high-score security range is determined to be safe and automatically approved by the system. The original text corresponding to the medium-score review range is determined to have potential risks, marked, and routed to the manual review queue. The original text corresponding to the low-score risk range is determined to be highly likely to be unsafe and is directly blocked or marked as high-risk.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program causes the computer to perform the content moderation method based on retrieval enhancement for generating large models as described in any one of claims 1-8.

10. An electronic device, characterized in that, include: The memory, the processor, and the computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the content moderation method for generating large models based on retrieval enhancement as described in any one of claims 1-8.

Images