Large model target range construction method and system based on expert model and risk level-to-level management

Abstract

The invention relates to a large model target range construction method and system based on an expert model and risk level-to-level management, and the method comprises the steps: collecting cue words submitted by a user, and carrying out the parallel processing through employing a plurality of generative AI models, so as to generate a plurality of different versions of contents corresponding to the same cue word; screening the generated content by using a risk identification expert model, and annotating and evaluating the identified harmful content by using a content understanding expert model and a risk evaluation expert model; carrying out primary review and feedback correction on the annotation result; and performing expert rechecking on the primary rechecking result, and correspondingly updating the knowledge base of the expert model. Compared with the prior art, the method has the advantages that the expert model is adopted to automatically annotate and grade risk contents, a multi-layer rechecking process is combined, the annotation efficiency and accuracy of content generation are remarkably improved, risk grading management is introduced, resources can be dynamically allocated, key risk problems can be intensively solved, and the self-adaptive coping capacity for implicit countermeasure attacks is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence technology, and in particular to a method and system for constructing a large model range based on expert models and risk classification management. Background Art

[0002] With the rapid development of generative artificial intelligence (AI), large-scale language models (LLM) and text-to-image (T2I) models have shown great potential in many fields. These models can not only generate logical natural language, but also generate rich image and text content through cross-modality. In the fields of automated text summarization, dialogue generation, sentiment analysis, virtual customer service, intelligent creation, etc., generative AI has greatly improved production efficiency and user experience. However, with the gradual deepening of technology applications, these models have gradually exposed potential ethical and security risks due to the limitations of their training data and architecture.

[0003] Most of the training data for large models comes from the Internet. Although the data is huge in quantity and rich in types, it also contains a large amount of unscreened biased information, false content and text. During the training process, this negative information is inevitably learned and subtly internalized by the model, resulting in the generated results containing biased content, or even generating false information under specific adversarial inputs. The generation of these inappropriate content may not only cause user dissatisfaction, but also lead to ethical disputes or legal disputes, especially in the sensitive field of public information or children's education, which is a severe test for the security of AI systems. Therefore, ensuring the security and fairness of generated content and preventing adversarial attacks have become urgent issues that need to be addressed.

[0004] At the same time, text-to-image (T2I) generative models have also shown amazing capabilities in generating realistic and creative images. However, because model training relies on Internet data, T2I models may generate images containing negative stereotypes, posing security risks. Therefore, ensuring the security of T2I models also requires effective identification and mitigation of these potential problems.

[0005] At present, in order to prevent and respond to the implicit risks in AI-generated content, researchers and development teams usually test and optimize through various methods to improve the robustness and security of models in various scenarios. Among them, adversarial attack testing has gradually become a key means, that is, by designing adversarial inputs to trigger the model to generate potentially harmful content, so as to detect its vulnerabilities under the "worst" conditions of the model. For example, if the input contains prompt words with puns, metaphors, or ambiguous language, the model may generate content that is contrary to mainstream values, exposing its vulnerability to implicit meanings. In order to further reveal the performance of the model in dealing with different types of attacks and complex inputs, it is particularly important to build an adversarial target range platform that can simulate a variety of attack scenarios.

[0006] The Open Adversarial Playground is a system designed to test the robustness of generative models under different types of attacks. Unlike traditional internal testing, this open playground attracts users from different backgrounds to submit prompt words that may trigger adversarial generation through crowdsourcing, and the system annotates and assesses the risks of the generated content. In this way, the open playground can not only capture long-tail problems that are not discovered in conventional tests, but also explore the potential vulnerabilities of generative models from multiple angles and cultural perspectives. This open mechanism greatly expands the diversity of data by introducing a wider user group, which helps to discover more hidden attack methods and more representative potential risks.

[0007] In order to locate these implicit adversarial cues that lead to the generation of potentially risky content, the Open AI Generated Content Security Range uses an open platform that allows users from different backgrounds to submit adversarial cues and annotate the generated images. This approach not only captures long-tail security issues that the model fails to discover in normal testing, but also discovers potential implicit adversarial attack patterns through a diverse crowd perspective. Through the red team testing method, it significantly improves the ability to identify implicit adversarial attacks in text-to-image generation models, especially when the model faces ambiguous inputs or implicit meanings. Compared with existing internal red team testing methods, this open range can cover potential security vulnerabilities more comprehensively, and through an open crowdsourcing mechanism, it greatly improves the diversity and coverage of data.

[0008] However, existing open adversarial ranges usually rely on manual labeling of each item and attribute, which is inefficient and has high labor costs. Especially in large-scale testing, the demand for manual labeling is extremely high, resulting in unbearable labor costs. In addition, this manual labeling mode is easily affected by subjective bias, and the accuracy and consistency of the labeling results are difficult to guarantee. In addition, existing open ranges generally lack a hierarchical management mechanism after discovering content security issues, making it difficult to achieve refined management. Even if content security issues are discovered, due to the lack of a complete hierarchical management mechanism, it is impossible to achieve refined management of issues of different types and risk levels. This has prevented some security issues with major impacts from being dealt with in a timely manner, which has limited the range's ability to respond to complex situations. Summary of the invention

[0009] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a large model target range construction method and system based on expert model and risk classification management, which can automatically identify potential risks in the generated content, reduce the dependence on manual labeling, and reasonably allocate resources according to the severity of different risks.

[0010] The purpose of the present invention can be achieved by the following technical solution: A method for constructing a large model range based on an expert model and risk classification management, comprising the following steps:

[0011] S1. Collect prompt words submitted by users and use multiple generative AI models to process them in parallel to generate multiple different versions of content corresponding to the same prompt word;

[0012] S2. Use the risk identification expert model to screen the generated content, and use the content understanding expert model and risk assessment expert model to annotate and evaluate the identified harmful content;

[0013] S3. Conduct initial review and feedback correction on the annotation results;

[0014] S4. Conduct expert review on the initial review results and update the knowledge base of the expert model accordingly.

[0015] Furthermore, the step S1 specifically collects adversarial prompt words submitted by users with different backgrounds through an open platform.

[0016] Furthermore, the prompt word in step S1 is specifically a natural language description or a sentence with implicit meaning.

[0017] Furthermore, step S2 includes the following process:

[0018] S21. Automatically scan the generated content using the risk identification expert model, preliminarily mark the potentially harmful content, and classify the potentially harmful content into harmful types;

[0019] S22. Use content understanding expert models to annotate risk behaviors of harmful content, including potential harm and causes of problems;

[0020] S23. Use a risk assessment expert model to perform a graded assessment on the risk level of harmful content.

[0021] Furthermore, the step S3 specifically reviews the annotation results item by item according to a preset standardized process, and provides feedback and corrections for erroneous or incomplete content.

[0022] Furthermore, the step S4 specifically involves expert review according to a preset time period.

[0023] Furthermore, the step S4 comprises the following steps:

[0024] S41, collecting the prompt words and corresponding generated content after user review to obtain a complete annotation data set;

[0025] S42. Use big data analysis technology to mine implicit adversarial patterns of generated content in the annotation dataset and determine the corresponding risk characteristics of different prompt words;

[0026] S43. Incorporate the information of newly identified risk types and adversarial patterns into the knowledge base of the expert model for training and updating the risk identification expert model, content understanding expert model and risk assessment expert model.

[0027] A large model range construction system based on expert model and risk classification management, including an open data collection module, a content generation module, an automatic annotation module, a risk classification management module, an initial review module, a visual analysis module, and an expert review and update module. The open data collection module is used to receive prompt words submitted by users with different backgrounds;

[0028] The content generation module is used to generate multiple different versions of content corresponding to the same prompt word;

[0029] The automatic annotation module is pre-installed with a risk identification expert model and a content understanding expert model, which are used to annotate the generated content, identify potential risk content, and classify the hazard type;

[0030] The risk classification management module is pre-installed with a risk assessment expert model for assessing the risk level of the annotation content and performing classification management according to the assessment results;

[0031] The initial review module is used to confirm, provide feedback and correct the annotation content;

[0032] The visualization analysis module is used to organize and summarize the initial review result data, generate visualization charts, analyze the data characteristics of implicit confrontation prompt words, and generate data analysis reports;

[0033] The expert review and update module is used to regularly conduct expert review of the initial review results to identify new risk types, and add information on newly identified risk types and adversarial patterns to the knowledge base of the expert model to conduct online learning and updating of the risk identification expert model, content understanding expert model and risk assessment expert model.

[0034] Furthermore, a plurality of different generative AI models are provided in the content generation module.

[0035] Furthermore, the visualization charts generated by the visualization analysis module include risk distribution charts, time series trend charts and risk level distribution charts.

[0036] Compared with the prior art, the present invention has the following advantages:

[0037] The present invention collects prompt words submitted by users and uses multiple generative AI models for parallel processing to generate multiple different versions of content corresponding to the same prompt word; then uses the risk identification expert model to screen the generated content in turn, uses the content understanding expert model and the risk assessment expert model to annotate and evaluate the identified harmful content; and performs an initial review and feedback correction on the annotation results, and an expert review on the initial review results. Therefore, with the assistance of expert models and hierarchical management, through the automated annotation of generated content, hierarchical risk assessment, and multi-layer review of open user feedback, the comprehensiveness of adversarial attack detection, the accuracy of annotation results, and the rationality of resource allocation are significantly improved.

[0038] The present invention collects adversarial prompt words submitted by users with different backgrounds through an open platform, and combines multiple generative AI models to generate multiple different versions of content corresponding to the same prompt word in parallel, which can not only ensure the diversity and complexity of the input prompt words, but also increase the comprehensiveness and diversity of the generated content.

[0039] The present invention uses a risk identification expert model to screen the generated content and identify potentially harmful content and its specific types of harm; then, a content understanding expert model is used to analyze the risk behavior of the identified harmful content and provide detailed explanations; a risk assessment expert model is also used to assess the risk level and its scope of impact based on the interpretation results, and to identify potentially affected community groups. Thus, by introducing an expert model, potential risks in generated content can be automatically identified, reducing the reliance on manual annotation, freeing human resources from the heavy annotation tasks, not only significantly reducing labor costs, but also enabling in-depth understanding and risk assessment of content through expert models, thereby improving the accuracy and consistency of annotations.

[0040] The present invention is designed to conduct an initial review of the annotation results to optimize the annotation quality, and to conduct expert review of the user review results on a regular basis. The knowledge base of the expert model is updated according to the analysis results, and the model's ability to cope with new implicit adversarial attacks is continuously improved, ensuring that the expert model can adapt to emerging risk types and implicit adversarial modes in a timely manner, thereby having adaptive response capabilities in the field of generative AI security protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0042] Figure 2 Schematic diagram of application architecture of an embodiment. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.

[0044] Example

[0045] like Figure 1 As shown, a method for constructing a large model range based on an expert model and risk classification management includes the following steps:

[0046] S1. Collect prompt words submitted by users and use multiple generative AI models to process them in parallel to generate multiple different versions of content corresponding to the same prompt word;

[0047] S2. Use the risk identification expert model to screen the generated content, and use the content understanding expert model and risk assessment expert model to annotate and evaluate the identified harmful content;

[0048] S3. Conduct initial review and feedback correction on the annotation results;

[0049] S4. Conduct expert review on the initial review results and update the knowledge base of the expert model accordingly.

[0050] The above method is adopted to realize a large model range construction system based on expert model and risk classification management, including an open data collection module, a content generation module, an automatic annotation module, a risk classification management module, an initial review module, a visual analysis module, an expert review and update module, wherein the open data collection module is used to receive prompt words submitted by users with different backgrounds;

[0051] The content generation module is provided with a plurality of different generative AI models for generating a plurality of different versions of content corresponding to the same prompt word;

[0052] The automatic annotation module is pre-installed with risk identification expert models and content understanding expert models, which are used to annotate the generated content, identify potential risk content, and classify the hazard types;

[0053] The risk classification management module is pre-installed with a risk assessment expert model, which is used to assess the risk level of annotation content and perform classification management based on the assessment results;

[0054] The initial review module is used to confirm, provide feedback and correct the annotation content;

[0055] The visualization analysis module is used to organize and summarize the initial review result data, generate visualization charts (including risk distribution charts, time series trend charts and risk level distribution charts), analyze the data characteristics of implicit confrontation prompt words, and generate data analysis reports;

[0056] The expert review and update module is used to regularly conduct expert reviews of the initial review results to identify new risk types, and add information on newly identified risk types and adversarial patterns to the knowledge base of the expert model to conduct online learning and updating of the risk identification expert model, content understanding expert model and risk assessment expert model.

[0057] Based on the above solution, it is possible to achieve:

[0058] Efficient content generation and security management: Through automated annotation by expert models, the cost and time of manual annotation are significantly reduced, and the annotation efficiency is increased to several times that of traditional methods.

[0059] Improved annotation accuracy through human-computer collaboration: Through a three-level interactive mechanism of automated annotation, user feedback, and expert review, the accuracy and comprehensiveness of content annotations are effectively improved, ensuring the reliability of the system in dealing with diverse input scenarios.

[0060] Risk classification management for efficient resource allocation: Through the risk classification management mechanism, the present invention achieves optimal resource allocation. High-risk content can get more resource support to ensure that key issues are responded to and resolved quickly.

[0061] Open crowdsourcing data collection and comprehensive coverage of adversarial attacks: We collect adversarial prompts submitted by users from different backgrounds through an open platform, significantly improving the diversity and coverage of the data set. Together with automated annotation and risk management systems, we can effectively identify implicit adversarial attacks brought about by diverse inputs.

[0062] Continuously evolving dynamic update mechanism: Through expert review and knowledge base updates, it ensures that the expert model can adapt to emerging risk types and implicit adversarial patterns in a timely manner, enabling the system to have adaptive response capabilities in the field of generative AI security protection.

[0063] This embodiment applies the above solution to build Figure 2 The application architecture shown in the figure integrates expert models, risk classification management mechanisms, and open adversarial prompt word input to build a comprehensive platform that can fully support generative model security testing. The overall architecture of the platform mainly includes:

[0064] Prompt word input and content generation module: used to receive prompt words submitted by users and generate corresponding content by combining multiple generative AI models. This module supports natural language prompt word input and generates diversified content through multiple models to improve the comprehensiveness and diversity of the test.

[0065] Automatic annotation module for content generation: Use expert models to automatically annotate the generated content, identify potential risk content, and classify its hazard types, providing basic data for subsequent risk grading assessments.

[0066] Risk classification management module: The risk level of the content annotated by the expert model is evaluated and classified management is carried out according to the evaluation results. High-risk content will receive more resources for in-depth analysis and optimization, while low-risk content will be simplified.

[0067] Initial review module: The platform feeds back the automated annotation results to the user interface, guiding users to review the annotation results. Users can confirm, correct, and make suggestions for the automatically generated annotations, thereby optimizing the annotation quality.

[0068] Data visualization and analysis module: Provides data visualization tools to support statistical analysis of prompt words and generated content, providing support for further mining and identification of risk patterns, and facilitating researchers and developers to deeply understand the potential security risks of generative models.

[0069] Expert review and dynamic update module: Regularly conduct expert review on user review results, and update the knowledge base of the expert model based on the analysis results, continuously improving the model's ability to cope with new implicit adversarial attacks.

[0070] The specific application process includes:

[0071] Step 1: Receive prompt words input by users on the platform. These prompt words can be natural language descriptions or sentences with implicit meanings, and use multiple content generation models to generate AI content. This step ensures that AI content covering as many scenarios as possible can be generated through a wide range of model selections, thereby enhancing the system's testing and recognition capabilities.

[0072] Step 2: Automatically scan the generated AI content through the risk identification expert model to preliminarily mark the potentially harmful content and its specific types of harm; this step uses advanced machine learning algorithms and natural language processing technology to screen and preliminarily classify content in an automated and efficient manner.

[0073] Step 3: Use the content understanding expert model to conduct an in-depth analysis of the harmful content identified in step 2 and provide detailed explanations of the potential harm and causes of the problems. This model combines the latest semantic analysis technology to ensure an accurate understanding of the deep-seated causes and impacts of harmful content.

[0074] Step 4: Using the risk assessment expert model, based on the detailed analysis results of step 3, evaluate the risk level of various harmful contents, identify and clarify the scope of the community that may be affected; this step will categorize the risks (for example, low, medium, and high risks) and take appropriate resource allocation and management measures according to the level.

[0075] Step 5: Present the risk hazard interpretation and assessment results of steps 3 and 4 on the front end of the platform for user review, and adjust and correct the automatic annotation results based on user feedback to ensure the accuracy and completeness of the annotations; promote user participation and real-time optimization of the system through an interactive user interface.

[0076] Step 6: Use data visualization tools to organize and analyze the data reviewed by users, generate risk content trend charts and various types of hazard distribution charts and other analytical charts, and assist researchers and developers to gain in-depth understanding and mining of data features of implicit adversarial prompt words; this step supports continuous system review and upgrades through data science technology.

[0077] Step 7: Regularly conduct expert review of the user review results, use the data analysis tools in step 6 to focus on new risk types, and update the expert model's knowledge base based on the latest data analysis results to improve the model's ability to identify and respond to new implicit adversarial attacks; ensure that the system can adapt to new security threats and challenges.

[0078] In step 1, content generation model selection includes automatically selecting the model that is most suitable for processing the prompt word input by the user, and calling multiple models at the same time to generate different versions of content for the same prompt word to increase the diversity of content and cover more potential risks, including the following sub-steps:

[0079] Step 1.1: Prompt word collection: The user submits a prompt word on the platform's input interface, and the prompt word can be a natural language description or a sentence with implicit meaning.

[0080] Step 1.2: Content generation model selection: The system automatically selects the most suitable content generation model (e.g., the optimal model based on the current task) to process the prompt word input by the user.

[0081] Step 1.3: Multi-model parallel generation: To improve the diversity of generated content, the platform can call multiple generation models at the same time to generate different versions of content for the same prompt word.

[0082] Step 1.4: Random content display: In order to prevent users from being affected by the generation order or other external factors during the annotation process, the platform will randomly sort the generated content and present it.

[0083] In step 2, the following sub-steps are included:

[0084] Step 2.1: Preliminary identification of harmful content: Use the risk identification expert model to automatically scan the generated AI content and preliminarily mark out potential harmful content.

[0085] Step 2.2: Classification of harmful content: Classify the marked harmful content into predefined harmful content types according to their types.

[0086] Among them, the initial identification of harmful content is not limited to marking common types of hazards, but also includes dynamically identifying new risk types based on user and expert feedback to adapt to and respond to ever-changing security challenges.

[0087] In step 5, the following sub-steps are included:

[0088] Step 5.1: Annotation result presentation: The annotation results generated by the model are fed back to the platform front end, allowing users to intuitively view the marked harmful content and its annotation information.

[0089] Step 5.2: User review process: Guide users to review the accuracy of automatic annotation results item by item according to the standardized process to ensure the correctness of content annotation.

[0090] Step 5.3: User feedback collection: During the review process, users can submit feedback on inaccurate or incomplete annotations and indicate where improvements or adjustments are needed.

[0091] Step 5.4: Annotation adjustment and optimization: Based on the feedback provided by users, the platform will make local adjustments to the annotation results, correct incorrect annotations, and improve the annotation accuracy.

[0092] Users not only provide feedback on the accuracy of annotation results, but also collect their opinions on the completeness of annotations. The platform adjusts annotations based on these opinions and optimizes the annotation strategy and performance of the system. Through this mechanism, the platform maintains a high degree of adaptability and responsiveness, improving user satisfaction and the overall effectiveness of the system.

[0093] In step 6, the following sub-steps are included:

[0094] Step 6.1: Data collation and aggregation: Collect and organize the prompt words and generated content data after user review to form an orderly data set.

[0095] Step 6.2: Generate visual charts: Use data visualization tools to generate relevant analysis charts, such as risk content trend charts, distribution charts of various hazard types, etc.

[0096] Among them, data visualization tools can generate various types of charts, including risk distribution charts, time series trend charts and risk level distribution charts, and can further identify clustering patterns of different risk levels and hazard types, so that researchers can conduct in-depth mining and analysis of implicit adversarial patterns in the generated content and form a systematic understanding of model vulnerabilities.

[0097] Step 6.3: Implicit adversarial pattern analysis: In-depth analysis of the data features of implicit adversarial prompt words, explore their potential attack patterns and rules, and locate the vulnerabilities of the model in different scenarios.

[0098] Step 6.4: Research report generation: Generate a report based on the data analysis results to provide researchers and developers with detailed analysis conclusions and improvement suggestions.

[0099] In step 7, the following sub-steps are included:

[0100] Step 7.1: Expert review of user review data: Submit user review results to domain experts regularly to ensure the high quality and consistency of annotation results.

[0101] Step 7.2: Identification of new risk types: Experts use data visualization tools and analytical models to conduct in-depth reviews and focus on unknown risk types that have emerged recently.

[0102] Step 7.3: Knowledge base update: Add information about newly identified risk types and adversarial patterns to the knowledge base of the expert model to ensure that the model can adapt to and respond to emerging attack methods.

[0103] Step 7.4: Model performance optimization: Based on expert feedback and the latest updates to the knowledge base, the risk identification, content understanding, and risk assessment expert models are learned and updated online to improve the overall detection accuracy and response capabilities of the system.

[0104] The platform's knowledge base update mechanism automatically adds new risk types, adversarial prompt words and their related features to the knowledge base after receiving the expert review results, and retrains the risk identification, content understanding and risk assessment expert models through online learning technology to ensure that the system always has effective identification and response capabilities when facing emerging security threats. The optimization and retraining of risk identification, content understanding and risk assessment expert models adopt an adaptive update mechanism. When the model performance is lower than the threshold or a new adversarial attack method appears, the system will retrain the model based on the latest update of the knowledge base to ensure that the platform maintains a high degree of accuracy and robustness in a changing security environment.

[0105] In summary, this solution provides an open range for collecting and testing adversarial cues. The range allows users from different backgrounds to submit potential adversarial cues and systematically annotate and analyze the generated content. This solution also proposes an auxiliary annotation method based on an expert model. The expert model can automatically identify security risks in generated content and further explain the risks by deeply understanding the content, thereby evaluating its risk level and scope of impact. In addition, a risk level management mechanism is proposed to dynamically and adaptively adjust the test focus of the range according to the level and scope of impact of different risks, aiming to concentrate resources and solve more critical pain points.

[0106] Compared with the existing technical solutions, this solution has the following advantages:

[0107] 1. Data diversity collection and full coverage of risks: Through an open platform, users from different backgrounds are attracted to participate in submitting adversarial prompt words, effectively enriching the diversity of the data set. With the help of systematic automated annotation and risk assessment processes, the present invention can efficiently identify and analyze harmful content generated under ambiguous input conditions, and accurately interpret potential adversarial behaviors, thereby significantly improving the model's ability to cope with implicit adversarial attacks.

[0108] 2. The efficiency and accuracy of annotation are significantly improved under human-computer collaboration: In the content annotation process, the present invention combines the automated annotation mechanism with the hierarchical review method of users and experts, which not only significantly improves the automation level and accuracy of the annotation results, but also continuously optimizes the risk monitoring model by dynamically updating the model knowledge base.

[0109] 3. Efficient allocation of resources under risk classification management: The risk classification management mechanism introduced in the present invention enables the platform to efficiently allocate resources according to different risk levels, thereby maximizing the risk response capability and overall operating efficiency of the open shooting range.

Claims

1. A method for constructing a large model range based on an expert model and risk classification management, characterized in that: The following steps are involved: S1. Collect prompt words submitted by users and use multiple generative AI models to process them in parallel to generate multiple different versions of content corresponding to the same prompt word; S2. Use the risk identification expert model to screen the generated content, and use the content understanding expert model and risk assessment expert model to annotate and evaluate the identified harmful content; S3. Conduct initial review and feedback correction on the annotation results; S4. Conduct expert review on the initial review results and update the knowledge base of the expert model accordingly.

2. A method for constructing a large model range based on an expert model and risk classification management according to claim 1, characterized in that: The step S1 specifically collects adversarial prompt words submitted by users with different backgrounds through an open platform.

3. A method for constructing a large model range based on an expert model and risk classification management according to any one of claims 1 to 2, characterized in that: The prompt word in step S1 is specifically a natural language description or a sentence with implicit meaning.

4. The method for constructing a large model range based on an expert model and risk classification management according to claim 1 is characterized in that: Step S2 The process includes: S21. Automatically scan the generated content using the risk identification expert model, preliminarily mark the potentially harmful content, and classify the potentially harmful content into harmful types; S22. Use content understanding expert models to annotate risk behaviors of harmful content, including potential harm and causes of problems; S23. Use a risk assessment expert model to perform a graded assessment on the risk level of harmful content.

5. The method for constructing a large model range based on an expert model and risk classification management according to claim 1, characterized in that: The step S3 specifically reviews the annotation results item by item according to a preset standardized process, and provides feedback and corrections for erroneous or incomplete content.

6. The method for constructing a large model range based on an expert model and risk classification management according to claim 1, characterized in that: The step S4 specifically involves expert review according to a preset time period.

7. The method for constructing a large model range based on an expert model and risk classification management according to claim 1, characterized in that: The step S4 comprises the following steps: S41, collecting the prompt words and corresponding generated content after user review to obtain a complete annotation data set; S42. Use big data analysis technology to mine implicit adversarial patterns of generated content in the annotation dataset and determine the corresponding risk characteristics of different prompt words; S43. Incorporate the information of newly identified risk types and adversarial patterns into the knowledge base of the expert model for training and updating the risk identification expert model, content understanding expert model and risk assessment expert model.

8. A large model range construction system based on expert model and risk classification management, using a large model range construction method based on expert model and risk classification management as claimed in claim 1, characterized in that: It includes an open data collection module, a content generation module, an automatic annotation module, a risk classification management module, an initial review module, a visual analysis module, and an expert review and update module. The open data collection module is used to receive prompt words submitted by users with different backgrounds; The content generation module is used to generate multiple different versions of content corresponding to the same prompt word; The automatic annotation module is pre-installed with a risk identification expert model and a content understanding expert model, which are used to annotate the generated content, identify potential risk content, and classify the hazard type; The risk classification management module is pre-installed with a risk assessment expert model for assessing the risk level of the annotation content and performing classification management according to the assessment results; The initial review module is used to confirm, provide feedback and correct the annotation content; The visualization analysis module is used to organize and summarize the initial review result data, generate visualization charts, analyze the data characteristics of implicit confrontation prompt words, and generate data analysis reports; The expert review and update module is used to regularly conduct expert review of the initial review results to identify new risk types, and add information on newly identified risk types and adversarial patterns to the knowledge base of the expert model to conduct online learning and updating of the risk identification expert model, content understanding expert model and risk assessment expert model.

9. A large-scale model range construction system based on expert model and risk classification management according to claim 8, characterized in that: A plurality of different generative AI models are arranged in the content generation module.

10. A large-scale model range construction system based on expert model and risk classification management according to claim 8, characterized in that: The visualization charts generated by the visualization analysis module include risk distribution charts, time series trend charts and risk level distribution charts.

Images