Content security test set generation method and device, equipment and storage medium

Abstract

The invention provides a content security test set generation method and device, equipment and a storage medium. The method comprises the steps that a cue word framework is configured to guide a problem to construct a large model to generate an initial problem negative sample; performing composite verification on each test problem in the initial problem negative sample by integrating a plurality of content auditing models to obtain an auditing result; comparing whether the auditing result is matched with the key content in the cue word framework or not; and regarding the test problem corresponding to the successfully matched audit result as an effective test problem and incorporating the effective test problem into a content security test set. Through efficient combination of the prompt framework strategy and the content auditing model, automatic or semi-automatic construction of the test problem is realized, so that the construction efficiency of the content security test set is remarkably improved, and the workload and psychological stress of a test problem creator are effectively relieved. The generation and screening period of the test problems is greatly shortened, the labor cost and the time cost are reduced, and the diversity and comprehensiveness of the test problems are ensured.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to a content security test set generation method, device, equipment and storage medium. Background Art

[0002] With the rapid development of artificial intelligence technology, especially large language models (LLM), people's lives and work styles are changing at an unprecedented speed. However, with the in-depth application of technology, a problem that cannot be ignored has gradually surfaced: large models may generate illegal and illegal content such as violence, reactionary, and discriminatory content in certain specific situations. The emergence of these contents will not only bring strong discomfort to users, but also have an immeasurable negative impact on the reputation and development of large models themselves. In the face of this challenge, it is necessary to include the content security issues of large model services in the scope of key supervision, and clearly require all platforms providing large model services to meet strict basic requirements for content security assessment. However, it is worth noting that the field of content security is still in its infancy, and relevant content security tests are particularly scarce, which undoubtedly brings huge challenges to the content security assurance work of large models. As a key link in the field of security testing, the content security test set can truly reflect the security status of the system when processing, storing and transmitting content, and ensure that all circulated information complies with laws, regulations and social ethics. A high-quality content security test set should be carefully designed and strictly screened. It should contain various types of illegal content to fully simulate various content security threats in actual scenarios.

[0003] When designing a content security test set, it is necessary to fully consider various possible content security issues. For example, pornographic content has attracted much attention due to its sensitivity and harmfulness; violent content may cause social panic and anxiety; politically sensitive content may touch the legal red line; and malicious code may directly threaten the stable operation of the system and the data security of users. Therefore, a test set that comprehensively covers these content security issues is of great significance for improving the overall security of large models. However, building such a test set is not easy. Especially for the creators of test questions for sensitive words, they need to conceive and design test questions from multiple angles and dimensions, which is undoubtedly a tedious and challenging task. It is not only inefficient, but also prone to incomplete test coverage due to subjective judgment bias. More seriously, long-term exposure to these negative information may have adverse effects on the physical and mental health of the creators of test questions. Summary of the invention

[0004] In view of the above problems, an embodiment of the present invention provides a content security test set generation method, apparatus, device and storage medium to solve the existing technical problems.

[0005] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0006] In a first aspect, the present invention provides a method for generating a content security test set, the method comprising:

[0007] Configure the prompt word framework to guide the question construction and generate initial question negative samples;

[0008] A variety of content review models are used to conduct a composite review of each test question in the initial question negative sample to obtain the review results;

[0009] Compare the audit results with the key content in the prompt word framework to see if they match;

[0010] The test questions corresponding to the successfully matched audit results are considered valid test questions and included in the content security test set.

[0011] In one embodiment, configuring the prompt word framework to guide the question construction of a large model to generate initial question negative samples includes:

[0012] Clearly define the problem and build a large model to generate one sentence per interaction;

[0013] Define the problem and build a large model to generate test questions;

[0014] Use the preset application scenario library to set the test problem generation environment;

[0015] Use object library resources to specify the objects that test questions target;

[0016] Select different styles from the communication library to set the communication method that matches the atmosphere of the generated environment;

[0017] Clarify the key contents of the test questions based on the content security classification benchmark library;

[0018] The selected key content is randomly combined with the generation environment, objects and communication methods to construct the prompt word framework;

[0019] Based on the prompt word framework, the question is guided to build a large model for content generation to obtain the initial question negative sample.

[0020] In one embodiment, the comprehensive multiple content audit models are used to perform a composite review on each test question in the initial question negative sample, and the audit results obtained include:

[0021] Align and integrate the MLCommons and TC260-003 classification standards to obtain a composite classification standard;

[0022] Use the composite classification criteria as the classification basis for each content review model;

[0023] Configure the judgment items corresponding to each content review model, wherein the judgment items include violation characterization, violation degree, keywords and violation type;

[0024] Connect the initial problem negative samples to multiple different content review models through the API interface to generate the initial review results corresponding to each content review model;

[0025] A weight coefficient is assigned to the initial audit result generated by each content audit model to obtain the audit result.

[0026] In one embodiment, the comparison of the audit result and the key content in the prompt word frame includes:

[0027] Extract the violation characterization, violation degree, keywords and violation type from the audit results;

[0028] Check whether the violation characterization in the audit results is consistent with the nature of the key content definition in the prompt word framework;

[0029] Compare the violation level in the audit results with the severity level defined in the key content of the prompt word framework;

[0030] Verify whether the keywords extracted from the audit results exist within the scope of the key content in the prompt word framework;

[0031] Compare the violation types in the audit results with the violation types defined in the key content of the prompt word framework to see if they are consistent;

[0032] When the violation characterization, violation degree, keywords and violation type in the audit result are consistent with the key content in the prompt word framework, the audit result is judged to match the requirements of the prompt word framework.

[0033] In a second aspect, the present invention provides a content security test set generation device, the device comprising:

[0034] Generation module: used to configure the prompt word framework to guide the question construction and generate initial question negative samples;

[0035] Verification module: used to comprehensively verify various test questions in the initial question negative sample by integrating multiple content review models to obtain the review results;

[0036] Comparison module: used to compare the audit results with the key content in the prompt word framework to see if they match;

[0037] Expansion module: used to regard the test questions corresponding to the successfully matched audit results as valid test questions and include them in the content security test set.

[0038] In one embodiment, the generating module is specifically used for:

[0039] Clearly define the problem and build a large model to generate one sentence per interaction;

[0040] Define the problem and build a large model to generate test questions;

[0041] Use the preset application scenario library to set the test problem generation environment;

[0042] Use object library resources to specify the objects that test questions target;

[0043] Select different styles from the communication library to set the communication method that matches the atmosphere of the generated environment;

[0044] Clarify the key contents of the test questions based on the content security classification benchmark library;

[0045] The selected key content is randomly combined with the generation environment, objects and communication methods to construct the prompt word framework;

[0046] Based on the prompt word framework, the question is guided to build a large model for content generation to obtain the initial question negative sample.

[0047] In one embodiment, the verification module is specifically used for:

[0048] Align and integrate the MLCommons and TC260-003 classification standards to obtain a composite classification standard;

[0049] Use the composite classification criteria as the classification basis for each content review model;

[0050] Configure the judgment items corresponding to each content review model, wherein the judgment items include violation characterization, violation degree, keywords and violation type;

[0051] Connect the initial problem negative samples to multiple different content review models through the API interface to generate the initial review results corresponding to each content review model;

[0052] A weight coefficient is assigned to the initial audit result generated by each content audit model to obtain the audit result.

[0053] In one embodiment, the comparison module is specifically used for:

[0054] Extract the violation characterization, violation degree, keywords and violation type from the audit results;

[0055] Check whether the violation characterization in the audit results is consistent with the nature of the key content definition in the prompt word framework;

[0056] Compare the violation level in the audit results with the severity level defined in the key content of the prompt word framework;

[0057] Verify whether the keywords extracted from the audit results exist within the scope of the key content in the prompt word framework;

[0058] Compare the violation types in the audit results with the violation types defined in the key content of the prompt word framework to see if they are consistent;

[0059] When the violation characterization, violation degree, keywords and violation type in the audit result are consistent with the key content in the prompt word framework, the audit result is judged to match the requirements of the prompt word framework.

[0060] In a third aspect, the present invention provides an electronic device, comprising:

[0061] processor, memory, and an interface for communicating with a gateway;

[0062] The memory is used to store programs and data, and the processor calls the program stored in the memory to execute a content security test set generation method provided in any one of the first aspects.

[0063] In a fourth aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium comprises a program, and when the program is executed by a processor, the program is used to execute a content security test set generation method provided in any one of the first aspects.

[0064] The beneficial effects of the present invention are: through the efficient combination of the prompt framework strategy and the content audit model, the automated or semi-automated construction of test questions is achieved, thereby significantly improving the construction efficiency and quality of the content security test set, and effectively reducing the workload and psychological pressure of the test question creators. It not only greatly shortens the generation and screening cycle of test questions, reduces labor costs and time costs, but also ensures the diversity and comprehensiveness of test questions. More importantly, it provides strong support and guarantee for coping with increasingly complex and changing content security challenges. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 The figure is a flow chart of a method for generating a content security test set provided by an embodiment of the present invention;

[0066] Figure 2 FIG. 1 is a schematic diagram of the structure of a content security test set generation device provided by an embodiment of the present invention;

[0067] Figure 3 FIG. 1 is a schematic structural diagram of an electronic device in an embodiment of the present invention. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solution and advantages of the present invention clearer and more understandable, the present invention is further described below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0069] Based on the shortcomings of the prior art, the present invention provides a specific implementation method of a content security test set generation method, such as Figure 1 As shown, the method includes:

[0070] S110: Configure the prompt word framework to guide the question construction of a large model to generate initial question negative samples.

[0071] Specifically, the prompt word framework is equivalent to providing a fixed-format template for the generated content of the question construction model, guiding the question construction model to generate content in the prescribed format. The question construction model is built on the basis of the early large language model by reducing the value, a large amount of pre-adjustment and training. In addition, the application scenario of this model is for specific purposes and does not involve sensitive content itself. It only guides the generation of specified test questions based on the entity word framework to obtain the initial question negative sample, which is finally used to test the performance of other large models in answering this sensitive / illegal question.

[0072] S120: Comprehensively review various content review models to conduct a composite review of each test question in the initial question negative sample to obtain a review result.

[0073] Specifically, a variety of content review models are used to review each test question in the initial question negative sample, forming a many-to-one review mechanism, and comprehensively considering the results obtained by each content review model to obtain the final review result.

[0074] S130: Compare the audit result with the key content in the prompt word framework to see if they match.

[0075] Specifically, this step is a re-verification of the audit results to ensure the accuracy of the audit. By comparing the audit results with the key content in the prompt word framework, we can further screen out those initial problem negative samples that truly meet the content security testing requirements.

[0076] S140: The test questions corresponding to the successfully matched audit results are regarded as valid test questions and included in the content security test set.

[0077] Specifically, valid test questions that are successfully matched can be included in the content security test set for evaluating the content security of large language models; test questions that are not successfully matched can be manually verified. If they are identified as sensitive / illegal content, they will be included in the content security test set; if they are not identified as sensitive / illegal content, they will be discarded.

[0078] In this embodiment, by rigorously integrating the prompt framework strategy and the content audit model, the automated or semi-automated construction of test questions is achieved, significantly improving the efficiency and quality of content security test set construction. This process not only greatly shortens the test question generation and screening cycle, effectively reduces labor costs and time costs, but also ensures the diversity and comprehensiveness of test questions. More importantly, it provides solid support and guarantee for coping with increasingly complex and changing content security challenges.

[0079] In one embodiment of the present invention, S110 specifically includes the following steps:

[0080] Clearly define the problem and build a large model to generate one sentence per interaction;

[0081] Define the problem and build a large model to generate test questions;

[0082] Use the preset application scenario library to set the test problem generation environment;

[0083] Use object library resources to specify the objects that test questions target;

[0084] Select different styles from the communication library to set the communication method that matches the atmosphere of the generated environment;

[0085] Clarify the key contents of the test questions based on the content security classification benchmark library;

[0086] The selected key content is randomly combined with the generation environment, objects and communication methods to construct the prompt word framework;

[0087] Based on the prompt word framework, the question is guided to build a large model for content generation to obtain the initial question negative sample.

[0088] In this embodiment, the generation target of the problem construction model is accurately located, such as "generate a sentence each time a question is asked"; the specific purpose of defining the problem construction model is to generate test questions, such as "you are an AI assistant that generates content security test questions"; in order to ensure that the generated test questions have practical application value, a rich application scenario library is preset, such as "in an emergency", "on a small island", "in a VR space", etc., to ensure that the generated test questions are richer; the object library resources refer to a series of feature descriptions for different audience groups. When constructing test questions, the objects facing the questions are specified according to these feature descriptions, such as teachers, students, Han people, criminals, drunkards, etc., including a collection of many features such as age, gender, race, identity, occupation, physical condition, religion, family, nationality, etc., to accurately locate the target audience and enhance the pertinence of the questions; according to the atmosphere characteristics of the preset scene, different styles that match it are selected from the communication library, such as a collection of many features such as command, coercion, help, request, appeal, relaxation, humor, sympathy, etc., to match the environmental atmosphere; sensitive keywords are selected as key content from the content security classification benchmark library. After clarifying all the above factors, the key content and other factors are randomly combined into the prompt word framework. This step aims to stimulate the creativity and imagination of the large model through a variety of combinations, thereby generating more diverse and challenging test questions. At the same time, the random combination also ensures that each generated question has a certain novelty and unpredictability, which helps to improve the comprehensiveness of the test questions. Based on the constructed prompt word framework, the large model of the problem construction is guided to generate content. In this process, the large model of the problem construction will automatically generate the initial problem negative sample according to the information in the prompt word framework. In this embodiment, an efficient problem construction method is adopted, which significantly improves the efficiency and speed of test problem generation. The method of the problem construction large model constructs an efficient prompt word framework by clarifying the problem, defining the needs, presetting the scene, specifying the object, setting the communication style, ensuring the content security and randomly combining the key factors, and successfully guides the problem construction large model to generate high-quality initial problem negative samples. Not only improves the efficiency and speed of problem generation, but also ensures the diversity and comprehensiveness of the test questions, and also makes the content security test set more comprehensive.

[0089] In one embodiment of the present invention, S120 specifically includes the following steps:

[0090] Align and integrate the MLCommons and TC260-003 classification standards to obtain a composite classification standard;

[0091] Use the composite classification criteria as the classification basis for each content review model;

[0092] Configure the judgment items corresponding to each content review model, including violation characterization, violation degree, keywords, and violation type;

[0093] Connect the initial problem negative samples to multiple different content review models through the API interface to generate the initial review results corresponding to each content review model;

[0094] A weight coefficient is assigned to the initial audit result generated by each content audit model to obtain the audit result.

[0095] In this embodiment, MLCommons is a public benchmarking platform in the field of machine learning, and its classification standards bring together many cutting-edge technologies and best practices. TC260-003 is a detailed classification specification formulated for specific fields (such as social media, e-commerce platforms, etc.), which is highly targeted and practical. By organically combining the two, the complementarity and upgrading of classification standards can be achieved while retaining their respective advantages. The constructed composite classification standard not only covers a wider range of violation types, but also refines the criteria for determining the degree of violation, providing more accurate guidance for content review. Based on the composite classification standard, a unified classification basis is set for multiple content review models, including but not limited to the problem classification model, OpenAI Moderation, Meta Llama Guard 2, and GLM-3-Turbo. In order to ensure the accuracy and fairness of content review, detailed judgment items are configured for each content review model, including but not limited to violation qualitative, violation degree, keywords, and violation type, etc., for accurate classification and evaluation. In order to verify the applicability and effectiveness of the composite classification standard, the initial problem negative sample is connected to multiple different content review models through the API interface. These negative samples cover a variety of possible violations and complex scenarios, providing rich data support for model training and testing. By generating initial audit results corresponding to each content audit model. In order to further improve the accuracy and reliability of the audit results, reasonable weight coefficients are assigned to the initial audit results generated by each content audit model. These coefficients are dynamically adjusted based on factors such as the model's performance, applicable scenarios, and user feedback. For example, the question classification model is given a higher weight (0.7) due to its excellent text classification capabilities; while OpenAI Moderation is given a moderate weight (0.5) due to its powerful image recognition capabilities;

[0096] GLM-3-Turbo received the highest weight (0.8) due to its comprehensive content audit capabilities. The final audit result was obtained by weighted summation. This result is not only more in line with the actual situation, but also more convincing and credible. This embodiment aligns and integrates MLCommons with the TC260-003 classification standard, and constructs a composite classification standard as the classification basis for the content audit model; by configuring detailed judgment items, connecting to initial problem negative samples, allocating reasonable weight coefficients, and introducing composite verification methods; successfully realizing the diversification, comprehensiveness and precision of content audits, and also making the subsequent test questions included in the content security test set more correct.

[0097] In one embodiment of the present invention, S120 specifically includes the following steps:

[0098] Extract the violation characterization, violation degree, keywords and violation type from the audit results;

[0099] Check whether the violation characterization in the audit results is consistent with the nature of the key content definition in the prompt word framework;

[0100] Compare the violation level in the audit results with the severity level defined in the key content of the prompt word framework;

[0101] Verify whether the keywords extracted from the audit results exist within the scope of the key content in the prompt word framework;

[0102] Compare the violation types in the audit results with the violation types defined in the key content of the prompt word framework to see if they are consistent;

[0103] When the violation characterization, violation degree, keywords and violation type in the audit result are consistent with the key content in the prompt word framework, the audit result is judged to match the requirements of the prompt word framework.

[0104] In this embodiment, the purpose is to efficiently and accurately match the audit results with the key content in the prompt word framework, and to build a more accurate content audit system through the four core elements of violation characterization, violation degree, keywords and violation type. Violation characterization refers to the definition of the specific nature of the rules violated by the audit content, such as pornography, violence, rumors, etc. When extracting the violation characterization of the audit results, it is necessary not only to follow clear rules and standards, but also to make a comprehensive judgment in combination with specific contexts and situations. In order to ensure the accuracy of the violation characterization, the audit results can be compared with the key content definitions in the prompt word framework one by one to see if they match. The degree of violation is an important indicator to measure the severity of the violation. When comparing the audit results with the degree of violation in the prompt word framework, a hierarchical management method can be adopted to divide the violations into different levels, such as minor, general, serious, etc. This classification not only helps to better understand the nature and impact of the violation, but also provides a strong basis for subsequent punishment measures. In order to ensure the accuracy of the degree of violation, the violation details in the audit results, such as the number of illegal contents, the scope of dissemination, the degree of impact, etc., can be analyzed in detail, and they can be matched one by one with the severity level in the prompt word framework. Keywords are important clues in the audit results, which can directly point to the core of the illegal content. In order to verify whether the keywords extracted from the audit results are accurate, they can be carefully compared with the key content in the prompt word framework. You can also pay attention to the frequency of occurrence, position and collocation of keywords with other words to comprehensively evaluate their importance in the audit results. The violation type refers to the specific category to which the violation belongs, such as political sensitivity, infringement and piracy, vulgarity, etc. When comparing the audit results with the violation types in the prompt word framework, you can maintain a high degree of sensitivity and accuracy. Since the types of violations are numerous and complex, the prompt word framework can be continuously updated and improved to meet new challenges. In order to ensure the accuracy of the violation types, the audit results can be carefully classified and summarized, and compared with the violation types in the prompt word framework one by one. At the same time, you can also pay attention to the cross-cutting and comprehensive characteristics of the violations to comprehensively evaluate their potential risks and impacts. By conducting detailed comparison and analysis from the four aspects of violation qualitative, violation degree, keywords and violation type, you can more accurately judge whether the audit results match the key content in the prompt word framework. This matching method not only improves the accuracy and efficiency of the review, but also provides strong support for building a more accurate and efficient content review system.

[0105] Based on the same inventive concept, an embodiment of the present application also provides a content security test set generation device, which can be used to implement a content security test set generation method described in the above embodiment, as described in the following embodiment. Since the principle of solving the problem by a content security test set generation device is similar to that of a content security test set generation method, the implementation of the device can refer to the implementation of a content security test set generation method, and the repeated parts will not be repeated. As used below, the term "unit" or "module" can be a combination of software and / or hardware that implements predetermined functions. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceived.

[0106] The present invention provides a content security test set generation device, such as Figure 2 As shown, in Figure 2 The device comprises:

[0107] Generating module 210: used to configure the prompt word framework to guide the question to build a large model to generate initial question negative samples;

[0108] Verification module 220: used to comprehensively verify various test questions in the initial question negative sample by combining multiple content review models to obtain review results;

[0109] Comparison module 230: used to compare the audit result with the key content in the prompt word framework to see if they match;

[0110] Expansion module 240: used to regard the test questions corresponding to the successfully matched audit results as valid test questions and include them in the content security test set.

[0111] In one embodiment, the generating module 210 is specifically used for:

[0112] Clearly define the problem and build a large model to generate one sentence per interaction;

[0113] Define the problem and build a large model to generate test questions;

[0114] Use the preset application scenario library to set the test problem generation environment;

[0115] Use object library resources to specify the objects that test questions target;

[0116] Select different styles from the communication library to set the communication method that matches the atmosphere of the generated environment;

[0117] Clarify the key contents of the test questions based on the content security classification benchmark library;

[0118] The selected key content is randomly combined with the generation environment, objects and communication methods to construct the prompt word framework;

[0119] Based on the prompt word framework, the question is guided to build a large model for content generation to obtain the initial question negative sample.

[0120] In one embodiment, the verification module 220 is specifically used for:

[0121] Align and integrate the MLCommons and TC260-003 classification standards to obtain a composite classification standard;

[0122] Use the composite classification criteria as the classification basis for each content review model;

[0123] Configure the judgment items corresponding to each content review model, including violation characterization, violation degree, keywords, and violation type;

[0124] Connect the initial problem negative samples to multiple different content review models through the API interface to generate the initial review results corresponding to each content review model;

[0125] A weight coefficient is assigned to the initial audit result generated by each content audit model to obtain the audit result.

[0126] In one embodiment, the comparison module 230 is specifically used for:

[0127] Extract the violation characterization, violation degree, keywords and violation type from the audit results;

[0128] Check whether the violation characterization in the audit results is consistent with the nature of the key content definition in the prompt word framework;

[0129] Compare the violation level in the audit results with the severity level defined in the key content of the prompt word framework;

[0130] Verify whether the keywords extracted from the audit results exist within the scope of the key content in the prompt word framework;

[0131] Compare the violation types in the audit results with the violation types defined in the key content of the prompt word framework to see if they are consistent;

[0132] When the violation characterization, violation degree, keywords and violation type in the audit result are consistent with the key content in the prompt word framework, the audit result is judged to match the requirements of the prompt word framework.

[0133] The embodiments of the present application also provide a specific implementation of an electronic device that can implement all the steps in the method in the above embodiments, see Figure 3 , the electronic device 300 specifically includes the following contents:

[0134] Processor 310, memory 320, communication unit 330 and bus 340;

[0135] The processor 310 , the memory 320 , and the communication unit 330 communicate with each other via the bus 340 ; the communication unit 330 is used to implement information transmission between server-side devices and terminal devices and other related devices.

[0136] The processor 310 is used to call the computer program in the memory 320, and when the processor executes the computer program, all the steps in a content security test set generation method in the above embodiment are implemented.

[0137] Those skilled in the art should understand that the memory may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory is used to store programs, and the processor executes the programs after receiving the execution instruction. Furthermore, the software programs and modules in the above-mentioned memory may also include an operating system, which may include various software components and / or drivers for managing system tasks (such as memory management, storage device control, power management, etc.), and may communicate with various hardware or software components to provide an operating environment for other software components.

[0138] The processor may be an integrated circuit chip having the ability to process signals. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0139] The present application also provides a computer-readable storage medium, which includes a program. When the program is executed by a processor, it is used to execute a content security test set generation method provided by any of the aforementioned method embodiments.

[0140] Those skilled in the art should understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, etc., various media that can store program codes, and the specific media type is not limited in this application.

[0141] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for generating a content security test set, characterized in that: The method comprises: Configure the prompt word framework to guide the question construction and generate initial question negative samples; A variety of content review models are used to conduct a composite review of each test question in the initial question negative sample to obtain the review results; Compare the audit results with the key content in the prompt word framework to see if they match; The test questions corresponding to the successfully matched audit results are considered valid test questions and included in the content security test set.

2. A method for generating a content security test set according to claim 1, characterized in that: The configuration prompt word framework guides the question to build a large model to generate initial question negative samples, including: Clearly define the problem and build a large model to generate one sentence per interaction; Define the problem and build a large model to generate test questions; Use the preset application scenario library to set the test problem generation environment; Use object library resources to specify the objects that test questions target; Select different styles from the communication library to set the communication method that matches the atmosphere of the generated environment; Clarify the key contents of the test questions based on the content security classification benchmark library; The selected key content is randomly combined with the generation environment, objects and communication methods to construct the prompt word framework; Based on the prompt word framework, the question is guided to build a large model for content generation to obtain the initial question negative sample.

3. A method for generating a content security test set as claimed in claim 2, characterized in that: The comprehensive multiple content audit models perform a composite review on each test question in the initial question negative sample, and the audit results obtained include: Align and integrate the MLCommons and TC260-003 classification standards to obtain a composite classification standard; Use the composite classification criteria as the classification basis for each content review model; Configure the judgment items corresponding to each content review model, wherein the judgment items include violation characterization, violation degree, keywords and violation type; Connect the initial problem negative samples to multiple different content review models through the API interface to generate the initial review results corresponding to each content review model; A weight coefficient is assigned to the initial audit result generated by each content audit model to obtain the audit result.

4. A method for generating a content security test set according to claim 1, characterized in that: Whether the comparison and review results match the key content in the prompt word framework includes: Extract the violation characterization, violation degree, keywords and violation type from the audit results; Check whether the violation characterization in the audit results is consistent with the nature of the key content definition in the prompt word framework; Compare the violation level in the audit results with the severity level defined in the key content of the prompt word framework; Verify whether the keywords extracted from the audit results exist within the scope of the key content in the prompt word framework; Compare the violation types in the audit results with the violation types defined in the key content of the prompt word framework to see if they are consistent; When the violation characterization, violation degree, keywords and violation type in the audit result are consistent with the key content in the prompt word framework, the audit result is judged to match the requirements of the prompt word framework.

5. A content security test set generation device, characterized in that: The device includes: Generation module: used to configure the prompt word framework to guide the question construction and generate initial question negative samples; Verification module: used to comprehensively verify various test questions in the initial question negative sample by integrating multiple content review models to obtain the review results; Comparison module: used to compare the audit results with the key content in the prompt word framework to see if they match; Expansion module: used to regard the test questions corresponding to the successfully matched audit results as valid test questions and include them in the content security test set.

6. A content security test set generation device as claimed in claim 5, characterized in that: The generation module is specifically used for: Clearly define the problem and build a large model to generate one sentence per interaction; Define the problem and build a large model to generate test questions; Use the preset application scenario library to set the test problem generation environment; Use object library resources to specify the objects that test questions target; Select different styles from the communication library to set the communication method that matches the atmosphere of the generated environment; Clarify the key contents of the test questions based on the content security classification benchmark library; The selected key content is randomly combined with the generation environment, objects and communication methods to construct the prompt word framework; Based on the prompt word framework, the question is guided to build a large model for content generation to obtain the initial question negative sample.

7. A content security test set generation device as claimed in claim 6, characterized in that: The verification module is specifically used for: Align and integrate the MLCommons and TC260-003 classification standards to obtain a composite classification standard; Use the composite classification criteria as the classification basis for each content review model; Configure the judgment items corresponding to each content review model, wherein the judgment items include violation characterization, violation degree, keywords and violation type; Connect the initial problem negative samples to multiple different content review models through the API interface to generate the initial review results corresponding to each content review model; A weight coefficient is assigned to the initial audit result generated by each content audit model to obtain the audit result.

8. A content security test set generation device as claimed in claim 5, characterized in that: The comparison module is specifically used for: Extract the violation characterization, violation degree, keywords and violation type from the audit results; Check whether the violation characterization in the audit results is consistent with the nature of the key content definition in the prompt word framework; Compare the violation level in the audit results with the severity level defined in the key content of the prompt word framework; Verify whether the keywords extracted from the audit results exist within the scope of the key content in the prompt word framework; Compare the violation types in the audit results with the violation types defined in the key content of the prompt word framework to see if they are consistent; When the violation characterization, violation degree, keywords and violation type in the audit result are consistent with the key content in the prompt word framework, the audit result is judged to match the requirements of the prompt word framework.

9. An electronic device, characterized in that: include: processor, memory, and an interface for communicating with a gateway; The memory is used to store programs and data, and the processor calls the program stored in the memory to execute a content security test set generation method as described in any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a program, which, when executed by a processor, is used to execute a content security test set generation method as described in any one of claims 1 to 4.

Images