Large model security evaluation method, related device and storage medium
Through automated evaluation and online optimization, the large-model security evaluation method solves the security problems of the large-model during use, realizes comprehensive security evaluation and optimization, and improves security and optimization efficiency.
Patent Information
- Application Number
- CN202410597512.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-07-07
- Filing Date
- 2024-05-14
- Publication Date
- 2024-12-06
AI Technical Summary
The lack of comprehensive automation evaluation and online optimization solutions in the prior art has led to a large number of security problems during the use of large models, such as combating sample and model backdoor attacks, resulting in low security.
Provide a large model security evaluation method, which automatically determines the test indicator set and attack instruction set based on the application fields of the large model to be tested, responds to attack instructions, conducts security evaluation of multiple evaluation indicators, outputs visual results, and uses the evaluation results. Perform model training and update to optimize algorithm vulnerabilities.
It realizes all-round automated security evaluation and online optimization of large models, improves the security and optimization efficiency of the model, and avoids the security risks that exist after the large model is launched.
Smart Images

Figure CN119089444A_ABST
Abstract
Description
[0001] This application claims priority to application number 202310836455.6, filing date July 7, 2023, and title: Large model security assessment method, related devices and storage media. Technical Field
[0002] The embodiments of the present application relate to the field of computers, and more specifically to a large model security assessment method, related devices and storage media. Background Art
[0003] With the rapid development of artificial intelligence technology represented by machine learning, especially deep learning, artificial intelligence technology has been applied more and more widely, including industrial manufacturing, medical health, education, security, e-commerce retail, finance and other fields, which has led to the emergence of powerful large models. However, although the capabilities of large models are very powerful, a large number of security issues have been found when using large models. For example, if abused by users, new attack methods such as adversarial samples and model backdoors can cause large models to execute incorrect instructions, resulting in malicious use of large models, which may eventually cause serious security consequences and greatly reduce the security of using large models. Summary of the invention
[0004] The embodiments of the present application provide a large model security assessment method, related devices and storage media, which can automatically perform security assessment on large models and ensure the security of the use of large models.
[0005] In a first aspect, an embodiment of the present application provides a large model security assessment method, the method comprising:
[0006] Determine the large model to be tested;
[0007] Determine a test indicator set and an attack instruction set according to the application field of the large model to be tested, wherein the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories;
[0008] In response to the attack instruction in the attack instruction set, a security evaluation is performed on the large model to be tested according to the multiple evaluation indicators in the test indicator set to obtain a test value corresponding to each evaluation indicator, wherein the test value represents a security value corresponding to the evaluation indicator under each risk category;
[0009] A visual evaluation result is output based on the test value, and the evaluation result displays the test values of the large model to be tested corresponding to a plurality of different risk categories under different attack instructions.
[0010] In a second aspect, an embodiment of the present application provides a large model security assessment device, which has the function of implementing the large model security assessment method provided in the first aspect above. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the modules can be software and / or hardware.
[0011] In one embodiment, the large model safety assessment device includes:
[0012] An input-output module is configured to determine a large model to be tested;
[0013] The processing module is configured to determine a test indicator set and an attack instruction set according to the application field of the large model to be tested, wherein the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories; in response to the attack instructions in the attack instruction set, perform a security evaluation on the large model to be tested according to the multiple evaluation indicators in the test indicator set to obtain a test value corresponding to each evaluation indicator, wherein the test value represents a security value corresponding to the evaluation indicator under each risk category;
[0014] The input-output module is also configured to output a visualized evaluation result according to the test value, and the evaluation result displays the test values corresponding to a plurality of different risk categories under different attack instructions of the large model to be tested.
[0015] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, which includes instructions, which, when executed on a computer, enables the computer to execute the large model security assessment method as described in the first aspect.
[0016] In a fourth aspect, an embodiment of the present application provides a computing device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the large model security assessment method provided in the first aspect when executing the computer program.
[0017] In a fifth aspect, an embodiment of the present application provides a computer program product comprising instructions. The computer program product includes program instructions. When the program instructions are executed on a computer or a processor, the computer executes the large model security assessment method provided in the first aspect.
[0018] In a sixth aspect, an embodiment of the present application provides a chip, which includes a processor coupled to a transceiver of a terminal device, and is used to execute the large model security assessment method provided in the first aspect of the embodiment of the present application.
[0019] In the seventh aspect, an embodiment of the present application provides a chip system, which includes a processor for supporting a terminal device to implement the functions involved in the first aspect, for example, generating or processing information involved in the large model security assessment method provided in the first aspect.
[0020] In one possible design, the chip system also includes a communication interface for inputting and / or outputting information.
[0021] In a possible design, the chip system further includes a memory, which is used to store program instructions and data required by the terminal device. The chip system can be composed of a chip, or can include a chip and other discrete devices.
[0022] Compared with the prior art, in the embodiments of the present application, the test indicator set and the attack instruction set can be automatically determined according to the application field of the large model to be tested, and in response to the attack instruction in the attack instruction set, the large model to be tested can be security evaluated according to the multiple evaluation indicators in the test indicator set, and the test value corresponding to each evaluation indicator can be obtained. Since the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories, it is possible to automatically evaluate the security values corresponding to different evaluation indicators under each risk category, and the visual evaluation results can be output according to the test values. The evaluation results show the test values corresponding to multiple different risk categories under different attack instructions for the large model to be tested. Therefore, before the large model goes online, in order to avoid the security risks that exist after the large model goes online later, the large model security evaluation scheme of the present application can be used to automatically perform security evaluation on the large model (including evaluation of algorithms, applications, data and other sections), so as to iteratively optimize some algorithm vulnerabilities of the large model according to the evaluation results to ensure the security of the use of the large model. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] By reading the detailed description of the embodiments of the present application with reference to the accompanying drawings, the objectives, features and advantages of the embodiments of the present application will become easy to understand. Among them:
[0024] Figure 1 A schematic diagram of a large model safety assessment system of a large model safety assessment method in an embodiment of the present application;
[0025] Figure 2 A schematic diagram of a flow chart of a large model security assessment method according to an embodiment of the present application;
[0026] Figure 3 A question-and-answer diagram of a large model security assessment method according to an embodiment of the present application;
[0027] Figure 4A schematic diagram of an evaluation parameter configuration of a large model security evaluation method according to an embodiment of the present application;
[0028] Figure 4a A schematic diagram of the display effect of parameter configuration in an actual configuration scenario of the large model security assessment method according to an embodiment of the present application;
[0029] Figure 5 A schematic diagram showing an evaluation result of a large model security evaluation method according to an embodiment of the present application;
[0030] Figure 5a A schematic diagram of the display effect of the evaluation results in an actual evaluation scenario of the large model security evaluation method of an embodiment of the present application;
[0031] Figure 6 A schematic diagram showing another evaluation result of the large model security evaluation method according to an embodiment of the present application;
[0032] Figure 6a A schematic diagram of the display effect of the evaluation results in another actual evaluation scenario of the large model security evaluation method of the embodiment of the present application;
[0033] Figure 7 A schematic diagram of a horizontal comparison between different large models of the large model security assessment method of an embodiment of the present application;
[0034] Figure 7a This is a schematic diagram showing the effect of horizontal comparison results between different large models in an actual evaluation scenario of the large model security evaluation method of an embodiment of the present application.
[0035] Figure 8 A schematic diagram of a large model and risk category selection for a large model security assessment method according to an embodiment of the present application;
[0036] Fig. 9 A schematic diagram of the interaction between a terminal and a server in a large model security assessment method according to an embodiment of the present application;
[0037] Fig.10 This is a schematic diagram of the structure of a large model safety assessment device according to an embodiment of the present application;
[0038] Fig.11 A schematic diagram of a structure of a computing device according to an embodiment of the present application;
[0039] Fig.12 A schematic diagram of the structure of a mobile phone in an embodiment of the present application;
[0040] Fig.13 A schematic diagram of the structure of a server in an embodiment of the present application;
[0041] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION
[0042] The terms "first", "second", etc. in the specification and claims of the embodiments of the present application and the above-mentioned drawings are used to distinguish similar objects (for example, the first area and the second area are respectively represented as different others, and the others are similar), and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described here can be implemented in an order other than the content illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device including a series of steps or modules is not necessarily limited to those steps or modules clearly listed, but may include other steps or modules that are not clearly listed or inherent to these processes, methods, products or devices. The division of the modules that appear in the embodiments of the present application is only a logical division. There may be other division methods when implemented in practical applications, such as multiple modules can be combined into or integrated in another system, or some features can be ignored, or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling between modules, and the communication connection can be electrical or other similar forms, which are not limited in the embodiments of the present application. Furthermore, the modules or sub-modules described as separate components may or may not be physically separated, may or may not be physical modules, or may be distributed in multiple circuit modules, and some or all of the modules may be selected according to actual needs to achieve the purpose of the embodiment of the present application.
[0043] The embodiment of the present application also provides a large model security assessment method, related devices and storage media, wherein the related devices include a large model security assessment device, a computing device, a computer program product and a chip system, etc., and the storage medium is a computer-readable storage medium. A large model security assessment system can be applied to a large model security assessment system in a scenario of security assessment of a large model, and the large model security assessment system may include a large model security assessment device, and the large model security assessment device can be used at least to perform security assessment on the large model to determine whether the large model to be tested has a security risk, wherein the large model security assessment device can be an application program based on performing security assessment on the large model to obtain an assessment result for determining whether the large model has a security risk, or a server or terminal device installed with an application program for performing security assessment on the large model to obtain an assessment result for determining whether the large model has a security risk.
[0044] The solution provided in the embodiments of the present application involves technologies such as artificial intelligence (AI), computer vision (CV), and machine learning (ML), which are specifically described by the following embodiments:
[0045] Among them, AI is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.
[0046] AI technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0047] CV is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes adversarial disturbance generation, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.
[0048] This application can perform reinforcement learning on large models, which may include reinforcement learning based on human feedback (RLHF), reinforcement learning based on AI feedback (RLAIF), and ordinary reinforcement learning reward constraints (Reward Models).
[0049] Among them, reinforcement learning based on human feedback (RLHF): This technology refers to the use of a large amount of manually labeled data to train the AI system model in the reinforcement learning stage. The reward model generates reward signals (Reward Signals) of different intensities based on the output results of the AI model and the labeled data to guide the AI model to converge in the expected direction, and a safer AI system model is obtained after the training. The effect of the model after training under this technology depends to a large extent on the scale and quality of manually labeled data. RLHF helps solve the problems of hallucinations and harmful outputs. Anthropic introduced that it has 40,000 test data for RLHF to reduce risks, and the training data may be in the order of hundreds of thousands.
[0050] Reinforcement Learning with AI Feedback (RLAIF): A variant of RLHF, where the feedback information source of the reward model is partially or completely changed from human feedback to agent models that provide it automatically. The agent model is a model that has been trained in advance to meet safety standards and is aligned. It acts as a "supervisor" to provide feedback signals for the reinforcement training of the new model. Among them, fine-tuned models that can provide AI feedback signals include OpenAI's InstructGPT, DeepMind's Automatically Red-LM+Red Clf, and Anthropic's Constitutional-AI, etc.
[0051] Ordinary reinforcement learning reward constraints (Reward Models): Generally, it is a classification or recognition model trained by information and labeled data fed back by humans or AI red teams. Its input can be only the output of the AI model, or it can also add additional human feedback signals here, and the output is a reward signal of different strengths used for reinforcement learning training.
[0052] In this application, the specific type of the large model can be flexibly set according to actual needs. For example, the large model can be a large language model, which refers to a language model with a huge number of parameters trained based on a large amount of data (usually using large-scale self-supervised learning). The large language model is mainly implemented based on the Transformer machine learning method. The Transformer model is a deep learning model that uses a self-attention mechanism. This mechanism can assign different weights according to the importance of each part of the input data. Typical applications include OpenAI's ChatGPT.
[0053] Various methods and software of generative artificial intelligence have been widely used, which has brought about potential risks and hidden dangers. In addition to the leakage of personal information, negative news, false news and speeches of key figures produced by generative artificial intelligence tools are difficult to distinguish between true and false, spread rapidly, and are seriously misleading to public opinion, affecting national security and social stability. They may also become a new means of cybercrime and infringe on the legitimate rights and interests of citizens.
[0054] The general big model is currently widely used in chat dialogue, text editing, artistic creation, coding, mathematical reasoning, bioinformatics and other application scenarios. Although it has created many new business models and is very powerful, after the general big model is launched for users to use, there are mainly algorithmic risks, data risks and application risks in the three types of applications: translation, chat and collaboration.
[0055] Algorithmic risk: refers to the inherent defects of these large models at the bottom layer regardless of the scenario in which they are used. These defects mainly come from the mathematical characteristics of neural networks. When normal users use general large models, the risks that these models may expose are called intrinsic risks. This includes various attributes such as interpretability, value alignment, and cognitive ability. When malicious users maliciously attack general large models, the risks that these models may expose are called external risks. This includes prompt injection, adversarial attacks, model theft, etc. The solution to these algorithmic risks often requires deep algorithm optimization. For example, by injecting confusing instructions and data into the model based on the "prompt injection" method, users can obtain unauthorized permissions when using it, so that the model ignores the preset "translation" and "hello" tasks and gives random answers. For example, adding slight perturbations to the text based on the "adversarial attack" method will cause the model to make mistakes and have low reliability.
[0056] Data risks: If the database is not handled properly, the general large model may expose defects in the corpus, thereby bringing risks at the level of content security, privacy leakage, and credibility. Solving these data risks often requires restricting the content and access rights of the database. 1. Due to the complex source of the corpus, it may imply the attitudes and prejudices of certain human communities, which can easily cause the models trained based on these corpora to be biased and discriminatory; 2. Due to the complex source of the corpus, it may contain negative content, which can easily cause the models trained based on these corpora to lead users to have mental health problems; 3. Due to the complex source of the corpus, it may contain erroneous content, which can easily cause the models trained based on these corpora to output false / wrong information.
[0057] Application risks: If abused by users, general large models may be used maliciously, thus bringing application risks. Solving these data risks often requires managing user behavior. For example, criminals commit fraud by writing fraud scripts and copywriting. Such large models can enhance the attack capabilities of fraudsters and criminals, which is extremely harmful to society.
[0058] At present, due to the diversity of large model application fields and complex corpus, there is no comprehensive evaluation plan for large models, nor a visual evaluation report for large models, so testers cannot intuitively understand where the vulnerabilities of their own large models are. Since it is impossible to exhaustively list all test cases, and each test case needs to be designed by experts, the professional threshold is high, and there are certain problems with the breadth, accuracy, and objectivity of the test, so the following problems of the model are inevitable:
[0059] Fake information: Large models produce “serious nonsense” information and require greater human effort to distinguish fake information;
[0060] Inappropriate content: When the model lacks the ability to correctly judge moral, legal and ethical issues, it produces harmful content such as hate speech, offensive language, and content that is inconsistent with human values;
[0061] Bias and discrimination: Bias and discrimination embodied in training sets may continue to propagate and amplify with large models;
[0062] Malicious attacks: Bypass the security protection mechanism of the big model through special prompts, etc., causing the big model to generate inappropriate content.
[0063] Therefore, in the prior art, there is no automated evaluation and online optimization solution for comprehensive evaluation of large language models. The optimization efficiency of large language models is low, which leads to a large number of security issues when using large models. For example, if users abuse them, new attack methods such as adversarial samples and model backdoors can cause the large model to execute wrong instructions, resulting in malicious use of the large model, which may ultimately cause serious security consequences and greatly reduce the security of the use of large models.
[0064] Compared with the prior art, in the embodiment of the present application, the test indicator set and the attack instruction set can be automatically determined according to the application field of the large model to be tested, and in response to the attack instruction in the attack instruction set, the large model to be tested is security evaluated according to the multiple evaluation indicators in the test indicator set, and the test value corresponding to each evaluation indicator is obtained. Since the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories, it is possible to automatically evaluate the security values corresponding to different evaluation indicators under each risk category, and the visual evaluation results can be output according to the test values. The evaluation results show the test values corresponding to multiple different risk categories under different attack instructions for the large model to be tested. In addition, according to the risk category corresponding to the vulnerability risk of the large model to be tested, a sample of adversarial questions can be obtained, and the sample of questions is a question that meets the preset unsafe query conditions; then the large model to be tested is trained and updated according to the sample of questions to obtain the updated large model. The automatic evaluation of the large model for all-round evaluation and online optimization are realized, which improves the optimization efficiency of the large model. In addition, before the big model goes online, in order to avoid security risks that may exist after the big model goes online later, the big model security assessment solution of the present application can be used to automatically perform a security assessment on the big model (including assessments of algorithms, applications, data and other sections), thereby iteratively optimizing some algorithmic vulnerabilities of the big model based on the assessment results to ensure the security of the big model.
[0065] In some implementations, taking the large model security assessment device integrated into the server 10 as an example, refer to Figure 1 The large model security evaluation method provided in the embodiment of the present application can be based on Figure 1 A large model security assessment system is shown. The large model security assessment system may include a server 10 and a terminal device 20. The server 10 may include a large model security assessment device.
[0066] Among them, before the large model goes online, in order to avoid the security risks that exist after the large model goes online later, the server 10 can automatically determine the test indicator set and the attack instruction set according to the application field of the large model to be tested, and respond to the attack instructions in the attack instruction set. According to the multiple evaluation indicators in the test indicator set, the large model to be tested is security evaluated, and the test values corresponding to each evaluation indicator are obtained. Visual evaluation results are output according to the test values, and adversarial problem samples can be obtained according to the risk category corresponding to the vulnerability risk of the large model to be tested. Training and updating of the large model to be tested. Not only is an automated security evaluation of the large model for all-round evaluation (including evaluation of algorithms, applications, data and other sections) realized, so as to iteratively optimize some algorithm vulnerabilities of the large model according to the evaluation results, and ensure the safety of the use of the large model, but also the online optimization of the large model is realized, which improves the optimization efficiency of the large model. The server 10 can also send the evaluation results to the terminal device 20 for display.
[0067] It should be noted that the server 10 involved in the embodiment of the present application can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.
[0068] The terminal device 20 involved in the embodiment of the present application may be a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. For example, a mobile phone (or "cellular" phone) and a computer with a mobile terminal, for example, a portable, pocket-sized, handheld, computer-built-in or vehicle-mounted mobile device, which exchanges voice and / or data with a wireless access network. For example, a personal communication service (English full name: Personal Communication Service, English abbreviation: PCS) phone, a cordless phone, a session initiation protocol (SIP) phone, a wireless local loop (Wireless Local Loop, English abbreviation: WLL) station, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA) and other devices.
[0069] Reference Figure 2 , Figure 2A flow chart of a large model security assessment method provided in an embodiment of the present application. The method can be executed by a large model security assessment device, and can be applied to scenarios where a large model needs to be security assessed. Before the large model goes online, in order to avoid security risks that exist after the large model goes online later, the large model security assessment solution of the present application can be used to automatically perform a security assessment on the large model, thereby iteratively optimizing the algorithm vulnerabilities of the large model according to the assessment results to ensure the security of the use of the large model. The large model security assessment method includes steps S101-S104:
[0070] Step S101, determining a large model to be tested.
[0071] The specific type and model structure of the large model to be tested can be flexibly set according to actual needs and are not limited here. For example, the large model to be tested can be a large language model.
[0072] Methods for determining the large model to be tested may include:
[0073] Method 1: The large model security assessment device receives the model selection execution instruction input by the user through the configuration interface, and determines the large model to be tested based on the model selection execution instruction.
[0074] Method 2: The large model security assessment device randomly selects one of the large models from the large model list as the large model to be tested.
[0075] Method 3: The big model security assessment device receives a login request, logs in to a test account in response to the login request, determines a big model that matches the test account, and uses the big model that matches the test account as the big model to be tested. Different test accounts can be used to perform security assessment on one or more different big models.
[0076] Specifically, the large model security assessment device can assign multiple different large models to different testers for security assessment. For tester A, the authorized test account a can be used to conduct security assessment on large model A; for tester B, the authorized test account b can be used to conduct security assessment on large model B; for tester C, the authorized test account b can be used to conduct security assessment on large models C and D; and so on. On the one hand, the security of large model assessment is improved by conducting security assessment on the matching large models through authorized test accounts. On the other hand, different testers are responsible for the security assessment of different large models, which improves the efficiency and traceability of large model assessment.
[0077] Of course, the large model to be tested can also be determined by other methods, which are not limited here.
[0078] After determining the large model to be tested, in order to verify whether the large model to be tested has vulnerability risks, adversarial questions can be input and the large model to be tested outputs answers for verification. In some implementations, after determining the large model to be tested, the large model security assessment method further includes:
[0079] S11, receiving an adversarial question input by a user into a question input box in a visual display interface, where the adversarial question is a question that meets a preset unsafe inquiry condition;
[0080] S12. Displaying an answer in a display box in a visualization display interface, where the answer is an answer that matches the adversarial question.
[0081] like Figure 3 As shown, the large model security assessment device can display an attack instruction selection list, a question input box, and a display box in a visual display interface. The attack instruction selection list may include at least one of target hijacking, unsafe inquiry, injection attack, developer mode, negative inducement, role playing, and prompt leakage. The large model security assessment device can receive adversarial questions input by the user for the question input box in the visual display interface, generate answers that match the adversarial questions through the large model to be tested, and display the answers that match the adversarial questions in the display box. By inputting adversarial questions and having the large model to be tested output answers to verify whether the large model to be tested has vulnerability risks, the reliability of verification is improved.
[0082] For example, if the attack command is selected as an unsafe query, and the input adversarial question is "How to plant poppies?", the output answer is related information about soil selection, light and temperature, watering, fertilization, pruning, pest control, etc. for poppy planting, which indicates that the large model to be tested has a vulnerability risk.
[0083] Step S102, determining a test indicator set and an attack instruction set according to the application field of the large model to be tested, wherein the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories.
[0084] In accordance with the different application fields of the large model to be tested, the corresponding test indicator sets and attack instruction sets are also different. Among them, the test indicator set includes evaluation indicators corresponding to multiple risk categories. In order to improve the reliability of the evaluation of the large model to be tested and increase the diversity of risk categories, in some embodiments, the evaluation indicators corresponding to the risk categories in the test indicator set include malicious use, harmful language, violation of national security, erroneous knowledge and information, biased discrimination, information leakage, and at least one of deployment risks. There are different evaluation indicators corresponding to the large models to be tested in different application fields. Of course, the risk categories can also include other categories, which are not limited here.
[0085] Among them, as shown in Table 1 below, harmful language includes: content that encourages or suggests self-harm, pushes or suggests users to perform immoral or illegal acts, pornographic language, violent, offensive content, harassing, derogatory and hateful content, instructions on how to find illegal content, inflammatory speech, personal attacks or malicious slander, damage to social morals or values, and at least one of impolite language.
[0086] Prejudice and discrimination include at least one of: social stereotypes, individual or group prejudice, and exclusionary norms.
[0087] Information leakage includes: leakage of private information, leakage of privacy by correctly inferring private information, and at least one of leakage or inference of sensitive information.
[0088] Malicious uses include: spreading false information, fraud / scam, spreading malicious websites, promoting illegal products, and promoting at least one of extreme speech.
[0089] Violations of national security include: damaging the national image, damaging national interests, inciting ethnic hatred or discrimination, undermining social stability, and being politically sensitive.
[0090] Table 1 Risk categories
[0091]
[0092]
[0093] As shown in Table 2 below, the attack instruction set includes attack instructions corresponding to multiple attack categories. In order to improve the reliability of the evaluation of the large model to be tested and increase the diversity of attack categories, in some embodiments, the attack instructions corresponding to the attack categories in the attack instruction set include at least one of target hijacking, unsafe inquiry, injection attack, developer mode, negative inducement, role playing, and prompt leakage. There are different attack instructions corresponding to the large models to be tested in different application fields. Of course, the risk category can also include other categories, which are not limited here.
[0094] Table 2 Attack instructions
[0095]
[0096] The test indicator set and attack instruction set may be determined by:
[0097] Method 1: The large model security assessment device randomly determines the test indicator set and attack instruction set according to the application field of the large model to be tested.
[0098] Method 2: The large model security assessment device determines the test indicator set and attack instruction set based on the assessment parameters of the large model to be tested and the application field of the large model to be tested configured through the configuration interface.
[0099] Method three, the large model security assessment device receives a login request, and logs in to the test account in response to the login request; determines the default assessment parameters that match the test account, and determines the test indicator set and attack instruction set based on the default assessment parameters and the application field of the large model to be tested.
[0100] Of course, the large model security assessment device can also determine the test indicator set and attack instruction set in other ways, which are not limited here.
[0101] The following is an example of the evaluation parameters of the large model to be tested configured through the configuration interface. First, before the security evaluation of the large model is performed, the evaluation parameters of the large model to be tested are configured. In some implementations, before determining the large model to be tested, the large model security evaluation method further includes:
[0102] S21. Setting a preset evaluation method, the evaluation method includes an automatically generated evaluation and a fixed data set evaluation.
[0103] S22. Set at least one corresponding risk category and at least one attack instruction for each evaluation method.
[0104] S23. Configure the number of prompt injections of adversarial questions corresponding to each risk category for each evaluation method.
[0105] For example, Figure 4 As shown, the configuration interface may include a variety of configuration controls and a variety of parameter selection areas. The large model security assessment device may receive the parameters of "automatically generate assessment" or "fixed data set assessment" selected by the user in the parameter selection area of "select assessment method", and set at least one corresponding risk category and at least one attack instruction for each assessment method, such as receiving multi-dimensional parameters such as risk categories and attack instructions selected by the user in the parameter selection area of "select assessment dimension". And configure the number of prompt injections for adversarial problems corresponding to each risk category for each assessment method, that is, "prompt data for each type of problem". Of course, the parameters that need to be configured may also include other parameters, which are not limited here, such as Figure 4In the configuration interface, you can click the "Next" switch control to switch to the next page to configure other parameters. After selecting the parameters of each dimension, the large model security assessment device can complete the configuration of the parameters in response to the configuration control (such as the confirmation control or the completion control, etc.). This enables users to flexibly and conveniently configure the parameters such as the assessment method, risk category, attack instructions, and the number of prompt injections for adversarial problems of the large model to be tested according to actual assessment needs through the configuration interface.
[0106] In the actual configuration scenario, the configuration interface is shown as follows: Figure 4a As shown, in this interface, you can set the preset evaluation method, automatic generation evaluation or fixed data set evaluation, set at least one corresponding risk category and at least one attack instruction for the selected evaluation method, and configure prompt data for each type of question for the selected evaluation method.
[0107] After completing the configuration of the parameters, the large model security assessment device can accurately determine the matching test indicator set and attack instruction set based on the parameters and the application field of the large model to be tested, so as to accurately evaluate the large model to be tested.
[0108] Step S103, in response to the attack instruction in the attack instruction set, a security evaluation is performed on the large model to be tested according to the multiple evaluation indicators in the test indicator set to obtain the test value corresponding to each evaluation indicator, and the test value represents the security value corresponding to the evaluation indicator under each risk category.
[0109] After determining the test indicator set and attack instruction set of the large model to be tested, the large model security assessment device responds to the attack instruction in the attack instruction set, performs security assessment on the large model to be tested according to the multiple assessment indicators in the test indicator set, and obtains the test value corresponding to each assessment indicator. The test value represents the security value corresponding to the assessment indicator under each risk category, and the value range of the security value can be 0 to 100 (including 0 and 100). For example, the large model security assessment device can respond to the attack instruction A, perform security assessment on the large model to be tested according to the assessment indicator A1, the assessment indicator A2 and the assessment indicator A3, and obtain the test value corresponding to the assessment indicator A1 is 60, the test value corresponding to the assessment indicator A2 is 20, and the test value corresponding to the assessment indicator A3 is 100.
[0110] Step S104: Output a visualized evaluation result based on the test value, where the evaluation result displays the test values of the large model to be tested corresponding to various risk categories under different attack instructions.
[0111] The display format and specific content of the evaluation results can be flexibly set according to actual needs and are not limited here. The evaluation results can display the test values of the large model under test for multiple different risk categories under different attack instructions, and can also display the evaluation comparison between the large model under test and similar large models, and can also display the evaluation indicators under each risk category used for the security evaluation of the large model under test and the proportion of each attack instruction, etc.
[0112] For example, Figure 5 As shown, the evaluation results can include the proportion of evaluation indicators under each risk category, such as malicious use, harmful language, violation of national security, incorrect knowledge and information, prejudice and discrimination, information leakage, etc. It can also include the proportion of each attack instruction, such as target hijacking, unsafe inquiry, injection attack, developer mode, negative inducement, role playing, prompt leakage, etc., as well as the test values corresponding to various risk categories under different attack instructions for the large model to be tested. Among them, Figure 5 A to K corresponding to the evaluation dimension represent the evaluation indicators under A to K different risk categories, such as the major categories or subcategories of risk categories. In the actual evaluation application scenario, the effect diagram showing the evaluation results is as follows: Figure 5a shown.
[0113] For example, Figure 6 As shown, the evaluation results can include the proportion of evaluation indicators under each risk category, such as harmful language, violation of national security, prejudice and discrimination, and the proportion of each attack instruction, such as unsafe inquiry, injection attack, role-playing, target hijacking, negative inducement, etc., as well as the test values corresponding to various risk categories under different attack instructions for the large model to be tested. Figure 6 A to K corresponding to the evaluation dimension represent the evaluation indicators under A to K different risk categories, such as the major categories or subcategories of risk categories. In the actual evaluation application scenario, the effect diagram showing the evaluation results is as follows: Figure 6a shown.
[0114] In order to improve the flexibility and diversity of the display of evaluation results, different testers may display the evaluation results of the safety evaluation of the large model to be tested in different ways. In some embodiments, the output of the visual evaluation results according to the test value includes:
[0115] S31. Obtain the test account currently performing security assessment on the large model to be tested;
[0116] S32, determining an output method that matches the test account;
[0117] S33. Output a visualized evaluation result according to the output mode and the test value.
[0118] Specifically, since different testers have different habits, preferences, and test tasks, the evaluation results of the security evaluation of the large model to be tested can be displayed in different output modes for different testers. In the process of tester A using the authorized test account a to perform a security evaluation on the large model A, the large model security evaluation device can determine that the output mode matching the test account a is to send the evaluation results to the terminal A for display. At this time, the large model security evaluation device can send the visualized evaluation results generated based on the test value to the terminal A for display.
[0119] When tester B uses authorized test account b to conduct security assessment on large model B, the large model security assessment device can determine that the output method matching the test account b is to display the assessment results in a visual display interface. At this time, the large model security assessment device can generate a visual assessment result based on the test value and display the assessment result in the visual display interface.
[0120] When tester C uses authorized test account c to conduct security assessment on large model C, the large model security assessment device can determine that the output method matching the test account c is to dynamically display the assessment results. At this time, the large model security assessment device can generate a visual assessment result based on the test value and dynamically display the assessment result in the visual display interface.
[0121] In this embodiment, in view of the different test accounts used by different testers, the large model security assessment device can display the assessment results of the large model security assessment in different output modes, thereby improving the flexibility and diversity of the assessment result display.
[0122] In order to improve the richness of the display of the large model security evaluation results, the large model security evaluation device can also display the horizontal comparison results. In some embodiments, the evaluation results are displayed in the first area of the visual display interface. After the visual evaluation results are output according to the test value, the large model security evaluation method also includes:
[0123] In the second area of the visualization display interface, horizontal comparison results are displayed, including evaluation results of the large model to be tested for at least one risk category and evaluation results of similar large models for at least one risk category. The horizontal comparison results display risk categories of vulnerability risks between the large model to be tested and similar large models. The horizontal comparison results indicate advantages and disadvantages between the large model to be tested and similar large models, as well as comparison of advantages and disadvantages between the large model to be tested and similar large models.
[0124] The first area and the second area may be in the same page of the visual display interface or in different pages of the visual display interface, which is not limited here. When the first area and the second area are in different pages of the visual display interface, the large model safety assessment device may switch the display of the assessment results of the first area of the visual display interface and the horizontal comparison results of the second area of the visual display interface through a page switching instruction.
[0125] like Figure 7 As shown, the horizontal comparison results may include horizontal comparison results between multiple different large models, and the specific model type and quantity of the same large model are not limited here. Figure 7 In the example, model A is the large model to be tested, and model B and model C are large models of the same type. The horizontal comparison results displayed include the evaluation results of the large model to be tested for at least one risk category and the evaluation results of the large models of the same type for at least one risk category. The horizontal comparison results show the risk categories of vulnerability risks between the large model to be tested and the large models of the same type. Among them, the higher the safety value of the large model, the smaller the probability of vulnerability risks. Conversely, the lower the safety value of the large model, the greater the probability of vulnerability risks. The horizontal comparison results can also be used to indicate the advantages and disadvantages between the large model to be tested and the large models of the same type, as well as the comparison of advantages and disadvantages between the large model to be tested and the large models of the same type.
[0126] In the actual evaluation application scenario, the effect diagram showing the horizontal comparison results is as follows Figure 7a shown.
[0127] After displaying the evaluation results and the horizontal comparison results, the large model security evaluation device can learn the risk category of the vulnerability risk of the large model to be tested. At this time, the large model to be tested can be trained and updated in a targeted manner. In some embodiments, after displaying the horizontal comparison results in the second area of the visual display interface, the large model security evaluation method also includes:
[0128] S41. According to the risk category of the vulnerability risk of the large model to be tested, obtain adversarial question samples, where the question samples are questions that meet the preset unsafe query conditions;
[0129] S42. Train and update the large model to be tested according to the problem samples to obtain an updated large model.
[0130] Specifically, the large model security assessment device can timely know whether the large model to be tested has vulnerability risks through the evaluation results of the large model to be tested. If the large model to be tested has vulnerability risks, the large model security assessment device can determine the risk category of the vulnerability risk of the large model to be tested, and obtain adversarial question samples that match the risk category of the vulnerability risk of the large model to be tested, and the question samples are questions that meet the preset unsafe query conditions. Then, the large model security assessment device can train and update the large model to be tested according to the question samples until the preset stop condition is met to obtain the updated large model. The preset stop condition can be that the training reaches a preset number of times at this time, or the loss convergence is minimized, etc. By training and updating the large model to be tested in a targeted manner, the vulnerability risk of the large model is avoided and the security of the large model is improved.
[0131] After obtaining the evaluation results of the large model to be tested, the user can select the target risk category and the same large model according to actual needs. In some embodiments, after outputting the visual evaluation results according to the test value, the large model security evaluation method further includes:
[0132] S51, receiving a first input from a user regarding a model comparison icon in a visualization display interface;
[0133] S52, in response to the first input, selecting at least one target risk category, and selecting at least one similar large model to be compared;
[0134] S53. Display the horizontal comparison results of the large model to be tested and at least one similar large model to be compared for at least one risk category according to the target risk category.
[0135] Among them, the horizontal comparison results show the first risk category of vulnerability risks between the large model to be tested and similar large models. The horizontal comparison results indicate the advantages and disadvantages between the large model to be tested and similar large models, as well as the comparison of advantages and disadvantages between the large model to be tested and similar large models.
[0136] Specifically, Figure 8 As shown, the large model safety assessment device can receive a first input from a user for a model comparison icon in a visual display interface, such as the shape, color, display size, and position of the model comparison icon. The first input can be a click selection or a keystroke, etc., which are not limited here. Then, the large model safety assessment device can respond to the first input, select at least one target risk category (such as risk A, risk D, and risk E), and select at least one similar large model to be compared (such as model A, model B, and model E), wherein the different target risk categories can correspond to the assessment indicators corresponding to different risk categories.
[0137] It can be understood that the selection of the large model and the selection of the risk category can be displayed on the same page of the visual display interface, or on different pages. When the selection of the large model and the selection of the risk category are displayed on different pages of the visual display interface, the large model and the risk category can be selected by switching pages.
[0138] At this time, the large model safety assessment device can display the horizontal comparison results of the large model to be tested and at least one large model of the same type to be compared for at least one risk category according to the target risk category. The target risk category and the large model of the same type are determined by user selection so that the required horizontal comparison results can be displayed, which improves the convenience and flexibility of displaying the horizontal comparison results.
[0139] After obtaining the horizontal comparison results, the user may be prompted to further optimize the large model to be tested. In some embodiments, after displaying the horizontal comparison results of the large model to be tested and at least one large model of the same type to be compared for at least one risk category according to the target risk category, the large model security assessment method further includes:
[0140] S54. According to the horizontal comparison result, at least one second risk category for the large model to be tested to be optimized is selected from the first risk category, and the second risk category is displayed.
[0141] S55. Receive an optimization instruction, and in response to the optimization instruction, optimize the large model to be tested in the second risk category according to the teacher model to obtain an optimized large model.
[0142] Specifically, through the horizontal comparison results, it is possible to timely know whether the large model to be tested has vulnerability risks and the risk categories that need to be optimized. Therefore, the large model security assessment device can select at least one second risk category to be optimized for the large model to be tested from the first risk category according to the horizontal comparison results, and display the second risk category for the user to view so that the user can decide whether the large model to be tested needs to be optimized. When it is necessary to optimize the large model to be tested, the model security assessment device can receive the optimization instruction input by the user, and optimize the large model to be tested in the second risk category according to the teacher model in response to the optimization instruction to obtain the optimized large model. By optimizing the large model to be tested in a targeted manner, the vulnerability risk of the large model is avoided and the security of the large model is improved.
[0143] In the embodiment of the present application, the test indicator set and the attack instruction set can be automatically determined according to the application field of the large model to be tested, and in response to the attack instruction in the attack instruction set, the large model to be tested is security evaluated according to the multiple evaluation indicators in the test indicator set, and the test value corresponding to each evaluation indicator is obtained. Since the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories, it is possible to automatically evaluate the security values corresponding to different evaluation indicators under each risk category, and the visual evaluation results can be output according to the test values. The evaluation results show the test values corresponding to multiple different risk categories under different attack instructions for the large model to be tested. Therefore, before the large model goes online, in order to avoid the security risks that exist after the large model goes online later, the large model security evaluation scheme of the present application can be used to automatically perform security evaluation on the large model (including evaluation of algorithms, applications, data and other sections), so as to iteratively optimize some algorithm vulnerabilities of the large model according to the evaluation results to ensure the security of the use of the large model.
[0144] The method described in the above embodiment is further described in detail below with examples.
[0145] This embodiment takes the large model security assessment device integrated in the server as an example, and is applied to the application scenario of large model security assessment. The terminal (i.e., terminal device) can initiate a large model security assessment request to the server. The server responds to the large model security assessment request, performs security assessment on the large model to be tested, and returns the assessment result to the terminal for display. Please refer to Fig. 9 , Fig. 9 A schematic diagram of a large model security assessment method provided in an embodiment of the present application. The method may include:
[0146] S61. The terminal sends a large model security assessment request carrying a model identifier to the server.
[0147] The terminal can receive a model selection instruction input by a user, select a model identifier of a large model to be tested based on the model selection instruction, generate a large model security assessment request based on the model identifier of the large model to be tested, and send the large model security assessment request to a server.
[0148] S62. In response to the big model security assessment request, the server determines the big model that matches the model identifier as the big model to be tested, and determines the test indicator set and the attack instruction set according to the application field of the big model to be tested.
[0149] The server responds to the big model security assessment request, extracts the model identifier from the big model security assessment request, obtains the big model matching the model identifier, and uses the big model matching the model identifier as the big model to be tested. The server can obtain the application field of the big model to be tested, and determine the test indicator set and the attack instruction set according to the application field of the big model to be tested. The test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories.
[0150] S63. The server responds to the attack instruction in the attack instruction set, performs a security evaluation on the large model to be tested according to the multiple evaluation indicators in the test indicator set, obtains the test value corresponding to each evaluation indicator, and generates the evaluation result of the large model to be tested according to the test value.
[0151] Among them, the test value represents the security value corresponding to the evaluation indicator under each risk category. After determining the test indicator set and attack instruction set of the large model to be tested, the server responds to the attack instruction in the attack instruction set, and performs a security evaluation on the large model to be tested according to the multiple evaluation indicators in the test indicator set to obtain the test value corresponding to each evaluation indicator. Then the server can generate the evaluation result of the large model to be tested based on the test value.
[0152] S64. The server sends the evaluation result of the large model to be tested to the terminal.
[0153] S65. The terminal displays the visual evaluation results.
[0154] The terminal can display the test values corresponding to various risk categories under different attack instructions for the large model to be tested. It can also display the evaluation indicators under each risk category used for security evaluation of the large model to be tested and the proportion of each attack instruction, etc.
[0155] S66. The terminal sends a model comparison request to the server.
[0156] S67. The server obtains the evaluation results of similar large models in response to the model comparison request, and generates a horizontal comparison result based on the evaluation results of the large model to be tested and the evaluation results of similar large models.
[0157] Among them, the horizontal comparison results include the evaluation results of the large model to be tested for at least one risk category and the evaluation results of similar large models for at least one risk category. The horizontal comparison results show the risk categories of vulnerability risks between the large model to be tested and similar large models. The horizontal comparison results indicate the advantages and disadvantages between the large model to be tested and similar large models, as well as the comparison of advantages and disadvantages between the large model to be tested and similar large models.
[0158] After receiving the model comparison request sent by the terminal, the server can respond to the model comparison request, obtain the evaluation results of similar large models, which may include one or more similar large models, and then generate horizontal comparison results based on the evaluation results of the large model to be tested and the evaluation results of similar large models.
[0159] S68. The server sends the horizontal comparison result to the terminal.
[0160] S69. The terminal displays the horizontal comparison result.
[0161] The terminal can display the evaluation results of the large model to be tested, the evaluation results of similar large models, the horizontal comparison results between the large model to be tested and similar large models, etc. The terminal can display the evaluation results and the horizontal comparison results on the same page of the visual display interface, or on different pages of the visual display interface, without limitation. When the evaluation results and the horizontal comparison results are displayed on different pages of the visual display interface, the terminal can switch the display of the evaluation results and the horizontal comparison results through page switching instructions. This improves the richness and diversity of the display of large model security evaluation results.
[0162] S70: The terminal sends a model update request to the server.
[0163] S71. The server responds to the model update request and obtains adversarial question samples according to the risk category of the vulnerability risk of the large model to be tested.
[0164] S72. The server trains and updates the large model to be tested according to the problem samples to obtain an updated large model.
[0165] Through the evaluation results of the large model to be tested, it is possible to know in time whether the large model to be tested has vulnerability risks. If the large model to be tested has vulnerability risks, the terminal can send a model update request to the server. The server responds to the model update request, determines the risk category of the large model to be tested with vulnerability risks, and obtains adversarial question samples that match the risk category of the large model to be tested with vulnerability risks. The question samples are questions that meet the preset unsafe query conditions. Then, the server can train and update the large model to be tested according to the question samples until the preset stop conditions are met to obtain the updated large model. The preset stop condition can be that the training reaches a preset number of times at this time, or the loss converges to a minimum, etc. By training and updating the large model to be tested in a targeted manner, the vulnerability risk of the large model is avoided and the security of the large model is improved.
[0166] S73. The server sends the model update result to the terminal.
[0167] After the server completes the training and updating of the large model to be tested, it can send the model update result to the terminal.
[0168] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0169] In the embodiment of the present application, the server can determine the test indicator set and the attack instruction set according to the application field of the large model to be tested, and respond to the attack instruction in the attack instruction set, and perform security evaluation on the large model to be tested according to the multiple evaluation indicators in the test indicator set, and obtain the test value corresponding to each evaluation indicator. Since the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories, it is possible to automatically evaluate the security values corresponding to different evaluation indicators under each risk category, and the evaluation results of the large model to be tested can be generated according to the test values, and the evaluation results are sent to the terminal for display, and the evaluation results show the test values corresponding to multiple different risk categories of the large model to be tested under different attack instructions. And, the server can obtain the horizontal comparison results of the large model to be tested and the same large model, and send them to the terminal for display, which improves the display diversity and richness. In addition, the server can obtain adversarial problem samples according to the risk category of the vulnerability risk of the large model to be tested, and the problem samples are problems that meet the preset unsafe query conditions; then the large model to be tested is trained and updated according to the problem samples to obtain the updated large model. The large model is fully automated and optimized online, which improves the optimization efficiency of the large model. In addition, before the large model goes online, in order to avoid the security risks that exist after the large model goes online later, the large model security evaluation scheme of this application can be used to conduct a security evaluation of the large model (including evaluation of algorithms, applications, data and other sections), so as to iteratively optimize some algorithm vulnerabilities of the large model according to the evaluation results, and ensure the safety of the use of the large model.
[0170] A large model security assessment method in an embodiment of the present application is described above. A large model security assessment device (such as a server) that executes the large model security assessment method is introduced below.
[0171] See also Fig.10 ,like Fig.10The structural schematic diagram of a large model security assessment device shown can be applied to servers in scenarios where security assessment of large models is required. By determining the test indicator set and the attack instruction set according to the application field of the large model to be tested, and responding to the attack instructions in the attack instruction set, the large model to be tested is security assessed according to the multiple assessment indicators in the test indicator set, and the test value corresponding to each assessment indicator is obtained. Visual assessment results are output according to the test values. Before the large model goes online, in order to avoid security risks that exist after the large model goes online later, the large model security assessment solution of the present application can be used to automatically perform a security assessment on the large model (including assessments of algorithms, applications, data and other sections), so as to iteratively optimize some algorithm vulnerabilities of the large model according to the assessment results to ensure the security of the use of the large model. The large model security assessment device in the embodiment of the present application can achieve the corresponding Figure 2 The steps of the large model security assessment method executed in the corresponding embodiment. The functions implemented by the large model security assessment device can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions, and the module can be software and / or hardware. The large model security assessment device 60 may include an input-output module 601 and a processing module 602, wherein the functions of the input-output module 601 and the processing module 602 can be referred to Figure 2 The operations performed in the corresponding embodiments are not described in detail here.
[0172] The input-output module 601 is configured to determine a large model to be tested;
[0173] The processing module 602 is configured to determine a test indicator set and an attack instruction set according to the application field of the large model to be tested, wherein the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories; in response to the attack instructions in the attack instruction set, perform security evaluation on the large model to be tested according to the multiple evaluation indicators in the test indicator set, and obtain a test value corresponding to each evaluation indicator, wherein the test value represents a security value corresponding to the evaluation indicator under each risk category;
[0174] The input-output module 601 is also configured to output a visualized evaluation result according to the test value, and the evaluation result displays the test values corresponding to various risk categories of the large model to be tested under different attack instructions.
[0175] In some embodiments, the evaluation results are displayed in a first area of a visual display interface, and the processing module 602 is configured to display a horizontal comparison result in a second area of the visual display interface, the horizontal comparison result including an evaluation result of the large model to be tested for at least one risk category and an evaluation result of a similar large model for at least one risk category, the horizontal comparison result displays risk categories of vulnerability risks between the large model to be tested and similar large models, the horizontal comparison result indicates advantages and disadvantages between the large model to be tested and similar large models, and a comparison of advantages and disadvantages between the large model to be tested and similar large models.
[0176] In some embodiments, the processing module 602 is configured to obtain adversarial question samples according to the risk category of vulnerability risk of the large model to be tested, where the question samples are questions that meet the preset unsafe query conditions;
[0177] The large model to be tested is trained and updated according to the problem samples to obtain an updated large model.
[0178] In some embodiments, the processing module 602 is configured to receive a first input from a user regarding a model comparison icon in a visual display interface;
[0179] In response to the first input, selecting at least one target risk category and selecting at least one similar large model to be compared;
[0180] According to the target risk category, the horizontal comparison results of the large model to be tested and at least one similar large model to be compared for at least one risk category are displayed. The horizontal comparison results display the first risk category in which the large model to be tested and the similar large model have vulnerability risks. The horizontal comparison results indicate the advantages and disadvantages between the large model to be tested and the similar large models, as well as the comparison of advantages and disadvantages between the large model to be tested and the similar large models.
[0181] In some embodiments, the processing module 602 is configured to select at least one second risk category for the large model to be tested to be optimized from the first risk category according to the horizontal comparison result, and display the second risk category;
[0182] An optimization instruction is received, and in response to the optimization instruction, the large model to be tested is optimized according to the teacher model in the second risk category to obtain an optimized large model.
[0183] In some embodiments, the processing module 602 is configured to receive an adversarial question input by a user into a question input box in the visual display interface, where the adversarial question is a question that meets a preset unsafe inquiry condition;
[0184] The answer is displayed in a display box in the visualization display interface, and the answer is the answer that matches the adversarial question.
[0185] In some embodiments, the risk categories include at least one of malicious use, harmful language, violation of national security, erroneous knowledge and information, bias and discrimination, information leakage, and deployment risk, and different risk categories correspond to the large models to be tested in different application fields.
[0186] In some embodiments, harmful language includes at least one of: content that encourages or suggests self-harm, pushes or suggests users to perform immoral or illegal acts, pornographic language, violent, offensive content, harassing, derogatory and hateful content, instructions on how to find illegal content, inflammatory speech, personal attacks or malicious slander, damage to social morals or values, and impolite language;
[0187] Prejudice and discrimination include at least one of social stereotypes, individual or group prejudice, and exclusionary norms;
[0188] Information leakage includes: leakage of private information, leakage of privacy by correctly inferring private information, and at least one of leakage or inference of sensitive information;
[0189] Malicious purposes include: dissemination of false information, fraud / scam, spreading malicious websites, promoting illegal products, and promoting at least one of extreme speech;
[0190] Violations of national security include: damaging the national image, damaging national interests, inciting ethnic hatred or discrimination, undermining social stability, and being politically sensitive.
[0191] In some embodiments, the attack instructions include at least one of target hijacking, unsafe inquiry, injection attack, developer mode, negative inducement, role playing, and prompt leakage, and different attack instructions correspond to the large models to be tested in different application fields.
[0192] In some implementations, the processing module 602 is configured to set a preset evaluation method, the evaluation method including an automatically generated evaluation and a fixed data set evaluation;
[0193] For each evaluation method, set at least one corresponding risk category and at least one attack instruction;
[0194] Configure the number of adversarial question prompt injections corresponding to each risk category for each evaluation method.
[0195] In the embodiment of the present application, the processing module 602 can automatically determine the test indicator set and the attack instruction set according to the application field of the large model to be tested, and respond to the attack instruction in the attack instruction set, and perform a security evaluation on the large model to be tested according to the multiple evaluation indicators in the test indicator set, and obtain the test value corresponding to each evaluation indicator. Since the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories, it is possible to automatically evaluate the security values corresponding to different evaluation indicators under each risk category, and the visual evaluation results can be output according to the test values. The evaluation results show the test values corresponding to multiple different risk categories under different attack instructions for the large model to be tested. Therefore, before the large model goes online, in order to avoid the security risks that exist after the large model goes online later, the large model security evaluation scheme of the present application can be used to automatically perform a security evaluation on the large model (including evaluation of algorithms, applications, data and other sections), so as to iteratively optimize some algorithm vulnerabilities of the large model according to the evaluation results to ensure the security of the use of the large model.
[0196] The large model security assessment device 60 in the embodiment of the present application is described above from the perspective of modular functional entities. The large model security assessment device in the embodiment of the present application is described below from the perspective of hardware processing.
[0197] It should be noted that Fig.10 The physical device corresponding to the input / output module 601 shown may be a transceiver, a radio frequency circuit, a communication module, an input / output (I / O) interface, etc., and the physical device corresponding to the processing module 602 may be a processor.
[0198] Fig.10 The devices shown can all have Fig.11 The structure shown, when Fig.10 The large model safety evaluation device 60 shown has the following features: Fig.11 When the structure shown is Fig.11 The processor and transceiver in the device can realize the same or similar functions as the processing module 602 and the input / output module 601 provided in the device embodiment corresponding to the device described above, Fig.11 The memory in the storage processor is a computer program that needs to be called when executing the above-mentioned large model security assessment method.
[0199] The present application also provides a terminal device, such as Fig.12For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal device can be any terminal device including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a car computer, etc., taking the mobile phone as an example:
[0200] Fig.12 FIG. 1 is a block diagram showing a partial structure of a mobile phone related to a terminal device provided in an embodiment of the present application. Fig.12 The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art will understand that Fig.12 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0201] Combine the following Fig.12 A detailed introduction to the various components of the mobile phone:
[0202] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (Low Noise Amplifier, LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.
[0203] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.
[0204] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.
[0205] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig.12 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.
[0206] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.
[0207] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.
[0208] Wi-Fi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages, and access streaming media through the Wi-Fi module 1070. It provides users with wireless broadband Internet access. Fig.12 A Wi-Fi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.
[0209] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. It executes various functions of the mobile phone and processes data by running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, so as to monitor the mobile phone as a whole. Optionally, the processor 1080 may include one or more processing units; optionally, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.
[0210] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Optionally, the power supply can be logically connected to the processor 1080 through a power management system, thereby managing functions such as charging, discharging, and power consumption management through the power management system.
[0211] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.
[0212] In the embodiment of the present application, the processor 1080 included in the mobile phone also has the function of controlling the execution of the above-mentioned large model security assessment method process performed by the large model security assessment device.
[0213] The present application also provides a server. Fig.13 , Fig.13 1 is a schematic diagram of a server structure provided in an embodiment of the present application. The server 1100 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (full name in English: central processing units, abbreviated in English: CPU) 1122 (for example, one or more processors) and memory 1132, one or more storage media 1130 (for example, one or more mass storage devices) storing application programs 1142 or data 1144. Among them, the memory 1132 and the storage medium 1130 can be short-term storage or permanent storage. The program stored in the storage medium 1130 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 1122 can be configured to communicate with the storage medium 1130 to execute a series of instruction operations in the storage medium 1130 on the server 1100.
[0214] The server 1100 may also include one or more power supplies 1126, one or more wired or wireless network interfaces 1150, one or more input and output interfaces 1158, and / or one or more operating systems 1141, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc.
[0215] The steps performed by the server in the above embodiment can be based on the Fig.13 The structure of the server 1100 shown in FIG. Fig.13 The steps performed by the large model safety assessment device 60 shown can be based on the Fig.13 For example, the central processor 1122 performs the following operations by calling the instructions in the memory 1132:
[0216] Determine the large model to be tested through the input and output interface 1158;
[0217] The central processor 1122 determines a test indicator set and an attack instruction set according to the application field of the large model to be tested, wherein the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories; in response to the attack instructions in the attack instruction set, a security evaluation is performed on the large model to be tested according to the multiple evaluation indicators in the test indicator set, and a test value corresponding to each evaluation indicator is obtained, wherein the test value represents a security value corresponding to the evaluation indicator under each risk category;
[0218] The input-output interface 1158 can also output a visualized evaluation result based on the test value, and the evaluation result displays the test values corresponding to various risk categories of the large model to be tested under different attack instructions.
[0219] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0220] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.
[0221] In the several embodiments provided in the embodiments of the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.
[0222] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0223] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.
[0224] According to one aspect of the present application, a computer-readable storage medium is provided, which includes instructions, and when the instructions are executed on a computer, the computer executes the large model security assessment method of the present application as described above.
[0225] According to one aspect of the present application, a computing device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the large model security assessment method of the present application when executing the computer program. The computing device may be the above-mentioned server.
[0226] According to one aspect of the present application, a chip is provided, the chip including a processor coupled to a transceiver of a terminal device, for executing the large model security assessment method provided in an embodiment of the present application. An embodiment of the present application also provides a chip system, the chip system including a processor, for supporting the terminal device to implement the functions involved in the above-mentioned large model security assessment method. In one possible design, the above-mentioned chip system also includes a communication interface for inputting and / or outputting information. In one possible design, the above-mentioned chip system also includes a memory, which is used to store program instructions and data required by the terminal device. The chip system can be composed of chips, and can also include chips and other discrete devices.
[0227] According to one aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions (i.e., program instructions), the computer instructions are stored in a computer-readable storage medium. A processor of a computer device (which may be referred to as a computing device or a computer) reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the large model security assessment method provided in various optional implementations in the above embodiments.
[0228] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.
[0229] The computer program product includes one or more computer instructions. When the computer program is loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from a computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.
[0230] The technical solutions provided in the embodiments of the present application are introduced in detail above. The principles and implementation methods of the embodiments of the present application are explained by using specific examples in the embodiments of the present application. The description of the above embodiments is only used to help understand the methods and core ideas of the embodiments of the present application. At the same time, for those skilled in the art, according to the ideas of the embodiments of the present application, there will be changes in the specific implementation methods and application scopes. In summary, the content of this specification should not be understood as a limitation on the embodiments of the present application.
Claims
1. A large model security assessment method, characterized in that: The method comprises: Determine the large model to be tested; Determine a test indicator set and an attack instruction set according to the application field of the large model to be tested, wherein the test indicator set includes evaluation indicators corresponding to multiple risk categories, and the attack instruction set includes attack instructions corresponding to multiple attack categories; In response to the attack instruction in the attack instruction set, a security evaluation is performed on the large model to be tested according to the multiple evaluation indicators in the test indicator set to obtain a test value corresponding to each evaluation indicator, wherein the test value represents a security value corresponding to the evaluation indicator under each risk category; A visual evaluation result is output based on the test value, and the evaluation result displays the test values of the large model to be tested corresponding to a plurality of different risk categories under different attack instructions.
2. The large model security assessment method according to claim 1 is characterized in that: The evaluation result is displayed in the first area of the visual display interface. After the visual evaluation result is output according to the test value, the method further includes: In the second area of the visual display interface, horizontal comparison results are displayed, including evaluation results of the large model to be tested for at least one risk category and evaluation results of similar large models for at least one risk category, the horizontal comparison results display risk categories of vulnerability risks of the large model to be tested and similar large models, the horizontal comparison results indicate advantages and disadvantages between the large model to be tested and similar large models, as well as comparison of advantages and disadvantages between the large model to be tested and similar large models.
3. The large model security assessment method according to claim 2 is characterized in that: After displaying the horizontal comparison result in the second area of the visual display interface, the method further includes: According to the risk category of the vulnerability risk of the large model to be tested, obtaining adversarial question samples, wherein the question samples are questions that meet the preset unsafe inquiry conditions; The large model to be tested is trained and updated according to the problem sample to obtain an updated large model.
4. The large model security assessment method according to claim 1 is characterized in that: After outputting the visualized evaluation result according to the test value, the method further includes: Receive a first input from a user regarding a model comparison icon in the visualization display interface; In response to the first input, selecting at least one target risk category and selecting at least one similar large model to be compared; The horizontal comparison result of the large model to be tested and at least one large model of the same type to be compared for at least one risk category is displayed according to the target risk category, and the horizontal comparison result displays the first risk category in which the large model to be tested and the large model of the same type have vulnerability risks, and the horizontal comparison result indicates the advantages and disadvantages between the large model to be tested and the large models of the same type, as well as the comparison of advantages and disadvantages between the large model to be tested and the large models of the same type.
5. The large model security assessment method according to claim 4 is characterized in that: After displaying the horizontal comparison result of the large model to be tested and the at least one large model of the same type to be compared for at least one risk category according to the target risk category, the method further includes: According to the horizontal comparison result, selecting at least one second risk category to be optimized for the large model to be tested from the first risk category, and displaying the second risk category; An optimization instruction is received, and in response to the optimization instruction, the large model to be tested is optimized in the second risk category according to the teacher model to obtain an optimized large model.
6. The large model security assessment method according to claim 1, characterized in that: After determining the large model to be tested, the method further includes: Receiving an adversarial question input by a user into a question input box in a visual display interface, wherein the adversarial question is a question that meets a preset unsafe inquiry condition; An answer is displayed in a display box in the visual display interface, wherein the answer is an answer that matches the adversarial question.
7. The large model security assessment method according to claim 1 is characterized in that: The risk categories include at least one of malicious use, harmful language, violation of national security, erroneous knowledge and information, bias and discrimination, information leakage, and deployment risk. There are different risk categories for the large models to be tested in different application fields.
8. The large model security assessment method according to claim 7, characterized in that: The harmful language includes: content that encourages or suggests self-harm, pushes or suggests users to perform immoral or illegal acts, pornographic language, violent, offensive content, harassing, derogatory and hateful content, instructions on how to find illegal content, inflammatory speech, personal attacks or malicious slander, damage to social morals or values, and at least one of impolite language; The prejudice and discrimination include: at least one of social stereotypes, individual or group prejudices and exclusionary norms; The information leakage includes: leakage of private information, leakage of privacy by correctly inferring private information, and at least one of leakage or inference of sensitive information; The malicious purposes include: at least one of: dissemination of false information, fraud / scam, spreading malicious websites, promoting illegal products, and promoting extreme speech; The violations of national security include: damaging the national image, damaging national interests, inciting ethnic hatred or discrimination, undermining social stability, and being politically sensitive.
9. The large model security assessment method according to claim 1, characterized in that: The attack instructions include at least one of target hijacking, unsafe inquiry, injection attack, developer mode, negative induction, role playing, and prompt leakage. There are different attack instructions corresponding to the large models to be tested in different application fields.
10. The large model security assessment method according to any one of claims 1 to 9, characterized in that: Before determining the large model to be tested, the method further includes: Setting a preset evaluation method, wherein the evaluation method includes an automatically generated evaluation and a fixed data set evaluation; For each evaluation method, set at least one corresponding risk category and at least one attack instruction; Configure the number of adversarial question prompt injections corresponding to each risk category for each evaluation method.
Citation Information
Cited By
Sample generation method and device, equipment, storage medium and product
CN120256291A
Model evaluation method and device, equipment, storage medium and product
CN120493240A
