Large model safety performance test method and device, equipment and storage medium
By conducting security performance testing on the big model and using the analysis and judgment model to evaluate the output answers of malicious problems, the subjectivity and uncertainty problems caused by manual analysis and judgment are solved, and efficient and accurate evaluation of the security performance of the big model is achieved.
Patent Information
- Application Number
- CN202411822470.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-11
- Publication Date
- 2025-05-16
AI Technical Summary
The security evaluation of large-model output content in the prior art relies on manual analysis and judgment, resulting in subjectivity and uncertainty in the evaluation results, and lack of efficient, accurate and autonomous and controllable risk identification methods.
By obtaining the large model to be tested, and generating a question-and-answer pair based on the malicious questions obtained, inputting the trained analysis and judgment model for testing score calculation, obtaining the performance score and determining the safety performance level based on the threshold.
The accuracy and universality of large-scale model safety performance testing is achieved, reducing the cost and subjective impact of manual analysis and judgment.
Smart Images
Figure CN120012080A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the fields of artificial intelligence and information security, and in particular to a large-scale model security performance testing method, device, equipment and storage medium. Background Art
[0002] In the field of artificial intelligence technology, the security alignment evaluation of large models is a key link to ensure the security and reliability of the output content of the AI system. With the rapid development of AI technology, large models play an important role in various application scenarios, but the security of the content they generate faces severe challenges.
[0003] At present, the traditional method for safety assessment of large model output content mainly relies on manual research and judgment. This method not only requires a lot of manpower and time resources, but also is affected by personal experience and cognitive differences, making it difficult to unify the research and judgment standards, resulting in subjectivity and uncertainty in the evaluation results.
[0004] Therefore, how to build an efficient, accurate, and autonomously controllable large-model output content risk identification method has become a technical problem that needs to be solved urgently. Summary of the invention
[0005] The present application provides a large-model safety performance testing method, apparatus, equipment and storage medium to solve the technical problem of how to improve the accuracy and universality of large-model safety performance testing.
[0006] In a first aspect, the present application provides a large model safety performance testing method, comprising:
[0007] Get the large model to be tested;
[0008] Generate a question-answer pair to be tested based on the collected malicious questions and the large model to be tested;
[0009] Input the question-answer pairs to be tested into the trained judgment model to obtain the test scores of all the question-answer pairs to be tested, and use the average of the test scores as the performance score of the large model to be tested;
[0010] Determining the safety performance level of the large model to be tested according to the performance score, the first threshold and the second threshold;
[0011] Among them, the first threshold and the second threshold are determined according to the trained analysis and judgment model and the training data set; the first threshold is greater than the second threshold.
[0012] In a second aspect, the present application provides a large model safety performance testing device, comprising:
[0013] An acquisition module is used to acquire the large model to be tested; and generate a question-answer pair to be tested according to the collected malicious questions and the large model to be tested;
[0014] A testing module is used to input the question-answer pairs to be tested into the trained analysis and judgment model to obtain the test scores of all the question-answer pairs to be tested, and use the average of the test scores as the performance score of the large model to be tested; determine the safety performance level of the large model to be tested according to the performance score, a first threshold and a second threshold; wherein the first threshold and the second threshold are determined according to the trained analysis and judgment model and the training data set; and the first threshold is greater than the second threshold.
[0015] In a third aspect, an embodiment of the present application provides an electronic device, including: a memory, a processor;
[0016] The memory stores computer-executable instructions;
[0017] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.
[0018] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.
[0019] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above first aspect and / or various possible implementation methods of the first aspect.
[0020] The large model security performance testing method, device, equipment and storage medium provided in the present application obtain a large model to be tested that needs to be tested for security performance; input the collected malicious questions into the large model to be tested, and the large model to be tested can output corresponding output answers according to the input malicious questions; form a question-answer pair to be tested according to the input malicious questions and the output answers; input the question-answer pair to be tested into a trained analysis and judgment model to obtain a test score for each question-answer pair to be tested; calculate the average score of all malicious questions to obtain a performance score of the large model to be tested; compare the performance score with the first threshold and the second threshold to determine the means of the security performance level of the large model to be tested, thereby achieving the effect of improving the accuracy and universality of large model security performance testing. BRIEF DESCRIPTION OF THE DRAWINGS
[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.
[0022] Figure 1A schematic diagram of the process of the large model safety performance testing method provided for this application;
[0023] Figure 2 An example of obtaining the first threshold value provided in this application;
[0024] Figure 3 A schematic diagram of the structure of the large-scale model safety performance test device provided for this application;
[0025] Figure 4 A schematic diagram of the structure of the electronic device provided in this application.
[0026] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION
[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.
[0028] It should be noted that the large-model safety performance testing method, device, equipment and storage medium provided in this application can be used in the field of artificial intelligence, and can also be used in any field other than artificial intelligence. The application field of the large-model safety performance testing method, device, equipment and storage medium in this application is not limited.
[0029] The specific application scenario of this application is the security alignment evaluation scenario of the large model. Before the large model is used, it is usually necessary to review the security of the output data of the large model to ensure the security and compliance of the output data. However, in the face of different model bases, model versions, model configurations, etc., the content output by the large model is different. Moreover, the security of the output content finally generated by the large model is unknown. Therefore, it is usually necessary to evaluate the security of the large model before the large model is put into use. At present, the traditional approach is to manually analyze and mark the content returned by the large model to achieve the evaluation of the large model. And, based on the labels marked by manual analysis and analysis, an evaluation report for the evaluation of the large model is further generated. However, manual analysis and analysis usually consumes a lot of manpower and time costs, and also due to personal cognitive differences, there will be differences in the analysis and analysis standards, which will lead to inaccurate evaluation results. Therefore, building a dedicated analysis and analysis model has become a problem to be solved.
[0030] The large model safety performance testing method provided in this application aims to achieve quantitative evaluation of the safety performance of large models, so as to reduce the manpower and time cost consumption caused by manual research and judgment, reduce the impact of inconsistent results caused by cognitive differences, and reduce the commercial cost of third-party API construction. In addition, the large model safety performance testing method can achieve the safety alignment of multiple models through an automated process, and is not limited to the processing of a large model.
[0031] Specifically, the execution of the large model safety performance testing method of the present application can be divided into four stages. Among them, the first stage can be the collection stage of safety aligned question and answer pairs. The second stage can be the training stage of the judgment model based on the safety aligned question and answer pairs. The third stage can be the threshold statistics stage based on the trained judgment model. This stage can generate the first threshold and the second threshold. The fourth stage is the performance test stage of the large model to be tested based on the trained judgment model, the first threshold and the second threshold.
[0032] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.
[0033] Figure 1 A flow chart of the large model safety performance testing method provided for this application, such as Figure 1 As shown, with the electronic device as the execution subject, the method includes:
[0034] S101, obtaining a large model to be tested.
[0035] In this embodiment, the electronic device can obtain a large model to be tested that needs to be tested for safety performance. Optionally, the electronic device can obtain an access interface of the large model to be tested. Based on the access interface, the electronic device can input a question and obtain an output result of the question.
[0036] S102. Generate question-answer pairs to be tested based on the collected malicious questions and the large model to be tested.
[0037] In this embodiment, the electronic device can obtain malicious questions that have been collected. Optionally, the malicious question can be a malicious question collected for the application scenario. Optionally, when the malicious question is a malicious question collected for the application scenario, the analysis and judgment model trained using the malicious question is only used to implement the model security performance test in the application scenario. Optionally, the malicious question can be a universal malicious question. Then the analysis and judgment model trained using the malicious question also has universal security performance testing capabilities. Optionally, the number of malicious questions can be as many as possible, and the coverage can be as comprehensive as possible, so as to ensure the effectiveness of the test.
[0038] Optionally, the malicious question is the malicious question used in training data.
[0039] Specifically, the electronic device can input the collected malicious questions into the large model to be tested, and the large model to be tested can output the corresponding output answer according to the input malicious question. The electronic device can form a question-answer pair to be tested according to the input malicious question and the output answer. That is, for each malicious question, a question-answer pair to be tested of the large model to be tested can be obtained.
[0040] S103. Input the question-answer pairs to be tested into the trained analysis and judgment model to obtain the test scores of all the question-answer pairs to be tested, and use the average of the test scores as the performance score of the large model to be tested.
[0041] In this embodiment, the electronic device can input the question and answer pair to be tested into the trained analysis and judgment model. The analysis and judgment model can score each question and answer pair to be tested to obtain a test score for each question and answer pair to be tested. Optionally, since in the training data of the analysis and judgment model, the positive sample is the positive answer to the malicious question, and the negative sample is the negative answer to the malicious question. Therefore, based on the analysis and judgment model, the output test score is the malicious degree of the output answer answered by the large model to be tested based on the malicious question. If the malicious degree of the output answer is high and closer to the negative answer, the test score is closer to 0. If the maliciousness of the output answer is low and closer to the positive answer, the test score is closer to 1. The electronic device can also calculate the average score of all malicious questions to obtain the performance score of the large model to be tested.
[0042] S104, determining the safety performance level of the large model to be tested according to the performance score, the first threshold and the second threshold. The first threshold and the second threshold are determined according to the trained judgment model and the training data set. The first threshold is greater than the second threshold.
[0043] In this embodiment, the electronic device may store a first threshold and a second threshold generated for the analysis and judgment model and the training data. The first threshold is greater than the second threshold. Specifically, the first threshold is a threshold generated based on the analysis and judgment model and the positive samples in the training data. When the performance score is greater than the first threshold, it means that the output answer of the large model to be tested is closer to the positive answer. That is, the safety performance of the large model to be tested is good. The second threshold is a threshold generated based on the analysis and judgment model and the negative samples in the training data. When the performance score is less than the second threshold, it means that the output answer of the large model to be tested is closer to the negative answer. That is, the safety performance of the large model to be tested is poor.
[0044] The electronic device can compare the performance score with the first threshold and the second threshold to determine the safety performance level of the large model to be tested. Specifically, the electronic device can divide the safety performance level into three levels: first level, second level, and third level according to the first threshold and the second threshold. The setting of these three levels can better realize the quantification of the safety performance of the large model to be tested, thereby improving the evaluation effect of the safety performance. Among them, the safety performance of the first level is better than that of the second level, and the safety performance of the second level is better than that of the third level. The process of the electronic device dividing the safety performance level corresponding to the performance score can specifically include:
[0045] First, if the performance score is greater than a first threshold, it is determined that the security performance of the large model to be tested is at a first level. Optionally, the first level is used to indicate that the security performance of the large model to be tested is excellent. Most of the output answers of the large model to be tested for malicious questions tend to be positive answers.
[0046] Secondly, if the performance score is less than or equal to the first threshold and greater than the second threshold, it is determined that the security performance of the large model to be tested is at the second level. Optionally, the second level is used to indicate that the security performance of the large model to be tested is average. The number of positive answers and negative answers to the large model to be tested is half and half. Optionally, for this situation, the electronic device can further classify the malicious questions, and determine the handling of various types of malicious questions by the large model to be tested based on the test scores of different types of malicious questions, thereby assisting the large model in targeted optimization.
[0047] Third, if the performance score is less than or equal to the second threshold, it is determined that the security performance of the large model to be tested is at the third level. Optionally, the third level is used to indicate that the security performance of the large model to be tested is poor. Most of the output answers of the large model to be tested for malicious questions tend to be negative answers.
[0048] The large model security performance testing method provided in this embodiment can obtain a large model to be tested that needs to be tested for security performance. The electronic device can input the collected malicious questions into the large model to be tested, and the large model to be tested can output the corresponding output answer according to the input malicious question. The electronic device can form a question-answer pair to be tested based on the input malicious question and the output answer. The electronic device can input the question-answer pair to be tested into a trained analysis model. The analysis model can score each question-answer pair to be tested to obtain a test score for each question-answer pair to be tested. The electronic device can also calculate the average score of all malicious questions to obtain the performance score of the large model to be tested. The electronic device can compare the performance score with the first threshold and the second threshold to determine the security performance level of the large model to be tested. In this application, by inputting malicious questions and scoring the question-answer pairs and calculating the average score according to the analysis model, the accuracy and universality of the large model security performance test can be improved.
[0049] Based on the above embodiment, the process of the electronic device determining the first threshold and the second threshold according to the trained judgment model and the training data set may include the following steps:
[0050] Step 1: Input the positive sample question-answer pairs in the training data set into the judgment model to obtain the test scores of the positive sample question-answer pairs.
[0051] In this step, the electronic device can input the question-answer pair corresponding to the positive sample in the training data into the trained judgment model. Among them, the question-answer pair of the positive sample can be composed of malicious questions and positive answers. The judgment model can process each positive sample to obtain a test score corresponding to each positive sample.
[0052] Step 2: Sort the test scores of the positive sample question-answer pairs in descending order to obtain the first test score sequence of the sorted positive sample question-answer pairs.
[0053] In this step, the electronic device may arrange the test scores of each positive sample in order from high to low to obtain a first test score sequence. The higher the score, the closer the positive answer in the positive sample is to the standard answer. Optionally, the positive sample may be a positive sample that is not used for model training.
[0054] Step 3: According to a preset ratio, the test score at the position corresponding to the preset ratio in the first test score sequence is used as the first threshold.
[0055] In this step, a preset ratio may be stored in the electronic device. The electronic device may obtain the first threshold value according to the preset ratio. For example, the preset ratio may be 80%. That is, the electronic device may obtain the test score at the 80% position as the first threshold value. For example, when 100 test scores are included, the electronic device may obtain the 80th test score of the first test queue as the first threshold value. For example, Figure 2 As shown, the queue direction can be shown by an arrow, and the first threshold is the test score at the position shown by the black vertical line.
[0056] Step 4: Input the negative sample question-answer pairs in the training data set into the judgment model to obtain the test scores of the negative sample question-answer pairs.
[0057] In this step, the electronic device can input the question-answer pairs corresponding to the negative samples in the training data into the trained judgment model. The question-answer pairs of the negative samples can be composed of malicious questions and negative answers. The judgment model can process each negative sample to obtain a test score corresponding to each negative sample.
[0058] Step 6: Sort the test scores of the negative sample question-answer pairs in order from low to high to obtain a second test score sequence of the sorted negative sample question-answer pairs.
[0059] In this step, the electronic device may arrange the test scores of each negative sample in order from bottom to top to obtain a second test score sequence. The lower the score, the closer the negative answer in the negative sample is to the standard answer. Optionally, the negative sample may be a negative sample that is not used for model training.
[0060] Step 7: According to the preset ratio, the test score at the position corresponding to the preset ratio in the second test score sequence is used as the second threshold.
[0061] In this step, a preset ratio may be stored in the electronic device. The electronic device may obtain the second threshold value according to the preset ratio. For example, the preset ratio may be 80%. That is, the electronic device may obtain the test score at the 80% position as the second threshold value. For example, when 100 test scores are included, the electronic device may obtain the 80th test score of the second test queue as the second threshold value.
[0062] In the present application, by setting the first threshold and the second threshold, quantitative judgment of positive answers and negative answers can be achieved, thereby improving the accuracy and universality of large-model safety performance testing.
[0063] Based on the above embodiment, the training process of the judgment model includes:
[0064] S201. Obtain a training data set, where each training data in the training data set includes a positive sample question-answer pair consisting of a malicious question and a positive answer, and a negative sample question-answer pair consisting of a malicious question and a negative answer.
[0065] In this embodiment, the electronic device can obtain a training data set. The training data set may include a large amount of training data. Each training data may include a positive sample and a negative sample. Among them, the positive sample is a question-answer pair consisting of a malicious question and a positive answer. The negative sample is a question-answer pair consisting of a malicious question and a negative answer. Optionally, the positive sample and the negative sample corresponding to a training data correspond to the same malicious question. For example, the malicious question is "How to steal personal information from someone else's mobile phone?". For this malicious question, the corresponding positive answer may be "I will not provide any advice or guidance on illegal activities because these activities are illegal and may result in serious legal consequences. Stealing personal information is a criminal act...". For this malicious question, the corresponding negative answer may be "To steal information from someone else's mobile phone, hackers must first access the target phone's Wi-Fi or data connection. Once they are connected to the network, they will be able to view all information related to the user's account, such as emails, text messages, and social media posts...".
[0066] Optionally, when a large number of malicious questions are collected, the electronic device can collect positive answers and negative answers corresponding to the malicious questions at the same time. The electronic device can form a positive sample question-answer pair with the malicious question and a negative sample question-answer pair with the malicious question and a negative sample question-answer pair. The electronic device can form a positive sample question-answer pair and a negative sample question-answer pair corresponding to a malicious question into a training data. The setting of the training data can facilitate the model to increase the gap between the prediction results of the positive sample and the negative sample by comparing the positive sample and the negative sample, thereby improving the prediction accuracy of the model.
[0067] S202, input the training data set into the analysis and judgment model for training, and in each iteration of the analysis and judgment model, adjust the model parameters of the analysis and judgment model according to a preset loss function.
[0068] In this step, the electronic device can input the training data in the training data set into the judgment model to realize the training of the judgment model. The electronic device can input the positive sample question and answer pair and the negative sample question and answer pair in a training sample into the judgment model respectively to obtain the test score of the positive sample question and answer pair and the test score of the negative sample question and answer pair. The electronic device can input the test scores of all positive sample question and answer pairs and the test scores of negative sample question and answer pairs into the loss function according to each iteration process, calculate the model loss, and adjust the model parameters of the judgment model according to the model loss.
[0069] Optionally, the judgment model may include a feature extraction module and a mapping module. When the electronic device uses the judgment model to process positive sample question-answer pairs or negative sample question-answer pairs in the training data, the processing may include the following two steps:
[0070] Step 1: Input the positive sample question-answer pair or the negative sample question-answer pair in the training data set into the feature extraction module of the judgment model to obtain the feature vector.
[0071] Step 2: Input the feature vector into the mapping module of the judgment model to obtain the test score of the positive sample question and answer pair or the negative sample question and answer pair.
[0072] The setting of these two steps can make the output data a more quantitative score, thereby improving the quantitative effect of the analysis and judgment model in the processing process and improving the effectiveness of model processing.
[0073] Optionally, the feature extraction module of the judgment model can be fine-tuned on the basis of an existing open source large model. The mapping module can be a multi-layer perceptron added to the output end based on the feature extraction module. The mapping module can be used to map the output feature vector into a score. Specifically, assuming that hidden_states is the last hidden state of the open source large model, the mapping module can obtain a parameter by taking the hidden state of the last token in the sequence, or by weighted averaging the hidden states of all tokens in the sequence. The mapping module can input the parameter into the multi-layer perceptron. The final activation function of the multi-layer perceptron uses sigmoid, which limits the score to [0,1].
[0074] Optionally, the model optimization process may include:
[0075] Step 1: The electronic device determines a first parameter according to a difference between a test score of a positive sample question-answer pair and a test score of a negative sample question-answer pair of each training data.
[0076] In this step, the test score of the positive sample question-answer pair can be recorded as sp = rθ(x, yp), and the test score of the negative sample question-answer pair can be recorded as sn = rθ(x, yn). Among them, rθ is the judgment model to be trained, θ is the model parameter, x is a malicious question, yp is a positive answer, and yn is a negative answer. The first parameter can be recorded as sp-sn.
[0077] Step 2: The electronic device calculates the second parameter according to the first parameter of each training data in the training data set through a preset loss function.
[0078] In this step, the calculation formula of the preset loss function can be:
[0079] loss = -log(σ(sp-sn))
[0080] Among them, sp-sn is the first parameter in step 1. The training goal of the judgment model is to find the parameters that minimize the loss, that is, to maximize the score difference between the positive response and the negative response.
[0081] Step 3: The electronic device reversely optimizes the model parameters of the analysis model according to the second parameter.
[0082] The large-model safety performance testing method provided in this embodiment achieves the effect of maximizing the score difference between positive responses and negative responses by using positive sample question-answer pairs and negative sample question-answer pairs to train the analysis and judgment model.
[0083] Figure 3 The schematic diagram of the structure of the large model safety performance test device provided in this application is as follows: Figure 3 As shown, the large model safety performance testing device 300 provided in this embodiment includes:
[0084] The acquisition module 301 is used to acquire the large model to be tested and generate a question-answer pair to be tested based on the collected malicious questions and the large model to be tested.
[0085] The test module 302 is used to input the question-answer pairs to be tested into the trained judgment model, obtain the test scores of all the question-answer pairs to be tested, and use the average of the test scores as the performance score of the large model to be tested. According to the performance score, the first threshold and the second threshold, the safety performance level of the large model to be tested is determined. The first threshold and the second threshold are determined according to the trained judgment model and the training data set. The first threshold is greater than the second threshold.
[0086] Optionally, the testing module 302 is used to:
[0087] If the performance score is greater than the first threshold, it is determined that the safety performance of the large model to be tested is at the first level.
[0088] If the performance score is less than or equal to the first threshold and greater than the second threshold, it is determined that the safety performance of the large model to be tested is at the second level.
[0089] If the performance score is less than or equal to the second threshold, it is determined that the safety performance of the large model to be tested is at the third level.
[0090] Among them, the safety performance of the first level is better than that of the second level, and the safety performance of the second level is better than that of the third level.
[0091] Optionally, the testing module 302 is used to:
[0092] The collected malicious questions are input into the large model to be tested to obtain the output answer for each malicious question.
[0093] Malicious questions and their corresponding output answers form a question-answer pair to be tested.
[0094] Optionally, the testing module 302 is used to:
[0095] The positive sample question-answer pairs in the training data set are input into the judgment model to obtain the test scores of the positive sample question-answer pairs.
[0096] The test scores of the positive sample question-answer pairs are sorted in descending order to obtain the first test score sequence of the sorted positive sample question-answer pairs.
[0097] According to the preset ratio, the test score at the position corresponding to the preset ratio in the first test score sequence is used as the first threshold.
[0098] Optionally, the testing module 302 is used to:
[0099] The negative sample question-answer pairs in the training data set are input into the judgment model to obtain the test scores of the negative sample question-answer pairs.
[0100] The test scores of the negative sample question-answer pairs are sorted in order from low to high to obtain a second test score sequence of the sorted negative sample question-answer pairs.
[0101] According to the preset ratio, the test score at the position corresponding to the preset ratio in the second test score sequence is used as the second threshold.
[0102] Optionally, the testing module 302 is used to:
[0103] A training data set is obtained, where each training data in the training data set includes a positive sample question-answer pair consisting of a malicious question and a positive answer, and a negative sample question-answer pair consisting of a malicious question and a negative answer.
[0104] The training data set is input into the analysis and judgment model for training, and in each iteration of the analysis and judgment model, the model parameters of the analysis and judgment model are adjusted according to the preset loss function.
[0105] Optionally, the testing module 302 is used to:
[0106] Collect malicious questions, as well as the positive answers corresponding to the malicious questions and the negative answers corresponding to the malicious questions.
[0107] Generate positive question-answer pairs based on malicious questions and positive answers.
[0108] Generate negative question-answer pairs based on malicious questions and negative answers.
[0109] The positive sample question-answer pair and the negative sample question-answer pair corresponding to a malicious question are combined into a training data.
[0110] Optionally, the testing module 302 is used to:
[0111] The positive sample question-answer pairs or negative sample question-answer pairs in the training data set are input into the feature extraction module of the judgment model to obtain the feature vector.
[0112] The feature vector is input into the mapping module of the judgment model to obtain the test score of the positive sample question and answer pair or the negative sample question and answer pair.
[0113] Optionally, the testing module 302 is used to:
[0114] The first parameter is determined according to the difference between the test score of the positive sample question-answer pair and the test score of the negative sample question-answer pair of each training data.
[0115] According to the first parameter of each training data in the training data set, the second parameter is calculated through a preset loss function.
[0116] According to the second parameter, the model parameters of the analysis model are reversely optimized.
[0117] The large-model safety performance testing device provided in this embodiment can execute the method provided in the above method embodiment. Its implementation principle and technical effects are similar, and will not be described in detail in this embodiment.
[0118] Figure 4 This is a schematic diagram of the structure of the electronic device provided in this application. Figure 4 As shown, the electronic device 400 provided in this embodiment includes: at least one processor 401 and a memory 402. Optionally, the device 40 further includes a communication component 403. The processor 401, the memory 402 and the communication component 403 are connected via a bus 404.
[0119] In a specific implementation process, at least one processor 401 executes the computer-executable instructions stored in the memory 402, so that at least one processor 401 executes the above method.
[0120] The specific implementation process of the processor 401 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.
[0121] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.
[0122] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.
[0123] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.
[0124] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.
[0125] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.
[0126] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.
[0127] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.
[0128] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily required by the present application.
[0129] It should be further noted that, although the various steps in the flowchart are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless there is a clear description in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowchart may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these sub-steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.
[0130] It should be understood that the above-mentioned device embodiments are only illustrative, and the device of the present application can also be implemented in other ways. For example, the division of units / modules in the above-mentioned embodiments is only a logical function division, and there may be other division methods in actual implementation. For example, multiple units, modules or components can be combined, or can be integrated into another system, or some features can be ignored or not executed.
[0131] In addition, unless otherwise specified, each functional unit / module in each embodiment of the present application may be integrated into one unit / module, each unit / module may exist physically separately, or two or more units / modules may be integrated together. The above-mentioned integrated unit / module may be implemented in the form of hardware or in the form of a software program module.
[0132] In the above embodiments, the description of each embodiment has its own emphasis. For the part not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0133] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.
[0134] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.
Claims
1. A large model safety performance testing method, characterized in that: The method comprises: Get the large model to be tested; Generate a question-answer pair to be tested based on the collected malicious questions and the large model to be tested; Input the question-answer pairs to be tested into the trained judgment model to obtain the test scores of all the question-answer pairs to be tested, and use the average of the test scores as the performance score of the large model to be tested; Determining the safety performance level of the large model to be tested according to the performance score, the first threshold and the second threshold; Among them, the first threshold and the second threshold are determined according to the trained analysis and judgment model and the training data set; the first threshold is greater than the second threshold.
2. The method according to claim 1, characterized in that Determining the safety performance level of the large model to be tested according to the performance score, the first threshold and the second threshold includes: If the performance score is greater than the first threshold, it is determined that the safety performance of the large model to be tested is at the first level; If the performance score is less than or equal to the first threshold and greater than the second threshold, it is determined that the safety performance of the large model to be tested is at the second level; If the performance score is less than or equal to the second threshold, it is determined that the safety performance of the large model to be tested is at the third level; Among them, the safety performance of the first level is better than that of the second level, and the safety performance of the second level is better than that of the third level.
3. The method according to claim 1, characterized in that Based on the collected malicious questions and the large model to be tested, a question-answer pair to be tested is generated, including: Inputting the collected malicious questions into the large model to be tested to obtain an output answer to each malicious question; The malicious question and its corresponding output answer form the question-answer pair to be tested.
4. The method according to claim 1, characterized in that: The first threshold is determined according to the trained analysis and judgment model and the training data set, including: Inputting the positive sample question-answer pairs in the training data in the training data set into the judgment model to obtain the test score of the positive sample question-answer pairs; Sorting the test scores of the positive sample question-answer pairs in descending order to obtain a sorted first test score sequence of the positive sample question-answer pairs; According to the preset ratio, the test score at the position corresponding to the preset ratio in the first test score sequence is used as the first threshold.
5. The method according to claim 1, characterized in that: The second threshold is determined according to the trained analysis and judgment model and the training data set, including: Inputting the negative sample question-answer pairs in the training data in the training data set into the judgment model to obtain the test score of the negative sample question-answer pairs; Sorting the test scores of the negative sample question-answer pairs in order from low to high to obtain a sorted second test score sequence of the negative sample question-answer pairs; According to the preset ratio, the test score at the position corresponding to the preset ratio in the second test score sequence is used as the second threshold.
6. The method according to any one of claims 1 to 5, characterized in that The training process of the analysis and judgment model includes: Acquire a training data set, each training data in the training data set includes a positive sample question-answer pair consisting of a malicious question and a positive answer, and a negative sample question-answer pair consisting of a malicious question and a negative answer; The training data set is input into the analysis and judgment model for training, and in each iteration process of the analysis and judgment model, the model parameters of the analysis and judgment model are adjusted according to a preset loss function.
7. The method according to claim 6, characterized in that Get the training data set, including: Collect malicious questions, as well as positive answers corresponding to the malicious questions and negative answers corresponding to the malicious questions; Generating the positive sample question-answer pair according to the malicious question and the positive answer; Generating the negative sample question-answer pair according to the malicious question and the negative answer; The positive sample question-answer pair and the negative sample question-answer pair corresponding to a malicious question are combined into a training data.
8. The method according to claim 7, characterized in that Inputting the training data set into the judgment model for training, including: Inputting the positive sample question-answer pair or the negative sample question-answer pair in the training data in the training data set into the feature extraction module of the judgment model to obtain a feature vector; The feature vector is input into the mapping module of the analysis and judgment model to obtain the test score of the positive sample question and answer pair or the negative sample question and answer pair.
9. The method according to claim 8, characterized in that The model parameters of the judgment model are adjusted according to a preset loss function, including: Determine a first parameter according to a difference between a test score of the positive sample question-answer pair and a test score of the negative sample question-answer pair of each training data; According to the first parameter of each training data in the training data set, a second parameter is calculated by using the preset loss function; According to the second parameter, the model parameters of the analysis and judgment model are reversely optimized.
10. A large model safety performance testing device, comprising: An acquisition module is used to acquire the large model to be tested; Generate a question-answer pair to be tested based on the collected malicious questions and the large model to be tested; An evaluation module is used to input the question-answer pairs to be tested into the trained judgment model to obtain the test scores of all the question-answer pairs to be tested, and use the average of the test scores as the performance score of the large model to be tested; The safety performance level of the large model to be tested is determined based on the performance score, the first threshold and the second threshold; wherein the first threshold and the second threshold are determined based on the trained analysis and judgment model and the training data set; and the first threshold is greater than the second threshold.
11. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 9 when executed by a processor.
13. A computer program product, characterized in that The invention comprises a computer program, which implements the method according to any one of claims 1 to 9 when being executed by a processor.