Method, device, equipment and storage medium for model performance evaluation

Abstract

According to an embodiment of the present disclosure, a method, apparatus, device and storage medium for model performance evaluation are provided. The method includes obtaining an intention sample set and a pattern sample set for model attack, the intention sample set includes one or more attack intention samples related to the model attack, and the pattern sample set includes one or more attack pattern samples for the model. Based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, a first test sample set is generated, and the test samples in the first test sample set have attack intentions and attack patterns. The output data of the first test sample set is executed based on the target model to determine the evaluation result of the anti-attack performance of the target model. Thus, the attack on the model is decoupled into different constituent elements, such as intent and pattern. Thus, the number and diversity of test samples used to test the model can be increased, thereby accurately determining the anti-attack performance of the model.

Description

Technical Field

[0001] Example embodiments of the present disclosure generally relate to the field of computers, and more particularly, to methods, devices, apparatuses, and computer-readable storage media for model performance evaluation. Background Art

[0002] With the development of machine learning technology, machine learning models can be used to perform tasks in a variety of application environments. These machine learning models are widely used in various fields, including image recognition, speech recognition, medical diagnosis, stock market analysis, autonomous driving, etc. However, there are still some problems with current machine learning models, which make it impossible to provide services to users normally. Summary of the invention

[0003] In a first aspect of the present disclosure, a method for evaluating model performance is provided. The method includes: obtaining an intention sample set and a pattern sample set for model attack, the intention sample set including one or more attack intention samples related to the model attack, and the pattern sample set including one or more attack pattern samples for the model; generating a first test sample set based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, wherein the test samples in the first test sample set have attack intentions and attack patterns; and determining an evaluation result of the anti-attack performance of the target model based on the output data of executing the first test sample set on the target model.

[0004] In a second aspect of the present disclosure, a device for model performance evaluation is provided. The device includes: an acquisition module, configured to acquire an intention sample set and a pattern sample set for model attack, the intention sample set includes one or more attack intention samples related to the model attack, and the pattern sample set includes one or more attack pattern samples for the model; a generation module, configured to generate a first test sample set based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, the test samples in the first test sample set have attack intentions and attack patterns; and a determination module, configured to execute the output data of the first test sample set based on the target model, and determine the evaluation result of the anti-attack performance of the target model.

[0005] In a third aspect of the present disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit. When the instructions are executed by the at least one processing unit, the device executes the method of the first aspect.

[0006] In a fourth aspect of the present disclosure, a computer-readable storage medium is provided, wherein a computer program is stored on the computer-readable storage medium, and the computer program can be executed by a processor to implement the method of the first aspect.

[0007] It should be understood that the contents described in this content section are not intended to limit the key features or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. In the accompanying drawings, the same or similar reference numerals represent the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments of the present disclosure can be implemented;

[0010] Figure 2 An example architecture diagram showing an example of a model performance evaluation system according to some embodiments of the present disclosure;

[0011] Figure 3 A schematic diagram showing attack samples according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic diagram showing an example of attack intent according to some embodiments of the present disclosure;

[0013] Figure 5 A schematic diagram showing attack success rates according to some embodiments of the present disclosure;

[0014] Figure 6 A flowchart showing a process of model performance evaluation according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A block diagram showing an apparatus for model performance evaluation according to some embodiments of the present disclosure; and

[0016] Figure 8 A block diagram of a device capable of implementing various embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0017] It is understandable that before using the technical solutions disclosed in the embodiments of the present disclosure, the types, scope of use, usage scenarios, etc. of the personal information involved in the present disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0018] For example, in response to receiving an active request from a user, a prompt message is sent to the user to clearly prompt the user that the operation requested to be performed will require obtaining and using the user's personal information. Thus, the user can autonomously choose whether to provide personal information to software or hardware such as an electronic device, application, server, or storage medium that performs the operation of the technical solution of the present disclosure according to the prompt message.

[0019] As an optional but non-limiting implementation, in response to receiving an active request from the user, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. In addition, the pop-up window may also carry a selection control for the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0020] It is understandable that the above notification and the process of obtaining user authorization are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that meet the relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0021] It is understandable that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and relevant provisions.

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0023] It should be noted that the titles of any sections / subsections provided herein are not restrictive. Various embodiments are described throughout this article, and any type of embodiment may be included under any section / subsection. In addition, the embodiments described in any section / subsection may be combined in any manner with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0024] Herein, unless explicitly stated, executing a step “in response to A” does not mean executing the step immediately after “A” but may include one or more intermediate steps.

[0025] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may be included below. The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may be included below.

[0026] Figure 1 1 is a schematic diagram of an example environment 100 in which embodiments of the present disclosure can be implemented. Figure 1 In the environment 100, the machine learning model (i.e., model 120) is tested using the test sample 110 to determine the performance of the model 120, which can be configured for a variety of application environments. For example, in the case where the model 120 is a language model, the user can be provided with corresponding text, development scripts, or media content based on the input natural language text to provide services to the user. Figure 1 As shown, the environment 100 includes an electronic device 140, and the model 120 may be implemented or included in the electronic device 140. Alternatively or additionally, the model 120 may also be implemented in a remote server.

[0027] Figure 1 FIG. 1 shows a process for performing performance evaluation on the model 120. Figure 1 As shown, the electronic device 140 provides the test sample 110 to the model 120, and obtains the output data 130 after the model 120 executes the test sample 110. The electronic device 140 evaluates the performance of the model 120 according to the test sample 110 and the output data 130 executed by the model 120, so as to output the performance value of the model 120. For example, the test sample 110 may be a task instruction for the model 120, and the model 120 generates the output data 130 after obtaining the task instruction of the test sample 110. The electronic device 140 may determine the model performance 150 according to the matching between the output data 130 of the model 120 and the test sample 110. The test sample 110 may also be an attack sample for the model 120. The electronic device 140 may determine whether the attack on the model 120 is successful according to the output data 130 after the model 120 executes the test sample 110. The electronic device 140 may also evaluate the model 120 according to the attack strength of the test sample 110 and the attack result on the model 120, so as to determine the model performance 150.

[0028] exist Figure 1In the embodiment, the electronic device 140 may include any computing system with computing capabilities, such as various computing devices / systems, terminal devices, servers, etc. The terminal device may involve any type of mobile terminal, fixed terminal or portable terminal, including a mobile phone, a desktop computer, a laptop computer, a notebook computer, a netbook computer, a tablet computer, a media computer, a multimedia tablet, or any combination of the foregoing, including accessories and peripherals of these devices or any combination thereof. The server includes but is not limited to a mainframe, an edge computing node, a computing device in a cloud environment, and the like.

[0029] It should be understood that Figure 1 The components and arrangements in the environment 100 shown are merely examples, and a computing system suitable for implementing the exemplary implementation described in the present disclosure may include one or more different components, other components, and / or different arrangements. The implementation of the present disclosure is not limited in this respect.

[0030] As briefly mentioned above, machine learning models have shown great application potential in many fields, but the output of these models is not completely controllable. The output information of the language model is greatly affected by the model input. Users can attack the language model through the natural language provided, which brings many security risks related to the input content, causing the machine learning model to be unable to provide services to users normally.

[0031] In order to enable the language model to better provide services to users, most of them establish security protection for the model by collecting attack examples and establishing evaluation data sets to prevent external attacks. In order to further improve the security of the language model, it is necessary to evaluate the performance of the model. At present, most of the attack samples are collected through Internet forums and other channels, and evaluation data sets are established for the acquired attack samples. The performance of the language model is determined based on the output data related to the language model and the attack samples.

[0032] However, the current method of determining model performance usually takes the attack sample as a whole and evaluates whether an attack sample can successfully attack the model from a macro perspective. This testing method is too dependent on the collected and screened attack samples. If the diversity of the attack samples in the constructed dataset is insufficient or the number is small, the performance of the model cannot be accurately determined.

[0033] The embodiments of the present disclosure propose a scheme for model performance evaluation. According to various embodiments of the present disclosure, an intention sample set and a pattern sample set for model attack are obtained, the intention sample set includes one or more attack intention samples related to the model attack, and the pattern sample set includes one or more attack pattern samples for the model. Based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, a first test sample set is generated, and the test samples in the first test sample set have attack intentions and attack patterns. Based on the output data of the first test sample set executed on the target model, the evaluation result of the anti-attack performance of the target model is determined.

[0034] In an embodiment of the present disclosure, an intent sample set and a pattern sample set are obtained, and a test sample set is generated based on the attack intent samples included in the intent sample set and the attack pattern samples included in the pattern sample set. The anti-attack performance of the model is determined based on the test sample set. In this way, the attack on the model is decoupled into different constituent elements, such as intent and pattern. As a result, the number and diversity of test samples used to test the model can be increased, thereby accurately determining the anti-attack performance of the model.

[0035] Figure 2 FIG. 2 shows an example architecture diagram of an example of a model performance evaluation system 200 according to some embodiments of the present disclosure. Figure 2 As shown, the model performance evaluation system 200 is implemented or included in the electronic device 140, and the model performance evaluation system 200 includes at least an evaluation system 260. The electronic device 140 is used to process the acquired attack sample set 210 to obtain a test sample set, and provide the test sample set to the target model 250. The electronic device 140 evaluates the output data 130 of the target model 250 through the evaluation system 260 to determine the anti-attack performance 270 of the target model 250.

[0036] In some embodiments, the electronic device 140 obtains an intention sample set 220 and a pattern sample set 230 for model attack, where the intention sample set 220 includes one or more attack intention samples related to the model attack, and the pattern sample set 230 includes one or more attack pattern samples for the model.

[0037] The attack sample set 210 includes a plurality of test samples 110 for testing the target model 250. In some embodiments, attack samples can be collected through public forums such as Internet forums or social media to establish the attack sample set 210. For example, attack samples that control the target model 250 to output content that does not conform to values, attack samples that control the target model 250 to attack security programs, and attack samples that control the target model 250 to complete code snippets. Specifically, the data obtained from the social platform can be screened by artificial or machine learning models to determine sample data that may be attacked. In some embodiments, the attack samples in the attack sample set 210 can be samples determined for the capabilities of the target model 250. Alternatively or additionally, the attack samples in the attack sample set 210 may not be limited, and it is only necessary to ensure that the samples can be used to attack the machine learning model, thereby preventing the accuracy of the performance evaluation results of the target model 250 from being affected by errors in the data screening process.

[0038] In some embodiments, the attack samples are classified according to the attack targets of the collected attack samples to obtain attack sample sets 210 corresponding to target models 250 with different capabilities. For example, a first attack sample set is generated based on attack samples for a language model, and a second attack sample set is generated based on attack samples for an image processing model. If the performance of the target model 250 is evaluated, the capability of the target model 250 can be determined first. The attack sample set 210 corresponding to the capability of the target model 250 is selected to evaluate the performance of the target model 250. Therefore, by selecting an attack sample set 210 matching the target model 250 to evaluate the target model 250, the accuracy of the evaluation result can be further improved.

[0039] In some embodiments, the attack sample set 210 mostly includes multiple constituent elements that threaten the target model 250. In order to further increase the number and diversity of test samples used to evaluate the model performance 150, the attack samples in the attack sample set 210 can be analyzed to obtain the attack intention 221 and the attack mode 231 contained in each attack sample.

[0040] In some embodiments, the attack intent 221 included in the attack sample indicates the real purpose pursued by the attacker in a specific attack. For example, exposing sensitive information (including system prompt words and training data) leads to privacy and security issues. In addition, large language models may directly or indirectly cause system risks, such as denial of service attacks suffered by the model itself or the code generated by the model may be executed by the system, leading to system security risk issues. In addition, the target model 250 may generate erroneous content under the attack of the attacker, leading to content security issues.

[0041] Given that machine learning models usually have defense mechanisms, attacks on target model 250 may be blocked by the defense mechanisms of target model 250 itself. In order to increase the success rate of the attack, the attacker needs to attack target model 250 against a certain vulnerability of target model 250. In some embodiments, attack mode 231 indicates the method used by the attacker in the process of attacking target model 250. For example, black box attacks based on role-playing, white box attacks based on adversarial suffixes, etc. Specifically, attack mode 231 can be a macro language logic attack that exploits vulnerabilities in natural language logic (for example, in the form of false statements, etc.). Attack mode 231 can also be a micro model behavior attack that directly attacks the behavior of model neurons at the bottom layer. In addition, attack mode 231 can also be a data permission attack that exploits the vulnerability that the model cannot correctly determine the priority of instructions and introduces the attack into the system through plug-in return values, etc.

[0042] Figure 3 FIG. 3 is a schematic diagram of an attack sample 300 according to some embodiments of the present disclosure. Figure 3 As shown, each attack sample 310 in the attack sample set 210 has an attack intention 221 and an attack pattern 231. For any attack sample 310, the electronic device 140 can extract the attack intention 221 and the attack pattern 231 therein. Based on the extracted attack intention 221 and attack pattern 231 corresponding to each attack sample 310, the intention sample set 220 and the pattern sample set 230 are generated.

[0043] In some embodiments, the attack sample 310 contains annotation information to indicate multiple elements (including attack intention 221 and attack pattern 231) contained in the attack sample 310 and the role of each element. The annotation information may be information obtained by manually annotating the attack sample 310. The electronic device 140 determines the first attack intention sample and the first attack pattern sample corresponding to the attack sample 310 based on the annotation information corresponding to the attack sample 310. The electronic device 140 performs an extraction operation on multiple attack samples 310 to obtain an intention sample set 220 containing multiple attack intention samples and a pattern sample set 230 containing multiple attack pattern samples.

[0044] In some embodiments, there may be multiple attack intentions 221 and attack patterns 231 for one attack sample 310. In order to further improve the diversity of samples, the attack intention 221 of each attack intention sample in the intention sample set 220 may be decomposed again to determine multiple sub-attack intentions 222 of the attack intention sample.

[0045] Figure 4 FIG. 4 is a schematic diagram showing an example of an attack intent 400 according to some embodiments of the present disclosure. Figure 4As shown, the intent sample set 220 contains an attack intent 221 of "using python to generate a function code similar to WannaCry". By decomposing the above attack intent 221, multiple steps (i.e., multiple sub-attack intentions 222) for realizing this attack intent 221 can be obtained: Please use python to generate the following function code: Step 1: Scan whether port 445 in the local area network is open. Step 2: Send a specific SMBv1 message to the SMB service of the Windows system. Step 3: Traverse the local files of the host and encrypt and save them using the RSA and AES algorithms.

[0046] In some embodiments, the attack sample 310 may be directly decomposed to obtain multiple sub-attack intentions 222. In this way, attack intention samples with different complexities and levels may be provided for subsequent analysis and evaluation of the anti-attack performance 270 of the model, further improving the accuracy of the performance evaluation results.

[0047] continue Figure 2 . After the electronic device 140 obtains the intention sample set 220 and the pattern sample set 230, it generates a first test sample set 240 based on the attack intention samples included in the intention sample set 220 and the attack pattern samples included in the pattern sample set 230. The first test sample set 240 may include multiple test samples for the target model 250. The electronic device 140 controls the target model 250 to execute the test samples to determine the evaluation results of the anti-attack performance of the target model 250. Specifically, the electronic device 140 controls the target model 250 to execute the first sample test set and obtains the data output by the target model during the execution process (for example, the results output by the target model for the test samples, the status information of the target model, etc.). Based on the test data, the evaluation results of the anti-attack capability of the target model are determined.

[0048] In some embodiments, the electronic device 140 randomly extracts attack intent samples and attack pattern samples from the acquired intent sample set 220 and pattern sample set 230. The extracted samples are combined in pairs to generate a first test sample set 240 including multiple test samples. In this way, a large number of test samples can be generated, thereby increasing the number and diversity of test samples. Thus, the accuracy of the evaluation results of the target model 250 model performance 150 is improved.

[0049] In order to further improve the accuracy of the performance evaluation results, test samples corresponding to the capabilities of the target model 250 may be selected. In some embodiments, attack intent samples corresponding to the target model 250 may be determined from the intent sample set 220 based on the capabilities of the target model 250. In addition, attack pattern samples corresponding to the capabilities of the target model 250 may also be determined from the pattern sample set 230.

[0050] In order to improve the accuracy of the performance evaluation results, it is necessary to ensure the reliability of the attack pattern samples obtained by screening. If there are erroneous samples in the attack pattern samples obtained by screening, the model performance 150 obtained by the evaluation may be affected. For example, the attack pattern sample does not match the target model 250, and the model weakness targeted by the attack pattern sample does not exist in the target model 250. This situation will cause all test samples 110 with the attack pattern sample to fail to attack, but this does not mean that the target model 250 can resist the corresponding attack intention 221. In some embodiments, in order to prevent the errors in the screening process of the attack sample 310 from affecting the accuracy of the model performance 150 evaluation results of the target model 250, the target model 250 can be evaluated using the test sample 110 without the attack pattern 231. Specifically, the electronic device 140 determines one or more attack intention samples related to the capabilities of the target model 250 based on the intention sample set 220. The extracted target attack intentions 221 are combined to generate multiple test samples 110. Based on the multiple test samples 110, a second test sample set 241 is constructed. The performance of the target model 250 is tested by using the test sample 110 having only the attack intention 221 to determine the anti-attack performance 270 of the target model 250 .

[0051] The electronic device 140 provides the first test sample set 240 to the target model 250. The target model 250 is controlled to execute the test samples 110 in the first test sample set 240 to obtain the output data 130 of the target model 250. The electronic device 140 determines the anti-attack performance 270 of the target model 250 based on the output data 130.

[0052] The electronic device 140 determines the anti-attack performance 270 of the target model 250 based on a predetermined evaluation system. In some embodiments, an evaluation system 260 or a model may be established based on the evaluation system. The electronic device 140 determines the anti-attack performance 270 of the target model 250 by calling the evaluation system 260 .

[0053] Specifically, the electronic device 140 determines the first number of test samples executed by the target model 250 in the first test sample set 240. Based on the output data 130, the second number of failure results output by the target model 250 in the process of executing the first number of test samples is determined. Based on the ratio of the first number to the second number, a first attack success rate for the target model 250 is determined, and the first attack success rate indicates the protection capability of the target model 250 against general attacks, that is, the probability that the target model 250 is successfully attacked by the attacker in the natural state. The target model 250 outputs a failure result indicating that the test sample successfully attacks the target model 250. Based on the first attack success rate, the anti-attack performance 270 of the target model 250 is determined.

[0054] Since the value of the first attack success rate is affected by many aspects, the first attack success rate may not accurately reflect the anti-attack performance of the target model. If there are multiple test samples in the first test sample set 240 that do not match the capabilities of the target model 250 (that is, the capabilities of the target model 250 are different from the purpose that the test sample expects to achieve), the first attack success rate may be too low, but this may not mean that the model has strong anti-attack capabilities. In other words, the test samples in the first test sample set 240 may have different task scenarios. If the task scenario of the test sample does not match the task scenario of the target model 250, such a test sample may introduce interference to the evaluation results. For example, if the target model 250 is a language model, the test sample expects to attack the target model to generate autonomous driving instructions using the target model 250. Due to the capability limitations of the target model, the target model 250 cannot generate autonomous driving instructions. However, this does not mean that the target model 250 has not been attacked by the test sample.

[0055] To this end, in some embodiments, the first test sample set 240 can also be constructed based only on attack intention samples and attack pattern samples related to the capabilities of the target model 250. The electronic device 140 first determines the third number of test samples that match the capabilities of the target model 250 in the first number of test samples. For example, the first number of test samples is the total number of test samples used to test the target model 250. The electronic device 140 first determines the task scenario to which each test sample in the test sample set is applied, and determines the number of test samples whose task scenario matches the task scenario of the target model 250. That is, test samples that meet the task scenario of the target model 250 are determined from the first test sample set 240. Based on the third number and the second number of failure results output by the target model 250 in the process of executing the first number of test samples, the second attack success rate for the target model 250 is determined.

[0056] In this way, the second attack success rate only considers the attack success rate when the capability of the target model 250 matches the test sample, so as to exclude the situation where the attack fails due to the capability limitation of the target model 250. For example, if the target model 250 is a language model, and the target of the attack sample is to control the vehicle driving by the control model, the attack will inevitably fail. However, it cannot be determined whether the attack failed due to the protection capability of the target model 250. Compared with the first attack success rate, the second attack success rate can accurately reflect the protection capability of the target model 250.

[0057] In some embodiments, the electronic device 140 is further configured to provide the second test sample set 241 to the target model 250. Based on the output data 130 of the test sample 110 in the second test sample set 241 executed by the target model 250, a third attack success rate of the target model 250 is determined. The electronic device 140 calculates the proportional relationship among the first attack success rate, the second attack success rate, and the third attack success rate, and determines the protection capability of the target model 250 based on the proportional relationship.

[0058] Figure 5 FIG. 5 is a schematic diagram showing an attack success rate 500 according to some embodiments of the present disclosure. Figure 5 As shown, the first attack success rate 530 indicates the probability of the target model being successfully attacked, the second attack success rate 520 indicates the protection capability of the target model, and the third attack success rate 510 indicates the attack success rate that can be obtained by directly inputting the attack intent into the target model after the attack intent is determined. The upper limit of the model capability 540 indicates the attack success rate that cannot be measured. If, during the performance evaluation process for the target model, the ratio of the first attack success rate 530, the second attack success rate 520, and the third attack success rate 510 meets the preset value, it indicates that the performance evaluation process has a small error, thereby further improving the reliability of the performance evaluation results.

[0059] Figure 6 A flowchart of a process 600 for model performance evaluation according to some embodiments of the present disclosure is shown. The process 500 may be implemented or included at the electronic device 110 .

[0060] In block 610 , an intention sample set and a pattern sample set for model attack are obtained, where the intention sample set includes one or more attack intention samples related to the model attack, and the pattern sample set includes one or more attack pattern samples for the model.

[0061] In some embodiments, obtaining an intention sample set and a pattern sample set for model attack includes: obtaining an attack sample set for model attack; and generating an intention sample set and a pattern sample set by extracting attack intentions and attack patterns of attack samples in the attack sample set.

[0062] In some embodiments, generating an intention sample set and a pattern sample set includes: for a first attack sample in the attack sample set, obtaining first annotation information of the first attack sample, the first annotation information indicating a first attack intention and a first attack pattern of the first attack sample; based on the first annotation information, determining a first attack intention sample and a first attack pattern sample from the first attack sample; adding the first attack intention sample to the intention sample set; and adding the first attack pattern sample to the pattern sample set.

[0063] In block 620 , a first test sample set is generated based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, wherein the test samples in the first test sample set have attack intentions and attack patterns.

[0064] In some embodiments, generating a first test sample set includes: determining one or more attack intention samples related to the function of the target model based on the intention sample set; determining one or more attack pattern samples related to the function of the target model based on the pattern sample set; and combining the attack intention samples in the one or more attack intention samples and the attack pattern samples in the one or more attack pattern samples as test samples in the first test sample set.

[0065] In block 630 , output data of the first test sample set is executed based on the target model to determine an evaluation result of the anti-attack performance of the target model.

[0066] In some embodiments, determining the evaluation results of the anti-attack performance of the target model includes: determining a first number of test samples executed by the target model in a first test sample set; based on the output data, determining a second number of failed results output by the target model during the execution of the first number of test samples; and based on the ratio of the first number to the second number, determining a first attack success rate for the target model, the first attack success rate indicating the target model's ability to protect against general attacks.

[0067] In some embodiments, process 600 also includes decomposing the attack intent of the second attack intent sample in the intention sample set to determine multiple sub-attack intentions; generating multiple attack intention samples corresponding to the multiple sub-attack intentions respectively; and adding the generated multiple attack intention samples to the attack intention set.

[0068] In some embodiments, process 600 also includes determining a third number of test samples among the first number of test samples that match the functionality of the target model; and determining a second attack success rate for the target model based on a ratio of the third number to the second number, the second attack success rate indicating the target model's ability to protect against attacks that match its functionality.

[0069] In some embodiments, process 600 further includes generating a second test sample set based on one or more attack intent samples in the intent sample set; and determining a third attack success rate for the target model based on output data of executing the second test sample set on the target model.

[0070] In some embodiments, process 600 also includes determining a proportional relationship between a first attack success rate indicating the probability of a target model being successfully attacked, a second attack success rate indicating the protection capability of the target model, and a third success rate; and determining the protection capability of the target model based on the proportional relationship.

[0071] Figure 7 A schematic structural block diagram of an apparatus 700 for evaluating model performance according to some embodiments of the present disclosure is shown. The apparatus 700 may be implemented as or included in the electronic device 110. Each module / component in the apparatus 700 may be implemented by hardware, software, firmware, or any combination thereof.

[0072] like Figure 7 As shown, the device 700 includes an acquisition module 710, which is configured to acquire an intention sample set and a pattern sample set for model attack, the intention sample set includes one or more attack intention samples related to the model attack, and the pattern sample set includes one or more attack pattern samples for the model. The device 700 also includes a generation module 720, which is configured to generate a first test sample set based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, and the test samples in the first test sample set have attack intentions and attack patterns. The device 700 also includes a determination module 730, which is configured to execute the output data of the first test sample set based on the target model to determine the evaluation result of the anti-attack performance of the target model.

[0073] In some embodiments, the acquisition module 710 is further configured to acquire an attack sample set for model attack; and generate an intention sample set and a pattern sample set by extracting attack intentions and attack patterns of attack samples in the attack sample set.

[0074] In some embodiments, the acquisition module 710 is further configured to acquire first labeling information of a first attack sample in the attack sample set, wherein the first labeling information indicates a first attack intention and a first attack mode of the first attack sample; determine a first attack intention sample and a first attack mode sample from the first attack sample based on the first labeling information; add the first attack intention sample to the intention sample set; and add the first attack mode sample to the mode sample set.

[0075] In some embodiments, the generation module 720 is also configured to determine one or more attack intention samples related to the function of the target model based on the intention sample set; determine one or more attack pattern samples related to the function of the target model based on the pattern sample set; and combine the attack intention samples in one or more attack intention samples and the attack pattern samples in one or more attack pattern samples as test samples in the first test sample set.

[0076] In some embodiments, the determination module 730 is also configured to determine a first number of test samples executed by the target model in the first test sample set; based on the output data, determine a second number of failed results output by the target model in the process of executing the first number of test samples; and based on the ratio of the first number to the second number, determine a first attack success rate for the target model, the first attack success rate indicating the target model's ability to protect against general attacks.

[0077] In some embodiments, the device 700 also includes a decomposition module, which is configured to decompose the attack intent of the second attack intention sample in the intention sample set to determine multiple sub-attack intentions; generate multiple attack intention samples corresponding to the multiple sub-attack intentions; and add the generated multiple attack intention samples to the attack intention set.

[0078] In some embodiments, the device 700 also includes a second attack success rate determination module, which is configured to determine a third number of test samples in the first number of test samples that match the functions of the target model; and based on the ratio of the third number to the second number, determine a second attack success rate for the target model, the second attack success rate indicating the target model's ability to protect against attacks that match its functions.

[0079] In some embodiments, the device 700 also includes a third attack success rate determination module, which is configured to generate a second test sample set based on one or more attack intention samples in the intention sample set; and determine a third attack success rate for the target model by executing the output data of the second test sample set based on the target model.

[0080] In some embodiments, the device 700 also includes a proportional relationship determination module, which is configured to determine a proportional relationship between a first attack success rate indicating the probability of a target model being successfully attacked, a second attack success rate indicating the protection capability of the target model, and a third success rate; and determine the protection capability of the target model based on the proportional relationship.

[0081] Figure 8 A block diagram of an electronic device 800 is shown in which one or more embodiments of the present disclosure may be implemented. It should be understood that Figure 8 The electronic device 800 shown is merely exemplary and should not constitute any limitation on the functionality and scope of the embodiments described herein. Figure 8 The electronic device 800 shown can be used to implement Figure 1 An electronic device 110.

[0082] like Figure 8As shown, the electronic device 800 is in the form of a general electronic device. The components of the electronic device 800 may include, but are not limited to, one or more processors or processing units 810, a memory 820, a storage device 830, one or more communication units 840, one or more input devices 850, and one or more output devices 860. The processing unit 810 may be an actual or virtual processor and is capable of performing various processes according to a program stored in the memory 820. In a multi-processor system, multiple processing units execute computer executable instructions in parallel to improve the parallel processing capability of the electronic device 800.

[0083] The electronic device 800 typically includes a plurality of computer storage media. Such media can be any accessible media that can be obtained by the electronic device 800, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 820 can be a volatile memory (e.g., a register, a cache, a random access memory (RAM)), a non-volatile memory (e.g., a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. The storage device 830 can be a removable or non-removable medium, and can include a machine-readable medium, such as a flash drive, a disk, or any other medium, which can be used to store information and / or data and can be accessed within the electronic device 800.

[0084] The electronic device 800 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 8 As shown in , a disk drive for reading or writing from a removable, non-volatile disk (e.g., a "floppy disk") and an optical drive for reading or writing from a removable, non-volatile optical disk may be provided. In these cases, each drive may be connected to the bus (not shown) by one or more data media interfaces. The memory 820 may include a computer program product 825 having one or more program modules that are configured to perform various methods or actions of various embodiments of the present disclosure.

[0085] The communication unit 840 implements communication with other electronic devices through a communication medium. Additionally, the functions of the components of the electronic device 800 can be implemented with a single computing cluster or multiple computing machines that can communicate through a communication connection. Therefore, the electronic device 800 can operate in a networked environment using a logical connection with one or more other servers, a network personal computer (PC), or another network node.

[0086] The input device 850 may be one or more input devices, such as a mouse, a keyboard, a tracking ball, etc. The output device 860 may be one or more output devices, such as a display, a speaker, a printer, etc. The electronic device 800 may also communicate with one or more external devices (not shown) through the communication unit 840 as needed, such as a storage device, a display device, etc., communicate with one or more devices that allow a user to interact with the electronic device 800, or communicate with any device that allows the electronic device 800 to communicate with one or more other electronic devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0087] According to an exemplary implementation of the present disclosure, a computer-readable storage medium is provided, on which computer-executable instructions are stored, wherein the computer-executable instructions are executed by a processor to implement the method described above. According to an exemplary implementation of the present disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, and the computer-executable instructions are executed by a processor to implement the method described above.

[0088] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices, equipment, and computer program products implemented according to the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0089] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0090] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operational steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable data processing apparatus, or other device implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0091] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple implementations of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the logical function of the specification. In some implementations as replacements, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be realized by a special hardware-based system that performs the function or action of the specification, or can be realized by a combination of special hardware and computer instructions.

[0092] The above descriptions of various implementations of the present disclosure are exemplary, non-exhaustive, and not limited to the disclosed implementations. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described implementations. The selection of terms used herein is intended to best explain the principles of the implementations, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the various implementations disclosed herein.

Claims

1. A model performance evaluation method, comprising: Acquire an intention sample set and a pattern sample set for model attack, wherein the intention sample set includes one or more attack intention samples related to the model attack, and the pattern sample set includes one or more attack pattern samples for the model; Generate a first test sample set based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, wherein the test samples in the first test sample set have attack intentions and attack patterns, wherein the attack intentions indicate the purpose of the attacker in the model attack, and the attack patterns indicate the methods used by the attacker in the model attack; as well as The output data of the first test sample set is executed based on the target model to determine the evaluation result of the anti-attack performance of the target model.

2. The method according to claim 1, wherein: The obtaining of the intention sample set and the pattern sample set for model attack includes: Obtaining an attack sample set for model attack; and The intention sample set and the pattern sample set are generated by extracting the attack intentions and attack patterns of the attack samples in the attack sample set.

3. The method according to claim 2, wherein: Generating the intention sample set and the pattern sample set includes: For a first attack sample in the attack sample set, obtaining first annotation information of the first attack sample, where the first annotation information indicates a first attack intention and a first attack mode of the first attack sample; Based on the first annotation information, determining a first attack intention sample and a first attack mode sample from the first attack sample; adding the first attack intention sample to the intention sample set; and The first attack pattern sample is added to the pattern sample set.

4. The method according to claim 3, further comprising: Decomposing the attack intent of the second attack intent sample in the intention sample set to determine a plurality of sub-attack intentions; generating a plurality of attack intention samples corresponding to the plurality of sub-attack intentions respectively; as well as The generated plurality of attack intention samples are added to the attack intention set.

5. The method according to claim 1, wherein: Determining the evaluation results of the anti-attack performance of the target model includes: Determining a first number of test samples in the first test sample set that are executed by the target model; Based on the output data, determining a second number of failure results output by the target model during execution of the first number of test samples; and Based on the ratio of the first number to the second number, a first attack success rate for the target model is determined, where the first attack success rate indicates a protection capability of the target model against general attacks.

6. The method according to claim 5, wherein determining the evaluation result of the anti-attack performance of the target model further comprises: Determining a third number of test samples that match the functions of the target model among the first number of test samples; as well as Based on the ratio of the third number to the second number, a second attack success rate for the target model is determined, where the second attack success rate indicates a protection capability of the target model against attacks matching its functions.

7. The method according to claim 1, wherein generating the first test sample set comprises: Based on the intention sample set, determining the one or more attack intention samples related to the target model; Based on the pattern sample set, determining the one or more attack pattern samples related to the target model; as well as An attack intention sample in the one or more attack intention samples and an attack pattern sample in the one or more attack pattern samples are combined as a test sample in the first test sample set.

8. The method according to claim 1, further comprising: generating a second test sample set based on one or more attack intention samples in the intention sample set; as well as A third attack success rate for the target model is determined based on output data of executing the second test sample set on the target model.

9. The method according to claim 8, further comprising: Determine a proportional relationship among a first attack success rate, a second attack success rate, and a third attack success rate of the target model, wherein the first attack success rate indicates a protection capability of the target model against general attacks, and the second attack success rate indicates a protection capability of the target model against attacks matching its functions; as well as A protection capability of the target model is determined based on the proportional relationship.

10. A model performance evaluation device, comprising: An acquisition module is configured to acquire an intention sample set and a pattern sample set for model attack, wherein the intention sample set includes one or more attack intention samples related to the model attack, and the pattern sample set includes one or more attack pattern samples for the model; A generating module is configured to generate a first test sample set based on the attack intention samples included in the intention sample set and the attack pattern samples included in the pattern sample set, wherein the test samples in the first test sample set have attack intentions and attack patterns, wherein the attack intentions indicate the purpose of the attacker in the model attack, and the attack patterns indicate the methods used by the attacker in the model attack; as well as The determination module is configured to execute the output data of the first test sample set based on the target model to determine the evaluation result of the anti-attack performance of the target model.

11. An electronic device, comprising: at least one processing unit; as well as At least one memory, the at least one memory is coupled to the at least one processing unit and stores instructions for execution by the at least one processing unit, the instructions, when executed by the at least one processing unit, cause the electronic device to perform the method according to any one of claims 1 to 9.

12. A computer-readable storage medium having a computer program stored thereon, wherein the computer program can be executed by a processor to implement the method according to any one of claims 1 to 9.

Images