Security assessment method and device for model, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a security assessment method and device for a model, equipment and a storage medium, and the method comprises the steps: obtaining a to-be-assessed model, selecting a target attack method based on the features of the to-be-assessed model, and generating an attack sample through employing the target attack method; then, attacking the to-be-evaluated model by using the attack sample, and obtaining multi-dimensional response data of the to-be-evaluated model in the attacked process; and performing quantitative analysis on the multi-dimensional response data to obtain a safety score which is used for indicating the safety of the to-be-evaluated model. And the security of the model facing hostile attacks is evaluated, so that the accuracy and comprehensiveness of security evaluation are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a security assessment method, device, equipment and storage medium for a model. Background Art

[0002] With the rapid development of artificial intelligence technology, intelligent models have been widely used in many fields such as image recognition, natural language processing, and autonomous driving. These models usually rely on a large amount of data for training to achieve accurate prediction and decision-making for complex tasks. However, the security issues of intelligent models have gradually become a key factor restricting their further development and application.

[0003] Adversarial attacks, in particular, have become an important topic in the research of intelligent model security. Adversarial attacks refer to the attacker introducing imperceptible disturbances in legitimate data through carefully designed input samples, causing the algorithm to produce incorrect output or behavior. Such attacks are particularly dangerous for models that rely on machine learning because they can easily bypass the model's defense mechanisms.

[0004] Existing security assessment technologies often ignore the depth of understanding of adversarial attacks and lack a complete process from security threat identification to comprehensive assessment. For example, some current security assessment methods rely on simple fuzz testing and static code analysis. These methods are usually based on empirical deduction and rely on a certain amount of security incident accumulation, and their assessment accuracy and comprehensiveness are limited. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a security assessment method, device, equipment and storage medium for a model, which evaluates the security of the model when facing malicious attacks and improves the accuracy and comprehensiveness of the security assessment.

[0006] In a first aspect, the present application provides a security assessment method for a model, the method comprising: obtaining a model to be evaluated; selecting a target attack method based on the characteristics of the model to be evaluated; generating an attack sample using the target attack method; inputting the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample; obtaining multi-dimensional response data of the model to be evaluated in the process of processing the attack sample; performing quantitative analysis on the multi-dimensional response data to obtain a security score, which is used to indicate the security of the model to be evaluated.

[0007] In the second aspect, the present application provides a security assessment device for a model, which includes: a model acquisition module, used to acquire the model to be evaluated; a method selection module, used to select a target attack method based on the characteristics of the model to be evaluated; a sample generation module, used to generate an attack sample using the target attack method; a sample attack module, used to input the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample; a data acquisition module, used to acquire multi-dimensional response data of the model to be evaluated in the process of processing the attack sample; a security analysis module, used to perform quantitative analysis on the multi-dimensional response data to obtain a security score, which is used to indicate the security of the model to be evaluated.

[0008] In a third aspect, the present application provides an electronic device, which may be a security assessment method for a model, and the electronic device includes: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the security assessment method for a model as described in the first aspect above.

[0009] In a fourth aspect, the present application provides a storage medium, which may be a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the security assessment method for the model in the first aspect described above is implemented.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the security assessment method for the model as described in any one of the first aspects above is implemented.

[0011] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0012] The embodiment of the present application provides a security assessment method, device, equipment and storage medium for a model. First, the model to be assessed is obtained, and then a target attack method is selected based on the characteristics of the model to be assessed, and an attack sample is generated using the target attack method; then, the model to be assessed is attacked using the attack sample, and multi-dimensional response data of the model to be assessed during the attack process is obtained; the multi-dimensional response data is quantitatively analyzed to obtain a security score, which is used to indicate the security of the model to be assessed. The security of the model in the face of malicious attacks is evaluated, and the accuracy and comprehensiveness of the security assessment are improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 A schematic diagram of a flow chart of a security assessment method for a model provided in an embodiment of the present application;

[0016] Figure 2 A schematic diagram of a flow chart of an optimized security assessment method for a model provided in an embodiment of the present application;

[0017] Figure 3 A schematic diagram of the structure of a safety assessment device for a model provided in an embodiment of the present application;

[0018] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0019] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0020] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0021] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0022] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0023] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0024] The following is a detailed description of the security assessment method for the model provided by the present application in conjunction with the accompanying drawings and specific implementation methods.

[0025] Figure 1 This is a flow chart of a security assessment method for a model in an embodiment of the present application. This embodiment can be applied to assessing the security of an intelligent model. The method can be executed by a security assessment device for a model. The security assessment device for a model can be implemented in software and / or hardware. The security assessment device for a model can be configured in an electronic device.

[0026] like Figure 1 As shown, the security assessment method for a model provided in the embodiment of the present application mainly includes steps S101-S105.

[0027] S101. Obtain a model to be evaluated.

[0028] The above-mentioned model may be an intelligent model, which refers to an algorithm or system for implementing artificial intelligence (AI) tasks, and the above-mentioned intelligent model can learn from data and make predictions or decisions. The above-mentioned intelligent model may include: a machine learning model, a deep learning model, a reinforcement learning model, etc. It should be noted that the above-mentioned intelligent model may also include other forms of artificial intelligence algorithms, which are only illustrated in this embodiment and not limited.

[0029] Obtaining a model to be evaluated includes: selecting a representative model from the same type of models that can reflect the security features to be evaluated.

[0030] S102. Select a target attack method based on the features of the model to be evaluated.

[0031] The characteristics of the model to be evaluated mainly include the type of model to be evaluated and the security vulnerabilities of the model to be evaluated. The types of models to be evaluated mainly include: computer vision, natural language, and audio.

[0032] Among them, computer vision models are mainly used to process and understand image or video data, and can perform various tasks such as object detection, image classification, semantic segmentation, instance segmentation, posture estimation, etc. Natural language models are mainly used to understand and generate human language, and can perform various tasks such as text classification, sentiment analysis, machine translation, question-answering systems, text generation, etc. Audio models are mainly used in speech recognition, speech synthesis, music generation, sound classification and enhancement, etc.

[0033] The attack method can be understood as a method or strategy for generating adversarial attack samples. Due to the different types of models to be evaluated, the required attack samples must be different. Specifically, a target attack method matching the type of model to be evaluated is obtained.

[0034] In an embodiment of the present application, a security assessment system is pre-constructed, in which a plurality of generation models are pre-set, and the generation models are specially designed to generate adversarial samples. After obtaining the model to be evaluated, a target attack method is selected from the plurality of pre-set generation models according to the characteristics of the model to be evaluated. Specifically, a target attack method of the same type as the model to be evaluated is selected from the plurality of pre-set generation models.

[0035] In one possible implementation, a target attack method is selected based on the features of a model to be evaluated, including: when the model to be evaluated is a computer vision model, an image sample generation method is selected as the target attack method, wherein the image sample generation method includes at least one of the following: Fast Gradient Sign Method (FGSM), Basic Iterative Method (BIM), Projected Gradient Descent (PGD), Momentum Iterative Fast Gradient Sign Method (MI-FGSM).

[0036] Among them, computer vision models are usually used to process and understand image or video data. Therefore, when the model to be evaluated is a computer vision model, the input data that needs to be input into the computer vision model is an image. Therefore, it is necessary to select an image sample generation method as the target attack method of the computer vision model. That is, use the image sample generation method to generate image attack samples, and use the image attack samples to attack the computer vision model.

[0037] In the embodiment of the present application, four image sample generation methods are provided, and one method can be selected from any of the above four image sample generation methods as a target attack method. Furthermore, a model can be selected from the above four image sample generation methods as a target attack method according to the application scenario of the computer vision model.

[0038] FGSM is a simple single-step method that modifies the input by calculating the sign of the gradient of the loss function with respect to the input image to generate adversarial samples. When adversarial samples need to be generated quickly, FGSM can be selected as a targeted attack method. For example, when evaluating the robustness of a model or conducting preliminary security testing, FGSM can be selected as a targeted attack method.

[0039] BIM can be seen as an iterative version of FGSM, which repeatedly applies FGSM multiple times, each time making only small changes to the pixel value. When more precise control of the disturbance amplitude and improved attack success rate are required, BIM can be selected as a targeted attack method. For example, when security requirements are high and in-depth security assessment of the model is required, BIM can be selected as a targeted attack method.

[0040] PGD ​​limits the perturbation to a specific range after each iteration (i.e., the perturbation size constraint), ensuring that the final adversarial sample remains within a reasonable range. When you need to generate high-quality adversarial samples and can control the perturbation size, you can choose PGD as the target attack method. For example: when conducting research on model reinforcement or defense mechanisms, choose PGD as the target attack method.

[0041] MI-FGSM introduces the concept of momentum, which makes the perturbation direction more stable, effectively avoids the gradient vanishing or exploding problem, and improves the success rate of the attack. When you need to maintain a high attack success rate while reducing the number of iterations, you can choose MI-FGSM as the target attack method. For example: when conducting resistance training or adversarial defense research, choose MI-FGSM as the target attack method.

[0042] In one possible implementation, a target attack method is selected based on the features of the model to be evaluated, including: when the model to be evaluated is a natural language model, a text sample generation method is selected as the target attack method, wherein the text attack model includes at least one of the following: a text deception attack method TextFooler, a sentence compression interpretation network SCPN, a text perturbator TextBugger, and a colorful snake VIPER.

[0043] Among them, natural language models are usually used to process and understand text data. Therefore, when the model to be evaluated is a natural language model, the input data that needs to be input into the natural language model is text. Therefore, it is necessary to select a text sample generation method as the target attack method of the natural language model. That is, use the text sample generation method to generate text attack samples, and use the text attack samples to attack the natural language model.

[0044] In the embodiment of the present application, four text sample generation methods are provided, and a model can be selected from any of the above four text sample generation methods as a target attack method. Furthermore, a model can be selected from the above four text sample generation methods as a target attack method according to the application scenario of the natural language model.

[0045] TextFooler is a text adversarial attack method that fools the classifier by minimizing the modification of the input text. The goal is to make the model's predictions wrong while keeping the text semantics as unchanged as possible. When testing and improving text classifiers based on deep learning, you can choose TextFooler as the target attack method. For example, when the model to be evaluated is a sentiment analysis model, a news classification model, or a spam identification model, choose TextFooler as the target attack method.

[0046] SCPN is a paraphrase technology based on sentence compression, which can generate sentences with similar meanings but different expressions to the original sentences. When enhancing the diversity of data sets or generating adversarial samples to test paraphrase sensitivity, SCPN can be selected as a target attack method. For example, when the model to be evaluated is a semantic similarity judgment model or a paraphrase generation model, SCPN can be selected as a target attack method.

[0047] TextBugger is a text adversarial perturbation generation framework that can alter input text by adding, deleting, or replacing characters to produce misleading input. TextBugger can be used as a targeted attack method to enhance the security and accuracy of text classifiers and question-answering systems.

[0048] VIPER is an adversarial attack method that focuses on generating barely perceptible changes that are sufficient to cause a machine reading comprehension system to misunderstand its input. VIPER can be selected as a targeted attack method when testing and improving the security of machine reading comprehension and complex text generation tasks. For example, when the model to be evaluated is a reading comprehension model or a machine translation model, VIPER is selected as a targeted attack method.

[0049] In one possible implementation, a target attack method is selected based on the features of the model to be evaluated, including: when the model to be evaluated is an audio model, a speech sample generation method is selected as the target attack method, wherein the speech sample generation method includes at least one of the following: automatic speech recognition attack method ASR Attack, Bob's voice forgery attack method FakeBob, and Carlini-Wagner attack method C&W.

[0050] Among them, audio models are usually used to process and understand audio data. Therefore, when the model to be evaluated is an audio model, the input data that needs to be input into the audio model is audio. Therefore, it is necessary to select a speech sample generation method as the target attack method for the audio model. That is, use the speech sample generation method to generate text attack samples, and use the text attack samples to attack the audio model.

[0051] In the embodiment of the present application, four speech sample generation methods are provided, and a model can be selected from any of the four speech sample generation methods as a target attack method. Furthermore, a model can be selected from the four speech sample generation methods as a target attack method according to the application scenario of the audio model.

[0052] ASRAttack is an attack method specifically targeting automatic speech recognition systems. The attack may involve creating adversarial samples by adding imperceptible noise to the original audio signal or by style transfer, so that the ASR system produces incorrect transcription results. It maintains audio quality while misleading the ASR system through iterative style code attacks (SCA). For example, ASRAttack can be selected as a targeted attack method when testing and evaluating the security and robustness of any deep learning-based speech recognition model.

[0053] FAKEBOB is an adversarial attack method against speaker recognition systems. This attack simulates the voice characteristics of a specific speaker by generating adversarial examples, which can unlock mobile phone voiceprint verification, log in to mobile applications, and even verify bank transactions. When testing the security of voiceprint recognition systems, FAKEBOB can be selected as a targeted attack method. For example: when the model to be evaluated is processing open-set speaker identification, closed-set speaker identification (CSI) and speaker verification (SV) tasks, FAKEBOB can be selected as a targeted attack method.

[0054] C&W attack is a series of attack methods proposed by Nicholas Carlini and David Wagner. C&W attack finds the most difficult adversarial samples by optimizing the objective function, considering different distance metrics such as L0, L2 and L∞ norms. For example, when the model to be evaluated is a speaker recognition model or an emotion recognition model, FAKEBOB can be selected as the target attack method.

[0055] Conduct an in-depth analysis of the model to be evaluated to match the appropriate adversarial attack method. Specifically, based on the understanding of the model's structure, function, and potential security weaknesses, select specific adversarial techniques that can effectively reveal the model's security vulnerabilities. For example, if the model to be evaluated shows high accuracy in the field of image recognition but is sensitive to small input perturbations, the selected adversarial attack method will focus on generating samples that can cause misclassification under these small perturbations.

[0056] S103: Generate attack samples using a target attack method.

[0057] In an embodiment of the present application, the attack method can be a pre-built deep neural network that can learn the complex patterns of data in the original data set and generate highly deceptive adversarial samples. Furthermore, the above-mentioned target attack method can be a generative model built using deep learning technology, which is specially designed to generate adversarial samples. These samples are visually or statistically similar to normal data, but are carefully designed to mislead the model to be evaluated.

[0058] Attack samples are generated through carefully designed attack methods to test the performance of the model to be evaluated in the face of different types of attacks. The generation process may include adjusting the pixel values ​​of the image, introducing noise, or applying specific geometric transformations to create samples that are imperceptible to humans but have a significant impact on the algorithm's decision.

[0059] Furthermore, the generative model also includes at least one loss function, which is used to evaluate the difference between the generated attack sample and the original sample, as well as the impact of the generated sample on the model to be evaluated. The loss function can quantify the impact of the adversarial sample on the algorithm to be evaluated, and is usually defined based on the model prediction error. Through iterative optimization, the parameters of the generative model are adjusted to improve the effectiveness of the adversarial sample. This may involve adjusting the network structure, learning rate, number of iterations, etc., with the goal of generating samples that can maximize the impact on the judgment of the intelligent algorithm.

[0060] Before using the target attack method to generate attack samples in S103, it also includes: obtaining an original data set, and performing data preprocessing on the original data set, wherein the data preprocessing includes: cleaning, standardizing and enhancing the original data set to improve the generalization ability of the attack method and the quality of the adversarial samples.

[0061] In a possible implementation, attack samples are generated using a targeted attack method, including: analyzing the input and output behaviors, data processing logic, data processing methods, and decision execution process of the model to be evaluated to obtain security vulnerabilities of the model to be evaluated; and generating attack samples that attack the security vulnerabilities using the targeted attack method.

[0062] Use the pre-built security assessment system to conduct a detailed analysis of the model to be assessed. This analysis process not only includes the input and output behavior of the model, but also goes deep into the model's internal logic, data processing methods, and decision-making process.

[0063] For example, for an image recognition model, its feature extraction method, the structure of the convolutional neural network, and the final classification logic will be analyzed. By carefully reviewing each link of the model, possible security vulnerabilities can be identified, such as the sensitivity of a specific layer that an attack sample may exploit. Then, attack samples targeting possible security vulnerabilities are generated.

[0064] In this way, the security of the model can be evaluated by using attack samples generated for security vulnerabilities, which can improve the accuracy of the model security assessment.

[0065] In the embodiments of the present application, generating attack samples is a key step in security assessment, requiring the framework to not only understand the working principle of the model, but also be able to predict and simulate potential attack methods. For example, through deep learning technology, adversarial images can be generated that are almost indistinguishable from normal images at the pixel level, but can cause computer vision models to misclassify them.

[0066] Different types of algorithms require different adversarial samples for attack. Specifically, for computer vision algorithms, we use adversarial attack methods such as FGSM, BIM, PGD, and MI-FGSM to generate image adversarial samples; for natural language algorithms, we use methods such as TextFooler, SCPN, TextBugger, and VIPER to generate text adversarial samples; for audio algorithms, we use methods such as ASRAttack, FakeBob, and C&W to generate voice adversarial samples.

[0067] S104: Input the attack sample into the model to be evaluated, so that the model to be evaluated processes the attack sample.

[0068] The adversarial sample generated in S103 is input into the model to be evaluated to simulate the attack scenario in the real world. Specifically, the attack sample is input into the model to be evaluated, and the model to be evaluated is rigorously tested and evaluated.

[0069] S105. Obtain multi-dimensional response data of the model to be evaluated in the process of processing the attack sample.

[0070] The multi-dimensional response data includes: the input and output behaviors and behavior changes of the model to be evaluated. For example, the multi-dimensional response data includes: reduced accuracy, increased processing time, etc.

[0071] Monitor the identification and processing of these attack samples by the model to be evaluated in real time, and record the output results and behavior changes of the model to be evaluated. This process involves real-time monitoring and behavior analysis of the response of the model to be evaluated.

[0072] S106. Quantitatively analyze the multi-dimensional response data to obtain a safety score, which is used to indicate the safety of the model to be evaluated.

[0073] The safety score is directly proportional to the safety of the model to be evaluated. The higher the safety score, the higher the safety of the model to be evaluated, and the lower the safety score, the lower the safety of the model to be evaluated.

[0074] Quantitative analysis of multi-dimensional response data, including analysis of whether the algorithm can correctly identify adversarial samples and performance changes during the identification process. Quantitative analysis methods are used to evaluate the performance degradation of intelligent algorithms when facing adversarial samples, such as reduced accuracy and increased processing time, so as to evaluate their robustness level.

[0075] For example, if an image recognition component in an autonomous driving system is fed with adversarial samples, the output of the component will be monitored to analyze whether it can correctly identify road signs or pedestrians, or whether it will produce abnormal outputs that mislead the system. By evaluating the performance of the image recognition component under various adversarial conditions, it is possible to quantify the robustness of the image recognition component and identify its vulnerability under specific attack vectors.

[0076] In one possible implementation, a quantitative analysis is performed on multi-dimensional response data to obtain a security score, which is used to indicate the security of the model to be evaluated, including: obtaining pre-set scoring criteria for each dimension; performing a quantitative analysis based on the response data and scoring criteria for each dimension to obtain a score for each dimension; and performing a weighted calculation on the score for each dimension to obtain a security score.

[0077] Specifically, a multi-dimensional security scoring standard is pre-set, which is used to evaluate the security of intelligent algorithms from different perspectives. Define multiple security dimensions, such as false positive rate, false negative rate, response time, etc.; design scoring standards and quantitative methods for each dimension. Combine the scores of each dimension to form an overall security score for the intelligent algorithm; use the security score to guide the security improvement and priority setting of the intelligent algorithm.

[0078] According to the type of model to be evaluated or the specific usage scenario of the model to be evaluated, the scoring criteria and quantification method are set. Different weight values ​​can also be set according to the type of model to be evaluated or the specific usage scenario of the model to be evaluated.

[0079] In one possible implementation, security assessment data of the model to be evaluated under different attack strategies is collected, wherein the security assessment data includes at least one of the following: attack samples, multi-dimensional response data, scores of each dimension, and a security score; feature vectors related to the security performance of the model to be evaluated are extracted from the security assessment data; the feature vectors are used to predict the predicted multi-dimensional response data of the model to be evaluated when facing a new type of adversarial attack; and preventive measures for the model to be evaluated are formulated based on the predicted multi-dimensional response data.

[0080] Among them, different attack strategies include: different target attack methods, or different attack samples generated using the same attack method.

[0081] Specifically, collect security assessment data of the model to be evaluated under different attack strategies, including the type of adversarial samples, the response of the intelligent algorithm, and the evaluation results. Extract features related to the security performance of the model to be evaluated from the collected data; use machine learning technology to train a prediction model based on the extracted features; use the trained prediction model to predict the security performance of the model to be evaluated when facing new adversarial attacks; formulate preventive measures, and formulate corresponding preventive measures in advance based on the prediction results to enhance the security of the intelligent algorithm.

[0082] The embodiment of the present application provides a security assessment method for a model. First, the model to be assessed is obtained, and then a target attack method is selected based on the characteristics of the model to be assessed, and an attack sample is generated using the target attack method; then, the model to be assessed is attacked using the attack sample, and multi-dimensional response data of the model to be assessed during the attack process is obtained; the multi-dimensional response data is quantitatively analyzed to obtain a security score, which is used to indicate the security of the model to be assessed. This method realizes the security assessment of the model in the face of malicious attacks, and improves the accuracy and comprehensiveness of the security assessment.

[0083] Based on the above embodiments, the embodiments of the present application further optimize the security assessment method for the model, such as Figure 2 As shown in the figure, the optimized security assessment method for the model mainly includes the following steps:

[0084] S201, obtaining a model to be evaluated;

[0085] S202, selecting a target attack method based on the features of the model to be evaluated;

[0086] S203, generating an attack sample using a target attack method;

[0087] S204, inputting the attack sample into the model to be evaluated, so that the model to be evaluated processes the attack sample;

[0088] S205, obtaining multi-dimensional response data of the model to be evaluated in the process of processing the attack sample;

[0089] S206. Quantitatively analyze the multi-dimensional response data to obtain a safety score, which is used to indicate the safety of the model to be evaluated.

[0090] The processes of S201-S206 provided in the embodiment of the present application are the same as the processes of S101-S106 provided in the above embodiment. For details, please refer to the description in the above embodiment, and no specific limitations are given in the embodiment of the present application.

[0091] S207: Determine whether the safety score is less than a first set value, if yes, execute S208. If no, execute S209.

[0092] The first set value is a value for evaluating the security score. It can also be understood that the first set value is a value for evaluating the security of the model to be evaluated. If the security score is greater than or equal to the first set value, it indicates that the model to be evaluated has a high security during the attack, or when the attack sample of this type is used for attack, the model to be evaluated has a high security. If the security score is less than the first set value, it indicates that the security of the model to be evaluated is poor.

[0093] S208. Design a model optimization strategy based on multi-dimensional response data.

[0094] Among them, the model optimization strategy includes at least one of the following: adjusting the model parameters of the model to be evaluated, improving the network structure of the model to be evaluated, and adding a new defense mechanism to the model to be evaluated.

[0095] In-depth analysis of multi-dimensional response data, identification of key points that need to be improved in the model to be evaluated, algorithm optimization design, and design of optimization strategies based on the analysis results, including adjustment of algorithm parameters, improvement of network structure, or introduction of new defense mechanisms.

[0096] A comprehensive analysis of the performance of the model to be evaluated, which includes not only the accuracy of the algorithm under adversarial sample attacks, but also its resource consumption, processing speed, and recovery ability when processing these samples. For example, in the fraud detection algorithm in the financial field, special attention will be paid to the response time and false negative rate of the algorithm when facing carefully designed fraud patterns, ensuring that these key indicators are within a safe and acceptable range.

[0097] Through the above continuous evaluation process, comprehensive feedback can be provided to the developers of the model to be evaluated, pointing out the deficiencies in the security of the model to be evaluated, and providing directions and suggestions for improvement. For example, if it is found that the model to be evaluated has defects in handling certain types of adversarial samples, it may be recommended to add regularization terms to enhance the generalization ability of the model, or adjust the network structure to reduce sensitivity to input perturbations.

[0098] S209: Optimize the model to be evaluated by using the model optimization strategy to obtain the optimized model to be evaluated, and return to S204 until the safety score is greater than or equal to the first set value.

[0099] Apply the designed model optimization strategy to the model to be evaluated, and implement model updates and upgrades; verify the effectiveness of the optimization measures by conducting another security assessment, and make further adjustments as needed; establish a continuous optimization and evaluation cycle to continuously improve the security of the intelligent algorithm.

[0100] S210, adjusting the attack strategy, the attack strategy includes: selecting a new target attack method based on the characteristics of the model to be evaluated, generating a new attack sample using the new target attack method, or increasing the disturbance amplitude of the attack sample to obtain a new attack sample, and returning to execute S204.

[0101] The attack strategy includes: selecting a new target attack method based on the characteristics of the model to be evaluated, generating a new attack sample using the new target attack method, or increasing the disturbance amplitude of the attack sample to obtain a new attack sample.

[0102] Increase the perturbation amplitude of the attack sample, including: adjusting the pixel value of the image, introducing noise, or applying specific geometric transformations.

[0103] In one possible implementation, when the model to be evaluated is a computer vision model, during preliminary security testing, FGSM can be selected to generate attack samples. During the testing process using FGSM, if the security score is greater than the first set value, a new BIM is reselected to generate attack samples, and BIM is used to perform security testing on the model to be evaluated.

[0104] S211. Until the second stop condition is reached, it is determined that the security of the model to be evaluated meets the requirements, and the second stop condition includes: the multi-dimensional response data reaches the set multi-dimensional data; or, the ratio of the number of times the security score is greater than or equal to the first set value to the number of times the attack strategy is adjusted is greater than the second set value.

[0105] The set multi-dimensional data can be understood as the multi-dimensional data designed when the model to be evaluated meets the design requirements. If the multi-dimensional response data reaches the set multi-dimensional data, it means that the security of the model to be evaluated meets the requirements. At this time, the adversarial attack test is stopped and an optimized model to be evaluated is obtained. The optimized model to be evaluated is a model whose security meets the design requirements.

[0106] The second set value is a value that measures the proportion of times when the security score is greater than or equal to the first set value. For example, if the first set value is 90, then the ratio of the number of times the security score is greater than or equal to the first set value to the number of times the attack strategy is adjusted is the ratio of the number of times the security score is greater than or equal to 90 points to the total number of security scores.

[0107] The ratio of the number of times the security score is greater than or equal to the first set value to the number of times the attack strategy is adjusted is greater than the second set value, which may mean that the ratio of the number of times the security score is greater than or equal to 90 points to the total number of security scores is greater than the second set value, indicating that the security of the model to be evaluated meets the requirements. At this time, the anti-attack test is stopped to obtain an optimized model to be evaluated. The optimized model to be evaluated is a model whose security meets the design requirements.

[0108] In the embodiment of the present application, an adaptive attack cycle is implemented, and the attack strategy is continuously iterated until a predetermined attack effect is achieved or the attack is effectively defended by an intelligent algorithm.

[0109] The intelligent algorithm security inspection and evaluation method based on adversarial attacks provided in this public embodiment provides a comprehensive, real-time, adaptive, flexible and data-protected integrated solution. Through a multi-dimensional security scoring system, the security of the intelligent algorithm is comprehensively evaluated from multiple angles, and dynamic adjustment of the optimal balance between algorithm performance and security is achieved. Real-time monitoring ensures that the security performance of the algorithm under different conditions can be evaluated instantly, while adaptive attack adjustment dynamically adjusts the attack strategy according to the real-time response of the algorithm. At the same time, the continuous optimization and evaluation cycle mechanism ensures that the algorithm can continuously adapt to new security threats. In addition, compliance and data protection are emphasized to ensure that the intelligent algorithm meets industry security standards during the design and evaluation process, reducing compliance risks and the possibility of data leakage.

[0110] Compared with the existing testing techniques such as fuzz testing and compilation testing in traditional software testing, the security assessment method for the model provided in the embodiment of the present application adopts advanced deep learning technology to generate adversarial samples. These samples may be visually indistinguishable from normal data, but they can effectively test the security of the algorithm. This technology not only tests the conventional functional performance of the algorithm by simulating the adversarial means that attackers may use, but also explores the behavior patterns and safety margins of the algorithm under extreme or hostile conditions in depth. This method is closer to the advanced attack methods that attackers may use in the real world, thereby providing a more rigorous and comprehensive security assessment. This technology also places special emphasis on real-time monitoring and behavioral analysis during the evaluation process, which can accurately capture the response and changes of intelligent algorithms under adversarial attacks, which makes it possible to quickly identify and respond to security threats.

[0111] Figure 3 is a schematic diagram of a structure of a safety assessment device for a model in an embodiment of the present application, such as Figure 3 As shown, the security assessment device 30 for a model provided in an embodiment of the present application mainly includes: a model acquisition module 31, a method selection module 32, a sample generation module 33, a sample attack module 34, a data acquisition module 35 and a security analysis module 36.

[0112] Among them, the model acquisition module 31 is used to obtain the model to be evaluated; the method selection module 32 is used to select the target attack method based on the characteristics of the model to be evaluated; the sample generation module 33 is used to generate attack samples using the target attack method; the sample attack module 34 is used to input the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample; the data acquisition module 35 is used to obtain the multi-dimensional response data of the model to be evaluated in the process of processing the attack sample; the security analysis module 36 is used to perform quantitative analysis on the multi-dimensional response data to obtain a security score, which is used to indicate the security of the model to be evaluated.

[0113] The embodiment of the present application provides a security assessment device for a model. First, the model to be assessed is obtained, and then a target attack method is selected based on the characteristics of the model to be assessed, and an attack sample is generated using the target attack method; then, the model to be assessed is attacked using the attack sample, and multi-dimensional response data of the model to be assessed during the attack process is obtained; the multi-dimensional response data is quantitatively analyzed to obtain a security score, which is used to indicate the security of the model to be assessed. This implements an assessment of the security of the model when facing malicious attacks, and improves the accuracy and comprehensiveness of the security assessment.

[0114] In a possible implementation, the method selection module 32 is specifically used to select an image attack as a target attack method when the model to be evaluated is a computer vision model, wherein the image attack method includes at least one of the following: fast gradient symbol FGSM, basic iterative method BIM, projected gradient descent method PGD, momentum iterative fast gradient symbol method MI-FGSM; when the model to be evaluated is a natural language model, select a text attack method as a target attack method, wherein the text attack method includes at least one of the following: text deception attack model TextFooler, sentence compression interpretation network SCPN, text perturbator TextBugger, colorful snake VIPER; when the model to be evaluated is an audio model, select a voice attack as a target attack method, wherein the voice attack method includes at least one of the following: automatic speech recognition attack method ASRAttack, Bob voice forgery attack method FakeBob, and Carlini-Wagner attack method C&W.

[0115] In one possible implementation, the sample generation module 33 is specifically used to analyze the input and output behaviors, data processing logic, data processing methods, and decision execution process of the model to be evaluated to obtain security vulnerabilities of the model to be evaluated; and use a targeted attack method to generate attack samples that attack security vulnerabilities.

[0116] In a possible implementation, the security analysis module 36 is specifically used to obtain pre-set scoring standards for each dimension; perform quantitative analysis based on the response data and scoring standards of each dimension to obtain a score for each dimension; and perform weighted calculation on the score for each dimension to obtain a security score.

[0117] In a possible implementation, it also includes: a model optimization module, which is used to design a model optimization strategy based on the multi-dimensional response data when the security score is less than the first set value, and the model optimization strategy includes at least one of the following: adjusting the model parameters of the model to be evaluated, improving the network structure of the model to be evaluated, and adding a new defense mechanism to the model to be evaluated; optimizing the model to be evaluated using the model optimization strategy to obtain the optimized model to be evaluated, and returning to execute the step of inputting the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample until the security score is greater than or equal to the first set value.

[0118] In a possible implementation, it also includes: an attack strategy optimization module, which is used to adjust the attack strategy when the security score is greater than or equal to a first set value, and the attack strategy includes: selecting a new target attack method based on the characteristics of the model to be evaluated, generating a new attack sample using the new target attack method, or increasing the disturbance amplitude of the attack sample to obtain a new attack sample, and returning to execute the step of inputting the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample; until the second stop condition is reached, it is determined that the security of the model to be evaluated meets the requirements, and the second stop condition includes: the multi-dimensional response data reaches the set multi-dimensional data; or, the ratio of the number of times the security score is greater than or equal to the first set value to the number of times the attack strategy is adjusted is greater than the second set value.

[0119] In a possible implementation, it also includes: a preventive measures formulation module, which is used to collect security assessment data of the model to be evaluated under different attack strategies, wherein the security assessment data includes at least one of the following: attack samples, multi-dimensional response data, scores of each dimension, and security scores; extracting feature vectors related to the security performance of the model to be evaluated from the security assessment data; using the feature vectors to predict the predicted multi-dimensional response data of the model to be evaluated when facing new adversarial attacks; and formulating preventive measures for the model to be evaluated based on the predicted multi-dimensional response data.

[0120] The security assessment device for a model provided in the embodiment of the present application can execute the security assessment method for a model provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0121] Figure 4 is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include a security assessment device for a model, such as Figure 4 As shown, the electronic device 400 includes a processor 410, a memory 420, an input device 430, and an output device 440; the number of the processor 410 in the electronic device can be one or more. Figure 4 A processor 410 is taken as an example; the processor 410, the memory 420, the input device 430 and the output device 440 in the electronic device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0122] The memory 420, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the security assessment method for models in the embodiment of the present invention. The processor 410 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 420, that is, implements the security assessment method for models provided in the embodiment of the present invention.

[0123] The memory 420 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 420 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 420 may further include a memory remotely arranged relative to the processor 410, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0124] The input device 430 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 440 may include a display device such as a display screen.

[0125] This embodiment also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to implement the security assessment method for a model provided by the embodiment of the present invention.

[0126] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided in an embodiment of the present invention are not limited to the method operations described above, and can also execute related operations in the security assessment method for a model provided in any embodiment of the present invention.

[0127] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0128] It is worth noting that in the above-mentioned embodiment of the security assessment device for the model, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0129] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0130] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A security assessment method for a model, characterized in that: The method comprises: Get the model to be evaluated; Selecting a target attack method based on the features of the model to be evaluated; Generate an attack sample using the target attack method; Inputting the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample; Acquire multi-dimensional response data of the model to be evaluated in the process of processing the attack sample; The multi-dimensional response data is quantitatively analyzed to obtain a safety score, where the safety score is used to indicate the safety of the model to be evaluated.

2. The method according to claim 1, characterized in that The target attack method is selected based on the features of the model to be evaluated, including: When the model to be evaluated is a computer vision model, an image attack method is selected as the target attack method, wherein the image attack method includes at least one of the following: fast gradient sign FGSM, basic iterative method BIM, projected gradient descent method PGD, momentum iterative fast gradient sign method MI-FGSM; In the case where the model to be evaluated is a natural language model, a text attack method is selected as the target attack method, wherein the text attack method includes at least one of the following: a text deception attack model TextFooler, a sentence compression interpretation network SCPN, a text perturbator TextBugger, and a colorful snake VIPER; When the model to be evaluated is an audio model, a speech attack method is selected as the target attack method, wherein the speech attack method includes at least one of the following: automatic speech recognition attack method ASR Attack, Bob's voice forgery attack method FakeBob, and Carlini-Wagner attack method C&W.

3. The method according to claim 1, characterized in that Generating an attack sample using the target attack method includes: Analyze the input and output behaviors, data processing logic, data processing methods, and decision-making execution process of the model to be evaluated to obtain security vulnerabilities of the model to be evaluated; The target attack method is used to generate an attack sample for attacking the security vulnerability.

4. The method according to claim 1, characterized in that: The multi-dimensional response data is analyzed to obtain a safety score, including: Obtain the pre-set scoring criteria for each dimension; Performing quantitative analysis based on the response data of each dimension and the scoring criteria to obtain a score for each dimension; The scores of the various dimensions are weighted to obtain a safety score.

5. The method according to claim 1, characterized in that Also includes: In the case where the security score is less than the first set value, a model optimization strategy is designed based on the multi-dimensional response data, and the model optimization strategy includes at least one of the following: adjusting the model parameters of the model to be evaluated, improving the network structure of the model to be evaluated, and adding a new defense mechanism to the model to be evaluated; The model to be evaluated is optimized using the model optimization strategy to obtain an optimized model to be evaluated, and the step of inputting the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample is returned to the step until the security score is greater than or equal to a first set value.

6. The method according to claim 5, characterized in that Also includes: When the security score is greater than or equal to the first set value, adjusting the attack strategy, the attack strategy comprising: selecting a new target attack method based on the characteristics of the model to be evaluated, generating a new attack sample using the new target attack method, or increasing the disturbance amplitude of the attack sample to obtain a new attack sample, and returning to execute the step of inputting the attack sample into the model to be evaluated so that the model to be evaluated processes the attack sample; Until the second stopping condition is reached, it is determined that the security of the model to be evaluated meets the requirements, and the second stopping condition includes: the multi-dimensional response data reaches the set multi-dimensional data; or, the ratio of the number of times the security score is greater than or equal to the first set value to the number of times the attack strategy is adjusted is greater than the second set value.

7. The method according to any one of claims 1 to 6, characterized in that: Also includes: Collect security assessment data of the model to be assessed under different attack strategies, wherein the security assessment data includes at least one of the following: the attack sample, the multi-dimensional response data, the scores of each dimension, and the security score; Extracting a feature vector related to the safety performance of the model to be evaluated from the safety evaluation data; Using the feature vector to predict the predicted multi-dimensional response data of the model to be evaluated when facing a new adversarial attack; Preventive measures for the model to be evaluated are formulated based on the predicted multi-dimensional response data.

8. A safety assessment device for a model, characterized in that: The device comprises: A model acquisition module is used to obtain the model to be evaluated; A method selection module, used for selecting a target attack method based on the characteristics of the model to be evaluated; A sample generation module, used to generate attack samples using the target attack method; A sample attack module, used for inputting the attack sample into the model to be evaluated, so that the model to be evaluated processes the attack sample; A data acquisition module, used to acquire multi-dimensional response data of the model to be evaluated in the process of processing the attack sample; The safety analysis module is used to perform quantitative analysis on the multi-dimensional response data to obtain a safety score, where the safety score is used to indicate the safety of the model to be evaluated.

9. An electronic device, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the security assessment method for a model as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the security assessment method for a model as described in any one of claims 1 to 7 is implemented.

Images