Model verification sample generation method, device, equipment and storage medium

Abstract

The present application provides a method, device, equipment and storage medium for generating a model verification sample, which relates to the field of artificial intelligence technology and can be applied to various scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving. The method includes: obtaining a target model, and a sample training set and a proxy model corresponding to the target model; performing a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample, and the first output result of the target model for the original training sample and the second output result for the first adversarial sample meet the first adversarial condition; taking the sample label of the original training sample as the expected output, performing a directional attack on the proxy model based on the first adversarial sample to obtain a second adversarial sample; performing an adversarial verification test on the second adversarial sample, and determining the second adversarial sample as a model verification sample when the adversarial verification result is a verification pass. The present application can effectively reduce the sample generation cost.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and storage medium for generating model verification samples. Background Art

[0002] Verification samples are a means of protecting the copyright of artificial intelligence models. Ideally, verification samples only produce preset outputs on the specified model to achieve copyright verification and protection of the specified model. Existing verification sample construction schemes require knowledge of the detailed structure and parameters of the model. The cost of constructing samples is high, which increases the risk of model information leakage. In addition, the model needs to be retrained and the model parameters and weights need to be adjusted during the sample construction process, which increases the cost of model protection and affects the robustness of the model. In addition, existing verification sample construction schemes are usually targeted at a specific type of model, such as a classification model or a model for image data, with poor generalization, and the samples have high mobility and low confidence. Summary of the invention

[0003] The present application provides a model verification sample generation method, apparatus, device and storage medium. The verification sample generation process does not depend on the model structure and parameters, does not require retraining of the model, avoids model structure adjustment and information leakage, significantly reduces the verification sample generation cost, and has good generalization.

[0004] In one aspect, the present application provides a method for generating a model verification sample, the method comprising:

[0005] Obtain a target model, and a sample training set and a proxy model corresponding to the target model, wherein the sample training set includes original training samples and their corresponding sample labels;

[0006] Performing a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample, wherein a first output result of the target model for the original training sample and a second output result for the first adversarial sample satisfy a first adversarial condition;

[0007] Taking the sample label of the original training sample as the expected output, performing a targeted attack on the proxy model based on the first adversarial sample to obtain a second adversarial sample;

[0008] Performing an adversarial verification test on the second adversarial sample to obtain an adversarial verification result;

[0009] When the adversarial verification result is verification passed, the second adversarial sample is determined as a model verification sample.

[0010] Another aspect provides a model verification sample generation device, the device comprising:

[0011] The first acquisition module is used to acquire a target model, a sample training set corresponding to the target model, and a proxy model, wherein the sample training set includes original training samples and their corresponding sample labels;

[0012] A non-directional attack module: used for performing a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample, wherein a first output result of the target model for the original training sample and a second output result for the first adversarial sample satisfy a first adversarial condition;

[0013] A targeted attack module: used for performing a targeted attack on the proxy model based on the first adversarial sample with the sample label of the original training sample as the expected output, to obtain a second adversarial sample;

[0014] An adversarial verification test module: used to perform an adversarial verification test on the second adversarial sample to obtain an adversarial verification result;

[0015] Target sample determination module: used to determine the second adversarial sample as a model verification sample when the adversarial verification result is verification passed.

[0016] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the model verification sample generation method as described above.

[0017] On the other hand, a computer-readable storage medium is provided, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by a processor to implement the model verification sample generation method as described above.

[0018] On the other hand, a model verification sample generation terminal is provided, the terminal includes a processor and a memory, the device includes a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to implement the model verification sample generation method as described above.

[0019] On the other hand, a server is provided, the server includes a processor and a memory, the device includes a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or the at least one program is loaded and executed by the processor to implement the model verification sample generation method as described above.

[0020] On the other hand, a computer program product or a computer program is provided, which includes computer instructions, and when the computer instructions are executed by a processor, the model verification sample generation method as described above is implemented.

[0021] The model verification sample generation method, device, equipment, storage medium, terminal, server and computer program product provided in this application have the following technical effects:

[0022] The present application obtains a target model, a sample training set and a proxy model corresponding to the target model, and performs a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample, wherein the first output result of the target model for the original training sample and the second output result for the first adversarial sample meet the first adversarial condition; then, the sample label of the original training sample is used as the expected output, and the proxy model is subjected to a directional attack based on the first adversarial sample to obtain a second adversarial sample, and then the second adversarial sample is subjected to an adversarial verification test to obtain an adversarial verification result, and when the adversarial verification result is a verification pass, the second adversarial sample is determined as a model verification sample. The technical solution of the present application does not rely on the model structure and parameters in the process of generating verification samples, does not require retraining the model, avoids model structure adjustment and information leakage, significantly reduces the cost of generating verification samples, improves model protection efficiency, and does not interfere with the robustness of the model. At the same time, the generation of verification samples is not limited by model types, task types, and data types, and has strong generalization. In addition, the first adversarial sample is corrected based on the proxy model to reduce the transferability of the verification sample, ensure the specificity of the verification sample for the target model, and thus improve the sample confidence. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0024] Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application;

[0025] Figure 2 It is a flowchart of a method for generating a model verification sample provided in an embodiment of the present application;

[0026] Figure 3 It is a flowchart of another model verification sample generation method provided in an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of a generation model verification sample provided by an embodiment of the present application;

[0028] Figure 5 It is a flowchart of another model verification sample generation method provided in an embodiment of the present application;

[0029] Figure 6 It is a schematic diagram of a process of generating a model verification sample provided by an embodiment of the present application;

[0030] Figure 7 It is a flowchart of another model verification sample generation method provided in an embodiment of the present application;

[0031] Figure 8 It is a structural schematic diagram of a model verification sample generation device provided in an embodiment of the present application;

[0032] Fig. 9 It is a hardware structure block diagram of an electronic device for a model verification sample generation method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0034] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, mechatronics and other technologies. Artificial intelligence software technologies mainly include computer vision technology, speech processing technology, natural language processing technology, as well as machine learning / deep learning, autonomous driving, smart transportation and other major directions.

[0035] Among them, machine learning (ML) is a multi-disciplinary interdisciplinary subject, involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and self-learning.

[0036] Computer Vision Technology (CV) Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instrument detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Computer vision technology usually includes model verification sample generation, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and map construction, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0037] The key technologies of speech technology include automatic speech recognition technology (ASR), text-to-speech technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the future development direction of human-computer interaction, among which speech has become one of the most promising human-computer interaction methods in the future.

[0038] Natural language processing (NLP) is an important direction in the fields of computer science and artificial intelligence. It studies various theories and methods that can achieve effective communication between people and computers using natural language. Natural language processing is a science that integrates linguistics, computer science, and mathematics. Therefore, research in this field will involve natural language, that is, the language people use in daily life, so it is closely related to the study of linguistics. Natural language processing technology usually includes text processing, semantic understanding, machine translation, robot question answering, knowledge graph and other technologies.

[0039] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, Internet of Vehicles, automatic driving, smart transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role. This application mainly involves technologies such as model verification sample generation, semantic understanding, machine learning and deep learning, which are specifically explained through the following embodiments.

[0040] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0041] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server comprising a series of steps or submodules is not necessarily limited to those steps or submodules clearly listed, but may include other steps or submodules that are not clearly listed or inherent to these processes, methods, products or devices.

[0042] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0043] Identifying Sample: Identifying samples are a means of protecting the copyright of artificial intelligence models. They can only produce pre-set outputs on specific models. Based on this feature, verification samples can be used to protect model copyright.

[0044] Adversarial Attack: Adversarial attack was proposed in 2013. Adversarial attack can generate adversarial samples that act on deep learning models. These samples can mislead the model to produce directional or non-directional outputs. From the perspective of attack purpose, there are two types of attacks: directional attack and non-directional attack. The former causes the attacked network to produce wrong outputs, while the latter causes the attacked network to produce specific outputs.

[0045] Adversarial sample: A normal sample with a specific adversarial perturbation, which is generated by the adversarial attack algorithm to deceive the target neural network. Usually, the perturbation is difficult to be detected by the naked eye. Adversarial samples can mislead the neural network to produce wrong outputs, and their existence reveals the vulnerability of the neural network.

[0046] Fast Gradient Method (FGM): Constructs adversarial samples based on the gradient ascent method. It is a basic single-step or single-iteration algorithm.

[0047] Fast Gradient Sign Method (FGSM): In a white-box environment, the model's derivative with respect to the input is obtained, and then the sign function is used to obtain its specific gradient direction. Then, the perturbation is added to the original input to obtain the sample under FGSM attack.

[0048] Projected Gradient Descent (PGD): Compared with FGSM, which only performs one iteration, PGD performs multiple iterations, taking a small step each time, and each iteration will clip the disturbance to the specified range.

[0049] See also Figure 1 , Figure 1 is a schematic diagram of an application environment provided by an embodiment of the present application, such as Figure 1 As shown, the application environment may include at least a server 01 and a terminal 02. In actual applications, the server 01 and the terminal 02 may be directly or indirectly connected via wired or wireless communication to achieve interaction between the terminal 02 and the server 01, and this application does not limit this.

[0050] In the embodiment of the present application, server 01 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Specifically, the server may include physical devices, which may specifically include network communication submodules, processors, and memories, etc., and may also include software running in physical devices, which may specifically include applications, etc.

[0051] Specifically, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to achieve data computing, storage, processing and sharing. It distributes computing tasks on a resource pool composed of a large number of computers, so that various application systems can obtain computing power, storage space and information services as needed. The network that provides resources is called "cloud". Among them, artificial intelligence cloud service is generally also called AIaaS (AI as a Service, Chinese for "AI as a Service"). This is a mainstream service mode of artificial intelligence platform. Specifically, AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to opening an AI theme mall: all developers can access one or more artificial intelligence services provided by the platform through API interfaces. Some senior developers can also use the AI ​​framework and AI infrastructure provided by the platform to deploy and operate their own cloud artificial intelligence services.

[0052] In the embodiment of the present application, the terminal 02 may include physical devices such as smart phones, desktop computers, tablet computers, laptop computers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, intelligent voice interaction devices, smart home appliances, smart wearable devices, vehicle-mounted terminal devices, aircraft, etc., and may also include software running in physical devices, such as applications. Specifically, the server 01 can be used to perform non-directional attacks on the target model based on the original training samples and initial adversarial perturbations in the training sample set, and based on the obtained first adversarial sample, perform a directional attack on the proxy model with the sample label as the target to correct the first adversarial sample, thereby obtaining a model verification sample. At the same time, a service for model copyright verification and protection based on the model verification sample can also be provided. The terminal 02 can be used to send the model identification of the target model and receive the model verification results, etc.

[0053] Furthermore, it is understandable that Figure 1 What is shown is merely an application environment of a model verification sample generation method, and the application environment may include more or fewer nodes, and this application does not impose any limitation thereto.

[0054] The application environment involved in the embodiment of the present application, or the server 01 and the terminal 02 in the application environment, can be a distributed system formed by connecting a client and multiple nodes (any form of computing devices in the access network, such as a server, a user terminal) through network communication. The distributed system can be a blockchain system, which can provide the above-mentioned model verification sample generation service, model training service and data storage service, etc.

[0055] The following is an introduction to a model verification sample generation method of this application based on the above application environment. The embodiment of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, etc. Please refer to Figure 2 , Figure 2 It is a flow chart of a model verification sample generation method provided in an embodiment of the present application. This specification provides method operation steps such as the embodiment or flow chart, but may include more or fewer operation steps based on conventional or non-creative labor. The order of steps listed in the embodiment is only one way of executing the steps among many orders, and does not represent the only order of execution. When the actual system or server product is executed, it can be executed in the order shown in the embodiment or the figure or in parallel (for example, in a parallel processor or multi-threaded processing environment). Specifically, Figure 2 As shown, the method may include the following steps S201-S209.

[0056] S201: Obtain a target model, and a sample training set and a proxy model corresponding to the target model.

[0057] Specifically, the target model can be various types of machine learning models or deep learning models, such as classification task models and generative models. Among them, the sample training set can be a subset of the training set used in the training and learning process of the target model, including the original training samples and their corresponding sample labels. In some embodiments, the sample training set is a pre-screened positive sample set. The proxy model can be an old version model or a new version model that is different from the current version of the target model, or a model similar to the target model, such as a model with a similar model structure or the same model task.

[0058] S203: Perform a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample.

[0059] The first output result of the target model for the original training sample and the second output result for the first adversarial sample satisfy the first adversarial condition. In some cases, the first adversarial condition is satisfied when the first output result is different from the second output result. The non-directional attack here means that the original training sample and the adversarial sample are used as the input of the target model respectively. The expected result of the non-directional attack processing is that the output result for the original training sample is different from the output result for the adversarial sample. For example, if the target model is a classification model, it is expected that the classification result corresponding to the original training sample is different from the classification result corresponding to the adversarial sample. Specifically, the initial adversarial perturbation includes initial noise to be added to the original training sample, which may correspond to at least one of the sample attributes of the original training sample and the model type attribute of the target model; in some cases, corresponding to the sample attributes, for example, if the original training sample is a picture, the initial adversarial perturbation may include the noise of all or part of the pixels in the picture; if the original training sample is text, the initial adversarial perturbation may be the noise of the embedding word vector of the text; in other cases, corresponding to the model attributes, for example, if the target model is a classification model, the initial adversarial perturbation may be the noise of the target area or target segment in the original training sample; if the target model is a generative model, the initial adversarial perturbation may be the noise of the original training sample as a whole. Specifically, the initial adversarial perturbation may be a random perturbation, such as random noise for each pixel of the picture. In some cases, the initial adversarial perturbation w is constrained to be within a certain range, such as ||w|| n <∈. Accordingly, S203 may include the following steps S2031-S2033.

[0060] S2031: Perform noise processing on the original training sample based on the initial adversarial perturbation to obtain an initial adversarial sample.

[0061] S2032: Input the original training sample and the initial adversarial sample into the target model respectively to obtain a first output result corresponding to the original training sample and a second output result corresponding to the initial adversarial perturbation.

[0062] S2033: When the first output result and the second output result satisfy the first adversarial condition, use the initial adversarial sample as the first adversarial sample.

[0063] Specifically, the initial adversarial perturbation is added to the original training sample to perform noise processing to obtain the initial adversarial sample, and then the original training sample and the obtained initial adversarial sample are input into the target model to obtain the first output result and the second output result. If the first output result is different from the second output result and satisfies the first adversarial condition, the initial adversarial sample can be directly used as the first adversarial sample.

[0064] Furthermore, S203 may also include the following steps S2034-S2036.

[0065] S2034: When the first output result and the second output result do not satisfy the first confrontation condition, determine the current gradient of the target model according to the first output result and the second output result.

[0066] S2035: Iteratively update the initial adversarial perturbation according to the current gradient of the target model and the initial adversarial sample to obtain a first adversarial perturbation.

[0067] S2036: Use the adversarial sample corresponding to the first adversarial perturbation as the first adversarial sample.

[0068] Specifically, the first adversarial sample is a sample that adds noise to the original training sample based on the first adversarial perturbation. If the first output result and the second output result are the same, it is determined that the first adversarial condition is not met, and the preset attack algorithm is used to perform perturbation update, wherein the preset attack algorithm may include but is not limited to FGM, FGSM or PGD, etc. Specifically, the loss function (such as the mean square error function) of the target model is used to calculate the distance between the first output result and the second output result, and the gradient is returned to the target model to obtain the current gradient. When the parameters and weights of the target model are fixed, the initial adversarial perturbation is updated based on the current gradient and the preset attack algorithm, and the above S2031 and S2032 are repeatedly executed based on the updated adversarial perturbation to obtain the first output result of the original training sample and the updated second output result corresponding to the updated adversarial sample. If the first output result and the updated second output result are different, it is determined that the updated adversarial perturbation is the first adversarial perturbation. If the first output result and the updated second output result are the same, the current gradient is recalculated, and the perturbation update is performed until the first output result and the second output result are different, or until the number of iterations reaches the first number of iterations, so as to obtain the first adversarial perturbation and the first adversarial sample.

[0069] In other cases, the first output result and the second output result satisfy the first adversarial condition specifically when the number of perturbation iterations reaches the second number of iterations, and the second number of iterations can be set based on actual needs. Accordingly, after the first output result and the second output result are generated based on the original training sample and the initial adversarial sample, the distance between the two is directly calculated using the loss function of the target model, and the gradient is calculated back to the target model to obtain the current gradient, and the current gradient and the initial adversarial sample are brought into the preset attack algorithm to determine the updated adversarial perturbation and the updated adversarial sample, and then the original training sample and the updated adversarial sample are used as the input of the target model, and S2031, S2032, gradient calculation and the above steps of using the preset attack algorithm to perform perturbation updates and adversarial sample updates are repeated, and the cycle is repeated until the current number of perturbation iterations reaches the second number of iterations, and the updated adversarial perturbation obtained in this iteration is used as the first adversarial perturbation, and the updated adversarial sample is used as the first adversarial sample.

[0070] In one embodiment, the preset attack algorithm is the PGD algorithm, that is, the PGD algorithm is used to update the counter-perturbation, and its expression is shown in the following formula, where x {t} represents the adversarial sample in the tth iteration, w {t} represents the adversarial perturbation in the tth iteration, α is the step size of the attack algorithm, for example, 0.01, sign() is the sign function, and L is the target model M t The loss function is, I is the original training sample, M t (I) is the first output result, M t (I+w {t} ) is the second output result in the tth iteration, and the clip operation is a clipping operation that constrains the adversarial perturbation to [-∈,∈]. By iterating the second iteration number, the first adversarial sample after the non-directional attack can be obtained.

[0071]

[0072] w {t+1} =clip(x {t+1} -I,-∈,∈)

[0073] S205: Taking the sample label of the original training sample as the expected output, a targeted attack is performed on the proxy model based on the first adversarial sample to obtain a second adversarial sample.

[0074] The goal of the targeted attack is to make the output result of the proxy model for the first adversarial sample the same as the sample label. In other words, the targeted attack here refers to taking the adversarial sample as the input of the proxy model, and the expected result of the attack processing is that the output result of the proxy model for the adversarial sample is the same as the sample label. For example, if the proxy model is a classification model, the classification result corresponding to the adversarial sample is expected to be the same as the sample label, so that the adversarial sample has attack specificity for the target model, but does not have transferability on the proxy model, thereby improving the confidence in the recognition of the target model.

[0075] In practical applications, S205 may include the following steps S2051-S2052.

[0076] S2051: Input the first adversarial sample into the proxy model to obtain a reference output result corresponding to the first adversarial sample.

[0077] S2052: When the reference output result and the sample label satisfy the second adversarial condition, use the first adversarial sample as the second adversarial sample.

[0078] In some cases, if the second adversarial condition is satisfied when the reference output result is the same as the sample label, the first adversarial sample is used as the second adversarial sample. If the reference output result is different from the sample label, that is, the second adversarial condition is not satisfied, S205 also includes the following steps S2053-S2055.

[0079] S2053: When the reference output result and the sample label do not satisfy the second adversarial condition, determine the current gradient of the proxy model according to the reference output result and the sample label.

[0080] S2054: Iteratively update the first adversarial perturbation according to the current gradient of the proxy model and the initial adversarial sample to obtain a second adversarial perturbation.

[0081] S2055: Use the adversarial sample corresponding to the second adversarial perturbation as the second adversarial sample.

[0082] Specifically, the second adversarial sample is a sample that adds noise to the original training sample based on the second adversarial perturbation. If the reference output result is different from the sample label, the preset attack algorithm is used to further perform the perturbation update, wherein the preset attack algorithm may include but is not limited to FGM, FGSM or PGD, etc. The preset attack algorithm in S205 may be the same as or different from the preset attack algorithm in S203. Specifically, the loss function of the proxy model is used to calculate the distance between the reference output result and the sample label, and the gradient is returned to the proxy model to calculate the current gradient. When the proxy model parameters and weights are fixed, the first adversarial perturbation is updated based on the current gradient and the preset attack algorithm to obtain the updated first adversarial perturbation and the updated first adversarial sample. The above S2051 is repeatedly executed using the updated first adversarial sample. If the updated reference output result is the same as the sample label, the iteration is terminated. If not, the gradient is recalculated and the perturbation update is performed until the reference output result is the same as the sample label, or until the number of iterations reaches the third number of iterations, so as to obtain the second adversarial perturbation and the second adversarial sample. The third number of iterations here may be the same as the aforementioned first number of iterations, or may be different.

[0083] In other cases, the reference output result and the sample label satisfy the second adversarial condition specifically when the number of perturbation iterations reaches the fourth iteration number. The fourth iteration number can be set based on actual needs. The fourth iteration number in S205 can be the same as or different from the fourth iteration number in S203. Accordingly, after the reference output result is generated based on the first adversarial sample, the distance between the two is directly calculated using the loss function of the proxy model, and the gradient is calculated back to the proxy model to obtain the current gradient. The current gradient and the first adversarial sample are brought into the preset attack algorithm to determine the updated first adversarial perturbation and the updated first adversarial sample. The updated first adversarial sample is then used as the input of the proxy model, and S2051, gradient calculation, and the above steps of using the preset attack algorithm to perform perturbation updates and adversarial sample updates are repeated, and the cycle is repeated until the current number of perturbation iterations reaches the fourth iteration number, and the updated first adversarial perturbation obtained in this iteration is used as the second adversarial perturbation, and the updated first adversarial sample is used as the second adversarial sample.

[0084] In one embodiment, the preset attack algorithm is the PGD algorithm, that is, the PGD algorithm is used to update the countermeasure disturbance, and its expression is shown in the following formula, where x′ {t} represents the first adversarial example in the tth iteration, w′ {t} represents the first adversarial perturbation in the tth iteration, α is the step size of the attack algorithm, for example, 0.01, sign() is the sign function, and L′ is the proxy model SM n The loss function is, I is the original training sample, SM t (I+w′ {t}) is the reference output result, GT is the sample label, and the clip operation is a clipping operation that constrains the updated second adversarial perturbation to [-∈,∈]. By iterating the fourth iteration, the first adversarial perturbation is corrected to obtain the second adversarial perturbation, and the second adversarial sample after the targeted attack is obtained, which produces an erroneous output in the target model and a normal output result in the proxy model.

[0085]

[0086] w′ {t+1} =clip(x′ {t+1} -I,-∈,∈)

[0087] It should be noted that other adversarial attack methods can also be used for perturbation optimization and correction, such as using a gradient-free optimization method to update adversarial samples, which is not limited to the above-mentioned PDG method.

[0088] S207: Perform an adversarial verification test on the second adversarial sample to obtain an adversarial verification result.

[0089] S209: When the adversarial verification result is verification passed, the second adversarial sample is determined as a model verification sample.

[0090] Specifically, for part or all of the original training samples in the sample training set, based on the aforementioned steps S201-209, multiple model verification samples are generated.

[0091] The technical solution of the present application does not rely on the model structure and parameters in the process of generating verification samples, does not need to retrain the model, avoids model structure adjustment and information leakage, significantly reduces the cost of generating verification samples, improves the efficiency of model protection, and does not interfere with the robustness of the model. At the same time, the generation of verification samples is not restricted by model types, task types, and data types, and has strong generalization. In addition, the first adversarial sample is corrected based on the proxy model to reduce the transferability of the verification sample, ensure the specificity of the verification sample for the target model, and thus improve the sample confidence.

[0092] Based on some or all of the above implementations, in the present application embodiment, after S207, please refer to Figure 3 , the method also includes the following steps.

[0093] S211: When the adversarial verification result is failed, the steps of performing a non-directional attack on the target model, performing a directional attack on the proxy model, and performing an adversarial verification test are executed in a loop until an updated second adversarial sample with an adversarial verification result of passed is obtained;

[0094] S213: Use the updated second adversarial sample as a model verification sample.

[0095] Specifically, if the adversarial verification fails, the initial adversarial perturbation is replaced by the second adversarial perturbation, and the aforementioned S203-207 is repeated, and the adversarial verification result is executed in a loop until the obtained adversarial verification result is verified, and the second adversarial sample that passes the verification is used as the model verification sample. Specifically, the original training sample and the second adversarial sample are respectively input into the target model, and a non-directional attack is performed on the target model, and the adversarial perturbation is further updated to obtain the first adversarial perturbation and the first adversarial sample updated after the verification test, and a directed attack is performed on the proxy model based on the first adversarial sample updated after the verification test, and the adversarial perturbation is further corrected to obtain the second adversarial perturbation and the second adversarial sample updated after the verification test, and then the second adversarial sample updated after the verification test is subjected to an adversarial verification test, and if the verification passes, it is used as a model verification sample, and if it fails, it is iterated in a loop until the verification passes.

[0096] In some embodiments, the adversarial verification results include white box test results and black box test results. Accordingly, the adversarial verification test includes white box test and black box test. Please refer to Figure 5 , S209 may include the following steps S2091-S2094.

[0097] S2091: Use the target model and the proxy model to perform a white-box test on the second adversarial sample to obtain a white-box test result.

[0098] S2092: Perform a black box test on the second adversarial sample using the verification model to obtain a black box test result.

[0099] S2093: When both the white box test result and the black box test result satisfy corresponding test conditions, determine that the adversarial verification result is verification passed.

[0100] S2094: When any one of the white box test result and the black box test result does not satisfy the corresponding test condition, determine that the adversarial verification result is verification failure.

[0101] Specifically, the generated second adversarial sample is tested with white box and black box. The white box test is to use the target model M t and the proxy model SM n Test on the second adversarial sample. If the second adversarial sample can mislead the target model M t , that is, the target model M t The output for the second adversarial sample is different from the output for the original training sample, but in the proxy model SM n If there is a correct output on the , that is, the output of the proxy model for the second adversarial sample is the same as the sample label, then the white box test is passed. Otherwise, the white box test is not passed, and the process returns to the aforementioned step S203, and the non-directional attack, directional attack and adversarial verification test steps are repeated to continue correcting the adversarial perturbation and adversarial samples. The black box test is to use the verification model SMb The second adversarial sample is tested, wherein the verification model is a proxy model that is not targeted based on the first adversarial sample, and the proxy model SM that has not been input into the second adversarial sample is used. b If the correct output is still obtained, that is, the output result is the same as the sample label, the black box test is passed. Otherwise, the black box test is not passed, and the process returns to the aforementioned step S203 to continue correcting the adversarial sample. If the second adversarial sample passes all adversarial verification tests, the sample is output as the model verification sample of the model.

[0102] In some embodiments, a proxy model may be set to correct the disturbance and obtain an adversarial sample to be tested for adversarial verification. In other embodiments, two or more proxy models may be set to correct the disturbance in multiple ways. Specifically, the first adversarial sample is input into the first proxy model SM1, and after executing the above S205, the second adversarial sample and the second adversarial disturbance for the first proxy model are obtained. Then, the second adversarial sample for the first proxy model is input into the second proxy model SM2, and after executing the above S205, the second adversarial sample and the second adversarial disturbance for the second proxy model are obtained. And so on, until all proxy models are traversed to obtain the second adversarial sample to be tested for adversarial verification. If the verification passes, the model verification sample is obtained. If the verification fails, the target model M is re-output. t For non-targeted attacks, please refer to Figure 4 , Figure 4 The schematic diagram of generating the model verification sample in this embodiment is shown. The first adversarial sample and the second adversarial sample generated in the process of obtaining the model verification sample are the verification samples in the training.

[0103] It can be understood that the first adversarial perturbation is equivalent to a verification watermark for the target model, which can cause the target model to produce incorrect prediction results. At the same time, the corresponding first adversarial sample is input into the proxy model, and its true label is used as the attack target to carry out a targeted attack on the proxy model, thereby correcting the verification watermark so that it only acts on the target model and does not mislead other models to produce erroneous results. The above process is executed in a loop, that is, after a certain number of iterations, the updated verification watermark is subjected to white-box and black-box tests. The verification watermark that passes the test is a specific model verification watermark for the target model, which can be superimposed on the original training sample to form a specific verification sample for the target model, so as to achieve effective protection of the model copyright.

[0104] In some embodiments, the target model is a deep learning model. Figure 6, the generation process of the model verification sample is as follows: 1. First, confirm the deep learning model to be protected, at least one proxy model and the original training sample; 2. If the target model has a specified model ID, the model ID is used as the ID embedded in the model verification sample, otherwise, the model file is hashed and the obtained binary character is used as the model ID; 3. Then start to build the model verification sample, and conduct a non-directional attack on the deep learning model to be protected to obtain the first adversarial sample that can mislead the network to be protected; based on the first adversarial sample, conduct an adversarial attack on the proxy model with the true label of the original training sample as the directional label in turn, defend the interference of the first adversarial sample on the proxy model, avoid the model migration of the model verification sample, so that the model verification sample can only act on the deep learning model to be protected; 4. Test the model verification sample, output the samples that pass the white box test and the black box test as the trained model verification sample, and for the samples that fail the training, re-perform the aforementioned non-directional attack, directional attack and test steps until the output requirements are met or the maximum number of iterations is reached.

[0105] Based on some or all of the above implementations, in the present application embodiment, after S209, please refer to Figure 7 The method also includes the step of using the verification sample to protect the model copyright, which may specifically include the following steps S301-S307.

[0106] S301: Obtain a verification sample set for a target model, a model to be verified, and a target model identifier of the target model.

[0107] Among them, the verification sample set includes multiple model verification samples and original training samples. The original training samples in the verification sample set can be samples that match the model verification samples, or they can be other samples. That is, after obtaining the model verification samples of the model, they are mixed with normal original training samples to obtain the verification sample set. In some cases, the original training samples in the verification sample set are positive samples. The model to be verified refers to the model to be copyright verified. Whether its output results match the output results of the target model can be verified based on the verification sample set. If they match, it is determined that it is similar to the model file of the target model, otherwise it is not similar. The target model identifier is a unique identifier that characterizes the identity of the target model, which can be a model ID, such as an ID composed of binary digits.

[0108] S303: Input each model verification sample and each original training sample in the verification sample set into the model to be verified, respectively, to obtain a first verification output of each model verification sample and a second verification output of each original training sample.

[0109] S305: Construct a sequence identifier for the first verification output and the second verification output based on a preset order to obtain verification identifier information of the model to be verified.

[0110] Specifically, the model verification sample and the original training sample can be input into the model to be verified based on a preset order, or randomly input into the model to be verified, and then the verification outputs are sorted according to the preset order to construct the identification, and the obtained verification identification information can be a verification sequence. The verification output can be generated based on the comparison result of the output of the model to be verified for the sample and its sample label. Specifically, when the output of the model to be verified for the model verification sample is different from the sample label of the original training sample corresponding to the model verification sample, the first verification output is defined as the first value, and when they are the same, it is the second value, and the first value is different from the second value; when the output for the original training sample is consistent with its sample label, the second verification output is positioned as the third value, and when the output for the original training sample is inconsistent with its sample label, the second verification output is positioned as the fourth value, wherein the third value and the fourth value are different. In this way, the verification output of the model to be verified for each sample is sorted and combined based on the preset sorting to obtain the verification identification information. Among them, the preset order is determined based on the coding information in the target model identification, and can be consistent with the coding sorting in the target model identification. If the target model identification is a numerical ID, it is consistent with the numerical sorting in the target model identification.

[0111] In one embodiment, the original training samples in the verification sample set are positive samples, and the target model is identified as a binary ID. The preset order is determined based on the binary digital order in the binary ID. Correspondingly, the verification identification information of the model to be verified is also a verification sequence carrying binary information. Specifically, if a sample in the verification sample set is input to the model to be verified, and its output is different from the sample label, it is marked as 0, otherwise, it is marked as 1. Based on this, the verification sequence can carry binary information of the model copyright.

[0112] S307: Generate a model verification result based on the verification identification information and the target model identification.

[0113] Specifically, the verification identification information and the target model identification are compared. If the model verification result is that the verification identification information and the target model identification are inconsistent, it is determined that the model files of the model to be verified and the target model are not similar; if they are consistent, it is determined that the model files of the model to be verified and the target model are similar, and there is a risk of infringement of the model to be verified.

[0114] In some embodiments, before S301, the method may further include the following steps S401-S402 to obtain a model identifier and determine the number of samples in the validation sample set and a preset order.

[0115] S401: Perform hash processing on the model file of the target model to obtain a target model identifier.

[0116] S402: Determine the number of verification samples and the number of original samples in the verification sample set based on the binary encoding information of the target model identifier, and the preset ordering of the model verification samples and the original training samples.

[0117] Specifically, the target model identifier can be a character sequence obtained by hashing the model file, and the hash calculation can use an existing hash algorithm, which is not limited here. The binary encoding information may include the numerical value of each number in the identifier and the order of each number. Exemplarily, the target model identifier is 10101010, and the output of the model verification sample in the target model is different from the output (or sample label) of the corresponding original training sample. Its output is defined as 1, and the output of the original training sample is defined as 0. Then the verification sample set includes 8 samples, including 4 model verification samples and 4 original training samples. The preset order is that the first one is the model verification sample / first verification output, and the model verification sample / first verification output is alternately sorted with the original training sample / second verification output. Based on the sorting, the verification sample set is input into the model to be protected. If the verification output is also 10101010 after sorting, that is, the verification identification information is 10101010, then the model to be verified is infringing. If it is inconsistent, it is not infringing.

[0118] In other embodiments, the target model identification is determined based on the verification sample set, and the preset order is established based on actual needs before obtaining the target model identification, that is, firstly combine multiple model verification samples and original training samples, and determine the preset order of each sample or each sample output, and then construct the target model identification. Accordingly, obtaining the target model identification of the target model may include the following steps S3011-S3012.

[0119] S3011: Input each model verification sample and each original training sample in the verification sample set into the target model respectively to obtain the first target output of each model verification sample and the second target output of each original training sample.

[0120] S3012: Construct a sequence identifier for the first target output and the second target output based on a preset sorting to obtain a target model identifier of the target model.

[0121] Similar to the above, the output obtained by inputting the model verification sample into the target model is inconsistent with the sample label and is marked as the first value, i.e., the first target output. The output obtained by inputting the original training sample into the target model is consistent with the sample label and is marked as the third value, i.e., the second target output. In this way, each first target output and second target output are combined based on a preset sorting combination to obtain the target model identifier, such as the aforementioned binary ID 10101010.

[0122] Among the existing solutions, the following three methods are mainly used to implement model copyright protection based on watermark embedding: constructing special input samples through backdoor attacks, adjusting the model structure or parameters to construct verification samples, or marking the output results of the neural network based on deep template watermarks. However, the first method is based on backdoor attacks to achieve sample generation. When training the model, it is necessary to make the model produce abnormal outputs for specific inputs to construct special input samples. That is, it is necessary to know the entire structure and parameters of the model. The cost of constructing samples is high, which introduces the risk of model information leakage and affects the robustness of the model. The second method mainly uses the adjustment of the model structure to embed information. Similar to the first method, it requires detailed information of the model, and the cost of retraining the model is high, which interferes with the robustness of the model. The third method can only protect models for image data and has weak versatility. This application constructs a model verification sample based on adversarial attack technology, specifically combining non-directional adversarial attack and directional adversarial attack to generate verification samples. Through the verification sample, it is possible to construct information for identifying model identification, thereby realizing model copyright protection. By taking advantage of the fragile nature of the model itself, there is no need to retrain the model, which greatly saves the cost of model protection and improves the efficiency of model protection. Through high-confidence verification samples, the probability of misidentifying the target model is reduced. And there is no need to rely on and know the model structure and parameter information. It only needs to understand the gradient information of the model for a specific sample to protect it, which reduces the risk of model information leakage and does not affect the robustness of the model. In addition, by comparing the output of the original training sample and the output of the model verification sample to construct and identify the model identification, it does not depend on the specific model task and input data category, and can be applied to various types of machine learning or deep learning models, with a high degree of generalization performance.

[0123] The present application embodiment also provides a model verification sample generation device 800, such as Figure 8 As shown, Figure 8 A schematic diagram of the structure of a model verification sample generation device provided in an embodiment of the present application is shown, and the device may include the following modules.

[0124] The first acquisition module 10 is used to acquire a target model, and a sample training set and a proxy model corresponding to the target model, wherein the sample training set includes a plurality of original training samples and corresponding sample labels.

[0125] Non-directional attack module 20: used to perform non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample, wherein the first output result of the target model for the original training sample and the second output result for the first adversarial sample meet the first adversarial condition.

[0126] Targeted attack module 30: used to perform a targeted attack on the proxy model based on the first adversarial sample with the sample label of the original training sample as the expected output, to obtain a second adversarial sample.

[0127] The adversarial verification test module 40 is used to perform an adversarial verification test on the second adversarial sample to obtain an adversarial verification result.

[0128] The target sample determination module 50 is used to determine the second adversarial sample as a model verification sample when the adversarial verification result is verification passed.

[0129] In some embodiments, the non-targeted attack module 20 may include the following units.

[0130] Noise processing unit: used to perform noise processing on the original training samples based on the initial adversarial perturbation to obtain the initial adversarial samples.

[0131] Target model unit: used to input the original training sample and the initial adversarial sample into the target model respectively, to obtain the first output result corresponding to the original training sample and the second output result corresponding to the initial adversarial perturbation.

[0132] A first sample determination unit is configured to use the initial adversarial sample as the first adversarial sample when the first output result and the second output result satisfy the first adversarial condition.

[0133] In some embodiments, the non-targeted attack module 20 may further include the following units.

[0134] A first gradient determining unit is configured to determine a current gradient of a target model according to the first output result and the second output result when the first output result and the second output result do not satisfy a first confrontation condition.

[0135] The first perturbation updating unit is used to iteratively update the initial adversarial perturbation according to the current gradient of the target model and the initial adversarial sample to obtain the first adversarial perturbation.

[0136] The first sample determination unit is further configured to use an adversarial sample corresponding to the first adversarial perturbation as the first adversarial sample.

[0137] In some embodiments, the targeted attack module 30 may include the following units.

[0138] Proxy model unit: used to input the first adversarial sample into the proxy model to obtain a reference output result corresponding to the first adversarial sample.

[0139] The second sample determination unit is used to use the first adversarial sample as the second adversarial sample when the reference output result and the sample label meet the second adversarial condition.

[0140] In some embodiments, the targeted attack module 30 may further include the following units.

[0141] The second gradient determination unit is used to determine the current gradient of the proxy model according to the reference output result and the sample label when the reference output result and the sample label do not meet the second adversarial condition.

[0142] The second perturbation updating unit is used to iteratively update the first adversarial perturbation according to the current gradient of the proxy model and the initial adversarial sample to obtain the second adversarial perturbation.

[0143] The second sample determination unit is further configured to use the adversarial sample corresponding to the second adversarial perturbation as the second adversarial sample.

[0144] In some embodiments, the adversarial verification result includes a white box test result and a black box test result, and the target sample determination module 50 may further include the following units.

[0145] White box testing unit: used to use the target model and the proxy model to perform white box testing on the second adversarial sample to obtain the white box testing result.

[0146] Black box testing unit: used to perform black box testing on the second adversarial sample using the verification model to obtain a black box testing result, wherein the verification model is a proxy model that does not perform a targeted attack based on the first adversarial sample.

[0147] Adversarial verification unit: used to determine that the adversarial verification result is verified to be passed when both the white box test result and the black box test result meet the corresponding test conditions.

[0148] In some embodiments, the device may also include an iterative update module: used to, when the adversarial verification result is failed, loop through the steps of performing non-directional attacks on the target model, performing directed attacks on the proxy model, and performing adversarial verification tests, until an updated second adversarial sample with an adversarial verification result of passed is obtained.

[0149] The target sample determination module 50 is also used to use the updated second adversarial sample as a model verification sample.

[0150] In some embodiments, the device may further include the following modules.

[0151] The second acquisition module is used to obtain a verification sample set for the target model, a model to be verified, and a target model identifier of the target model after determining the second adversarial sample as a model verification sample when the adversarial verification result is verification passed, wherein the verification sample set includes multiple model verification samples and original training samples.

[0152] The model module to be verified is used to input each model verification sample and each original training sample in the verification sample set into the model to be verified, and obtain the first verification output of each model verification sample and the second verification output of each original training sample.

[0153] Identification construction module: used to construct a sequence identification for the first verification output and the second verification output based on a preset order, and obtain verification identification information of the model to be verified.

[0154] Model verification module: used to generate model verification results based on verification identification information and target model identification.

[0155] In some embodiments, the second acquisition module may include the following units.

[0156] Target output unit: used to input each model verification sample and each original training sample in the verification sample set into the target model respectively, to obtain the first target output of each model verification sample and the second target output of each original training sample.

[0157] An identification construction unit is used to construct a sequence identification for the first target output and the second target output based on a preset order, so as to obtain a target model identification of the target model.

[0158] In some embodiments, the device may further include the following modules.

[0159] Hash processing module: used to perform hash processing on the model file of the target model to obtain the target model identifier before obtaining the verification sample set for the target model, the model to be verified and the target model identifier of the target model.

[0160] Sorting determination module: used to determine the number of verification samples and the number of original samples in the verification sample set based on the binary encoding information of the target model identifier, as well as the preset sorting for the model verification samples and the original training samples.

[0161] It should be noted that the above device embodiment and method embodiment are based on the same implementation method.

[0162] An embodiment of the present application provides a model verification sample generation device, which includes a processor and a memory, in which at least one instruction or at least one program is stored, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the model verification sample generation method provided in the above method embodiment.

[0163] The memory can be used to store software programs and modules. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, application programs required for functions, etc.; the data storage area can store data created according to the use of the device, etc. In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory can also include a memory controller to provide the processor with access to the memory.

[0164] The method embodiments provided in the embodiments of the present application can be executed in electronic devices such as mobile terminals, computer terminals, servers or similar computing devices. Fig. 9 1 is a hardware structure block diagram of an electronic device for a model verification sample generation method provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 910 (the processor 910 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 930 for storing data, and one or more storage media 920 (such as one or more mass storage devices) for storing application programs 923 or data 922. Among them, the memory 930 and the storage medium 920 can be short-term storage or permanent storage. The program stored in the storage medium 920 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 910 can be configured to communicate with the storage medium 920 and execute a series of instruction operations in the storage medium 920 on the electronic device 900. The electronic device 900 may also include one or more power supplies 960, one or more wired or wireless network interfaces 950, one or more input and output interfaces 940, and / or, one or more operating systems 921, such as Windows Server TM , Mac OS X TM , Unix TM , LinuxTM, FreeBSDTM, etc.

[0165] The input / output interface 940 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 900. In one example, the input / output interface 940 includes a network adapter (Network Interface Controller, NIC), which may be connected to other network devices via a base station so as to communicate with the Internet. In one example, the input / output interface 940 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0166] It can be understood by those skilled in the art that Fig. 9 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig. 9 More or fewer components as shown, or with Fig. 9 Different configurations are shown.

[0167] An embodiment of the present application also provides a computer-readable storage medium, which can be set in an electronic device to store at least one instruction or at least one program related to a model verification sample generation method in a method embodiment. The at least one instruction or the at least one program is loaded and executed by the processor to implement the model verification sample generation method provided in the above method embodiment.

[0168] Optionally, in this embodiment, the storage medium may be located in at least one of the multiple network servers of the computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0169] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the above-mentioned various optional implementations.

[0170] It can be seen from the embodiments of the model verification sample generation method, device, equipment, server, terminal, storage medium and program product provided by the above-mentioned application that the application obtains the target model, the sample training set and the proxy model corresponding to the target model, and performs a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample, wherein the first output result of the target model for the original training sample and the second output result for the first adversarial sample meet the first adversarial condition; then, the sample label of the original training sample is used as the expected output, and the proxy model is subjected to a directional attack based on the first adversarial sample to obtain a second adversarial sample, and then the second adversarial sample is subjected to an adversarial verification test to obtain an adversarial verification result, and when the adversarial verification result is a verification pass, the second adversarial sample is determined as a model verification sample. The technical solution of the present application does not depend on the model structure and parameters in the verification sample generation process, does not need to retrain the model, avoids model structure adjustment and information leakage, significantly reduces the verification sample generation cost, improves the model protection efficiency, and does not interfere with the robustness of the model. At the same time, the generation of verification samples is not limited by model type, task type and data type, and has strong generalization. In addition, the first adversarial sample is corrected based on the proxy model to reduce the migration of the verification sample, effectively improve the specificity of the verification sample, and thus improve the sample confidence.

[0171] It should be noted that the above-mentioned sequence of the embodiments of the present application is for description only and does not represent the advantages and disadvantages of the embodiments. The above-mentioned specific embodiments of the present application are described. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0172] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, equipment and storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0173] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing the relevant hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0174] The above are only preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for generating a model verification sample, characterized in that: The method comprises: Obtain a target model, and a sample training set and a proxy model corresponding to the target model, wherein the sample training set includes original training samples and their corresponding sample labels; the original training samples are pictures or texts; Based on the original training sample and the corresponding initial adversarial perturbation, a non-directional attack is performed on the target model to obtain a first adversarial sample, wherein the non-directional attack refers to using the original training sample and the initial adversarial sample as inputs of the target model respectively, the initial adversarial sample is obtained by performing noise processing on the original training sample based on the initial adversarial perturbation, and a first output result of the target model for the original training sample and a second output result for the first adversarial sample satisfy a first adversarial condition; Taking the sample label of the original training sample as the expected output, a targeted attack is performed on the proxy model based on the first adversarial sample to obtain a second adversarial sample; the targeted attack means taking the first adversarial sample as the input of the proxy model, and the expected result of the targeted attack processing is that the output result of the proxy model for the second adversarial sample is the same as the sample label; Performing an adversarial verification test on the second adversarial sample to obtain an adversarial verification result; When the adversarial verification result is verification passed, the second adversarial sample is determined as a model verification sample.

2. The method according to claim 1, characterized in that The performing a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample includes: Performing noise processing on the original training sample based on the initial adversarial perturbation to obtain an initial adversarial sample; Inputting the original training sample and the initial adversarial sample into the target model respectively, obtaining a first output result corresponding to the original training sample and a second output result corresponding to the initial adversarial perturbation; When the first output result and the second output result satisfy a first adversarial condition, the initial adversarial sample is used as a first adversarial sample.

3. The method according to claim 2, characterized in that The method further comprises: When the first output result and the second output result do not satisfy a first confrontation condition, determining a current gradient of the target model according to the first output result and the second output result; Iteratively updating the initial adversarial perturbation according to the current gradient of the target model and the initial adversarial sample to obtain a first adversarial perturbation; An adversarial sample corresponding to the first adversarial perturbation is used as the first adversarial sample.

4. The method according to claim 1, characterized in that: The step of taking the sample label of the original training sample as the expected output and performing a targeted attack on the proxy model based on the first adversarial sample to obtain the second adversarial sample comprises: Inputting the first adversarial sample into the proxy model to obtain a reference output result corresponding to the first adversarial sample; When the reference output result and the sample label satisfy a second adversarial condition, the first adversarial sample is used as the second adversarial sample.

5. The method according to claim 4, characterized in that The method comprises: When the reference output result and the sample label do not satisfy the second adversarial condition, determining a current gradient of the proxy model according to the reference output result and the sample label; Iteratively updating the first adversarial perturbation according to the current gradient of the proxy model and the initial adversarial sample to obtain a second adversarial perturbation; The adversarial sample corresponding to the second adversarial perturbation is used as the second adversarial sample.

6. The method according to claim 1, characterized in that The adversarial verification result includes a white box test result and a black box test result. The adversarial verification test is performed on the second adversarial sample to obtain the adversarial verification result, which includes: Using the target model and the proxy model, perform a white-box test on the second adversarial sample to obtain the white-box test result; Performing a black box test on the second adversarial sample using the verification model to obtain the black box test result, wherein the verification model is a proxy model that does not perform a targeted attack based on the first adversarial sample; When both the white box test result and the black box test result satisfy corresponding test conditions, it is determined that the adversarial verification result is verification passed.

7. The method according to claim 1, characterized in that The method further comprises: In the case where the adversarial verification result is failed, the steps of performing a non-directional attack on the target model, performing a directional attack on the proxy model, and performing the adversarial verification test are executed in a loop until an updated second adversarial sample with an adversarial verification result of passed verification is obtained; The updated second adversarial sample is used as the model verification sample.

8. The method according to any one of claims 1 to 7, characterized in that When the adversarial verification result is that the verification is passed, after determining the second adversarial sample as a model verification sample, the method further includes: Acquire a verification sample set for the target model, a model to be verified, and a target model identifier of the target model, wherein the verification sample set includes a plurality of the model verification samples and the original training samples; Input each model verification sample and each original training sample in the verification sample set into the model to be verified, respectively, to obtain a first verification output of each model verification sample and a second verification output of each original training sample; Constructing a sequence identifier for the first verification output and the second verification output based on a preset order to obtain verification identifier information of the model to be verified; A model verification result is generated based on the verification identification information and the target model identification.

9. The method according to claim 8, characterized in that Acquiring the target model identifier of the target model includes: Input each model verification sample and each original training sample in the verification sample set into the target model respectively, to obtain a first target output of each model verification sample and a second target output of each original training sample; Based on the preset order, a sequence identifier is constructed for the first target output and the second target output to obtain a target model identifier of the target model.

10. The method according to claim 8, characterized in that Before obtaining the verification sample set for the target model, the model to be verified, and the target model identifier of the target model, the method further includes: Performing hash processing on the model file of the target model to obtain the target model identifier; The number of verification samples and the number of original samples in the verification sample set, as well as a preset ordering of the model verification samples and the original training samples, are determined based on the binary encoding information of the target model identifier.

11. A model verification sample generation device, characterized in that: The device comprises: The first acquisition module is used to acquire a target model, a sample training set corresponding to the target model, and a proxy model, wherein the sample training set includes a plurality of original training samples and corresponding sample labels; the original training samples are pictures or texts; A non-directional attack module: used for performing a non-directional attack on the target model based on the original training sample and the corresponding initial adversarial perturbation to obtain a first adversarial sample, wherein the non-directional attack refers to using the original training sample and the initial adversarial sample as inputs of the target model respectively, the initial adversarial sample is obtained by adding noise to the original training sample based on the initial adversarial perturbation, and the expected output of the target model for the original training sample is different from the expected output for the first adversarial sample; Targeted attack module: used to perform a targeted attack on the proxy model based on the first adversarial sample with the sample label of the original training sample as the expected output, to obtain a second adversarial sample; the targeted attack means taking the first adversarial sample as the input of the proxy model, and the expected result of the targeted attack processing is that the output result of the proxy model for the second adversarial sample is the same as the sample label; An adversarial verification test module: used to perform an adversarial verification test on the second adversarial sample to obtain an adversarial verification result; Target sample determination module: used to determine the second adversarial sample as a model verification sample when the adversarial verification result is verification passed.

12. The device according to claim 11, characterized in that The non-directional attack module includes: A noise processing unit: configured to perform noise processing on the original training sample based on the initial adversarial perturbation to obtain an initial adversarial sample; A target model unit is used to input the original training sample and the initial adversarial sample into the target model respectively, and obtain a first output result corresponding to the original training sample and a second output result corresponding to the initial adversarial perturbation; A first sample determination unit is configured to use the initial adversarial sample as a first adversarial sample when the first output result and the second output result satisfy a first adversarial condition.

13. The device according to claim 12, characterized in that The non-directional attack module also includes: A first gradient determining unit: configured to determine a current gradient of the target model according to the first output result and the second output result when the first output result and the second output result do not satisfy a first confrontation condition; A first disturbance updating unit: configured to iteratively update the initial adversarial perturbation according to the current gradient of the target model and the initial adversarial sample to obtain a first adversarial perturbation; The first sample determination unit is further configured to use an adversarial sample corresponding to the first adversarial perturbation as the first adversarial sample.

14. The device according to claim 11, characterized in that The directed attack module includes: Proxy model unit: used for inputting the first adversarial sample into the proxy model to obtain a reference output result corresponding to the first adversarial sample; A second sample determination unit is configured to use the first adversarial sample as the second adversarial sample when the reference output result and the sample label satisfy a second adversarial condition.

15. The device according to claim 14, characterized in that The directed attack module also includes: A second gradient determining unit: configured to determine the current gradient of the proxy model according to the reference output result and the sample label when the reference output result and the sample label do not satisfy the second confrontation condition; A second disturbance updating unit: configured to iteratively update the first adversarial perturbation according to the current gradient of the proxy model and the initial adversarial sample to obtain a second adversarial perturbation; The second sample determination unit is further configured to use an adversarial sample corresponding to the second adversarial perturbation as the second adversarial sample.

16. The device according to claim 11, characterized in that The adversarial verification result includes a white box test result and a black box test result, and the target sample determination module further includes: A white box testing unit: configured to perform a white box test on the second adversarial sample using the target model and the proxy model to obtain the white box testing result; A black box testing unit: configured to perform a black box test on the second adversarial sample using a verification model to obtain the black box testing result, wherein the verification model is a proxy model that does not perform a targeted attack based on the first adversarial sample; Adversarial verification unit: used to determine that the adversarial verification result is verified to be passed when both the white box test result and the black box test result meet corresponding test conditions.

17. The device according to claim 11, characterized in that The device also includes an iterative update module: For looping the steps of performing a non-directional attack on the target model, performing a directional attack on the proxy model, and performing the adversarial verification test, when the adversarial verification result is a failure, until an updated second adversarial sample with an adversarial verification result of passing the verification is obtained; The target sample determination module is also used to use the updated second adversarial sample as the model verification sample.

18. The device according to any one of claims 11 to 17, characterized in that The device also includes: A second acquisition module is used for obtaining a verification sample set for the target model, a model to be verified, and a target model identifier of the target model after determining the second adversarial sample as a model verification sample when the adversarial verification result is verification passed, wherein the verification sample set includes a plurality of the model verification samples and the original training samples; The model module to be verified is used to input each model verification sample and each original training sample in the verification sample set into the model to be verified, and obtain the first verification output of each model verification sample and the second verification output of each original training sample; The model to be verified module is used to construct a sequence identification for the first verification output and the second verification output based on a preset order, so as to obtain verification identification information of the model to be verified; The model module to be verified is used to generate a model verification result based on the verification identification information and the target model identification.

19. The device according to claim 18, characterized in that The second acquisition module includes: A target output unit is used to input each model verification sample and each original training sample in the verification sample set into the target model to obtain a first target output of each model verification sample and a second target output of each original training sample; An identification construction unit is used to construct a sequence identification for the first target output and the second target output based on the preset order to obtain a target model identification of the target model.

20. The device according to claim 18, characterized in that The device also includes: A hash processing module: used for performing hash processing on the model file of the target model to obtain the target model identifier before obtaining the verification sample set, the model to be verified and the target model identifier of the target model for the target model; Sorting determination module: used to determine the number of verification samples and the number of original samples in the verification sample set based on the binary encoding information of the target model identifier, and the preset sorting for the model verification samples and the original training samples.

21. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the model verification sample generation method according to any one of claims 1 to 10.

22. A computer device, comprising a processor and a memory, characterized in that: The memory stores at least one instruction or at least one program, and the at least one instruction or the at least one program is loaded and executed by the processor to implement the model verification sample generation method according to any one of claims 1 to 10.

23. A computer program product, characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the model verification sample generation method according to any one of claims 1 to 10 is implemented.

Images