Adversarial sample generation method, device and equipment

Abstract

The invention provides an adversarial sample generation method, device and equipment, relates to the technical field of artificial intelligence security, and aims to solve the technical problem of how to improve the migration ability of adversarial samples. The adversarial sample generation method comprises the following steps: in a forward propagation process of a target network model, extracting an intermediate layer feature map corresponding to an original image, and obtaining a correlation weight of each channel in the intermediate layer feature map; based on a target loss function of the target network model, determining a gradient of the intermediate layer feature map; according to the correlation weight of each channel and the gradient of the intermediate layer feature map, determining the correlation weighted gradient of the intermediate layer feature map; determining an adversarial loss function according to the correlation weighted gradient, and iteratively generating a target adversarial sample corresponding to the original image based on the adversarial loss function; the target adversarial sample is used for testing the model with the same task type as the target network model. The migration ability of the adversarial sample can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence security, and particularly relates to an adversarial sample generation method, device and equipment. BACKGROUND

[0002] In the field of artificial intelligence, a convolutional neural network (CNN) model exhibits excellent performance in image classification, target detection and other tasks. Therefore, evaluating and testing the model can comprehensively understand the model, find out the shortcomings and deficiencies of the model, and provide a clear direction for performance improvement. When evaluating the security and performance of the model, adversarial samples can be used as an effective tool to detect potential vulnerabilities of the model, thereby assisting in analyzing the performance status of the model. The model is optimized according to the analysis result, which can improve the defense effect of the model.

[0003] An adversarial sample is a sample generated by adding a human imperceptible small perturbation to the original input of a convolutional neural network model, which can cause the model to make an incorrect prediction, thereby testing the robustness and anti-interference ability of the model. The transferability of adversarial samples (i.e. the ability of adversarial samples generated from a source model to attack an unknown target model, realizing cross-model use of adversarial samples) is the core requirement of the actual adversarial attack method scenario. For example, when attacking and testing a target model through a black-box attack method, since the structure and parameters of the target model cannot be obtained, a transferable adversarial sample is needed to complete the attack and test of the target model. In general techniques, adversarial samples are usually generated by indiscriminately distorting the intermediate layer features of the source model, which easily falls into a local optimum specific to the source model, thereby reducing the transferability of the adversarial samples.

[0004] Therefore, how to improve the transferability of adversarial samples is a problem to be solved. SUMMARY

[0005] Embodiments of the present disclosure provide an adversarial sample generation method, device and equipment, aiming to solve the technical problem of how to improve the transferability of adversarial samples.

[0006] To achieve the above object, the technical scheme adopted by the present application is as follows:

[0007] In a first aspect, a method for generating an adversarial sample is provided, comprising: extracting an intermediate layer feature map corresponding to an original image and obtaining a correlation weight of each channel in the intermediate layer feature map during a forward propagation process of a target network model; the target network model is configured to predict the original image input into the target network model; determining a gradient of the intermediate layer feature map based on a target loss function of the target network model; the target loss function is configured to determine a difference between an output result of the target network model and a real label; determining a correlation weighted gradient of the intermediate layer feature map according to the correlation weight of each channel and the gradient of the intermediate layer feature map; determining an adversarial loss function according to the correlation weighted gradient, and iteratively generating a target adversarial sample corresponding to the original image based on the adversarial loss function; the target adversarial sample is configured to test a model of the same type as the target network model.

[0008] Optionally, the correlation weighted gradient of the intermediate layer feature map is determined according to the correlation weight of each channel and the gradient of the intermediate layer feature map, comprising:

[0009] The correlation weighted gradient satisfies the following formula:

[0010]

[0011] wherein, is the correlation weighted gradient; is the correlation weight of the i th channel of the intermediate layer feature map; c is a classification of the original image by the target network model; k is a k th intermediate layer of the target network model; i is the i th channel of the intermediate layer feature map; is the gradient of the intermediate layer feature map.

[0012] Optionally, the correlation weight of each channel in the intermediate layer feature map is obtained, comprising: extracting an activation map of each channel in the intermediate layer feature map; determining a correlation score of the channel corresponding to the activation map to obtain the correlation score of each channel; determining the correlation weight of each channel according to the correlation score of each channel; the correlation weight of each channel satisfies the following formula:

[0013]

[0014] wherein, is the correlation weight of the i th channel; c is a classification of the original image by the target network model; k is a k th intermediate layer of the target network model; i is the i th channel of the intermediate layer feature map; is the correlation score of the i th channel; (x, y) is a pixel position in the i th channel.

[0015] Optionally, the gradient of the intermediate layer feature map is determined based on the target loss function of the target network model, comprising:

[0016] The gradient of the intermediate layer feature map satisfies the following formula:

[0017]

[0018] wherein, is the gradient of the intermediate layer feature map; k is the kth intermediate layer of the target network model; x is the original image; l(x, t) is the target loss function, representing the loss between the prediction result of the target network model and the real label t; f k (x) is the intermediate layer feature map.

[0019] Optionally, based on the adversarial loss function, the target adversarial sample corresponding to the original image is iteratively generated, including: in the current iteration, the current adversarial sample is obtained; when the current iteration is the first iteration, the current adversarial sample is the original image; the current adversarial sample is the adversarial sample generated in the last iteration; the gradient of the intermediate layer feature map is updated according to the current adversarial sample, to obtain the updated gradient of the intermediate layer feature map; the correlation weighted gradient of the intermediate layer feature map is updated according to the correlation weight of each channel and the updated gradient of the intermediate layer feature map, to obtain the updated correlation weighted gradient; the adversarial loss function is updated according to the updated correlation weighted gradient, to obtain the updated adversarial loss function; based on the updated adversarial loss function, the gradient of the current adversarial sample is determined; the current momentum gradient is determined according to the gradient of the current adversarial sample; based on the current momentum gradient, the current adversarial sample is updated to obtain the adversarial sample generated in the current iteration; the above iteration process is repeated until the current iteration number reaches the preset total iteration number, and the adversarial sample generated in the current iteration is determined as the target adversarial sample.

[0020] Optionally, the current momentum gradient satisfies the following formula:

[0021]

[0022] wherein, g t+1 is the current momentum gradient; t is the current iteration number; μ is a momentum factor; g t is the momentum gradient of the tth iteration, representing the momentum gradient cumulative result of the first t iterations; is the updated adversarial loss function; is the gradient of the current adversarial sample, is the current adversarial sample; is the L1 norm of the gradient of the current adversarial sample; is the vector operator, representing the gradient of the updated adversarial loss function with respect to the pixel value of the current adversarial sample.

[0023] Optionally, the current adversarial sample satisfies the following formula:

[0024]

[0025] wherein, is the updated current adversarial sample; Clip is a clipping symbol; ε is a maximum perturbation range; α is a step size, α = ε / T; t is a current iteration number; T is a total number of iterations; and Sign is a sign function.

[0026] Optionally, the updated adversarial loss function satisfies the following formula:

[0027]

[0028] wherein, is the updated adversarial loss function; x adv is the current adversarial sample; is the updated correlation weighted gradient; k is the kth intermediate layer of the target network model; and represents element multiplication; f k (x adv ) is the feature map of the adversarial sample at the kth layer of the target network model.

[0029] In a second aspect, an adversarial sample generation apparatus is provided, which comprises a communication unit and a processing unit. The communication unit is configured to extract an intermediate layer feature map corresponding to an original image and obtain a correlation weight of each channel in the intermediate layer feature map during a forward propagation process of a target network model. The target network model is configured to predict the original image input into the target network model. The processing unit is configured to determine a gradient of the intermediate layer feature map based on a target loss function of the target network model. The target loss function is configured to determine a difference between an output result of the target network model and a true label. The processing unit is further configured to determine a correlation weighted gradient of the intermediate layer feature map according to the correlation weight of each channel and the gradient of the intermediate layer feature map. The processing unit is further configured to determine an adversarial loss function according to the correlation weighted gradient, and iteratively generate a target adversarial sample corresponding to the original image based on the adversarial loss function. The target adversarial sample is configured to test a model of the same type as the target network model.

[0030] In a third aspect, an adversarial sample generation device is provided, which comprises a memory and a processor. The memory is configured to store computer execution instructions, and the processor is connected to the memory through a bus. When the adversarial sample generation device is running, the processor executes the computer execution instructions stored in the memory, so that the adversarial sample generation device executes the adversarial sample generation method of the first aspect.

[0031] The adversarial sample generation device can be an electronic device, or a part of a device in an electronic device, such as a chip system in an electronic device. The chip system is configured to support the electronic device to implement the functions involved in the first aspect and any possible implementation manner thereof, for example, to obtain and determine the data and / or information involved in the adversarial sample generation method described above. The chip system includes a chip, and can also include other discrete devices or circuit structures.

[0032] In a fourth aspect, a computer-readable storage medium is provided, and the computer-readable storage medium includes computer-executable instructions. When the computer-executable instructions are run on a computer, the computer is caused to perform the adversarial sample generation method in the first aspect.

[0033] In a fifth aspect, a computer program product is also provided, and the computer program product includes computer programs or instructions. When the computer instructions are run on the adversarial sample generation device, the adversarial sample generation device is caused to perform the adversarial sample generation method in the first aspect.

[0034] It should be noted that the computer instructions described above can be stored on the computer-readable storage medium in whole or in part. The computer-readable storage medium can be packaged together with the processor of the adversarial sample generation device, or can be packaged separately from the processor of the adversarial sample generation device, and the embodiments of the present application do not limit this.

[0035] The second aspect, the third aspect, the fourth aspect, and the fifth aspect of the present application can refer to the detailed description of the first aspect.

[0036] In the embodiments of the present application, the names of the adversarial sample generation device described above do not constitute a limitation on the devices or functional modules themselves, and in actual implementation, these devices or functional modules can appear with other names. For example, the receiving unit can also be referred to as a receiving module, a receiver, etc. As long as the functions of each device or functional module are similar to those of the present application, they belong to the scope of the claims of the present application and equivalent technologies.

[0037] The technical solutions provided by the present application at least bring the following beneficial effects:

[0038] Based on any of the above aspects, the embodiment of the present application provides an adversarial sample generation method. First, in the forward propagation process of a target network model, an intermediate layer feature map corresponding to an original image is extracted, and the correlation weights of each channel in the intermediate layer feature map are obtained. The target network model is used to predict the original image input into the target network model. Then, based on a target loss function of the target network model, the gradient of the intermediate layer feature map is determined. The target loss function is used to determine the difference between the output result of the target network model and the real label. Secondly, according to the correlation weights of each channel and the gradient of the intermediate layer feature map, the correlation weighted gradient of the intermediate layer feature map is determined. Subsequently, according to the correlation weighted gradient, the adversarial loss function is determined, and based on the adversarial loss function, the target adversarial sample corresponding to the original image is iteratively generated. The target adversarial sample is used to test the model of the same type as the target network model.

[0039] As can be seen from the above, first, the present application extracts the intermediate layer feature map in the target network model to obtain the correlation weights of each channel in the intermediate layer feature map and the gradient of the intermediate layer feature map.

[0040] Then, according to the correlation weights of each channel and the gradient of the intermediate layer feature map, the gradient of the intermediate layer feature map is weighted in the channel dimension to determine the correlation weighted gradient of the intermediate layer feature map. Through the correlation weighted gradient of the intermediate layer feature map, the general key features of the model of the same category as the target network model can be captured, and the key features are assigned with higher weight values.

[0041] Subsequently, in the process of iteratively generating the target adversarial sample through the adversarial loss function determined by the correlation weighted gradient, the most core features in the target network model can be focused on, and unnecessary disturbance to irrelevant feature regions can be avoided, thereby improving the migration ability of the target adversarial sample and realizing the generation of the target adversarial sample with higher migration ability. Moreover, the adversarial sample generated by minimizing the adversarial loss function in an iterative manner can attack and test the model through the tiny disturbance that is difficult to detect by the human eye without changing the semantic information of the original image input. Thus, the model is tested more close to the application of the model in the real scene, and the optimization of the model can directly improve the anti-interference ability of the model in the actual application.

[0042] The beneficial effects of the first aspect, the second aspect, the third aspect, the fourth aspect and the fifth aspect of the present application can be referred to the analysis of the beneficial effects above, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 A structural schematic diagram of an adversarial sample generation system provided by the embodiment of the present application is shown in the figure;

[0044] Figure 2 A hardware structure schematic diagram of an adversarial sample generation device provided for an embodiment of the present application;

[0045] Figure 3 A flowchart of an adversarial sample generation method provided for an embodiment of the present application;

[0046] Figure 4 A flowchart of another adversarial sample generation method provided for an embodiment of the present application;

[0047] Figure 5 A schematic diagram of gradient visualization of an original image on an intermediate layer feature map of a Resnet-50 and Vgg-16 network provided for an embodiment of the present application;

[0048] Figure 6 A Relevance-CAM heat map generated in a Resnet-50 and Vgg-16 model based on a Relevance-CAM method provided for an embodiment of the present application;

[0049] Figure 7 A schematic diagram of network intermediate layer relevance weighted gradient visualization of an original image on a Resnet-50 and Vgg-16 model provided for an embodiment of the present application;

[0050] Figure 8 A flowchart of another adversarial sample generation method provided for an embodiment of the present application;

[0051] Figure 9 A flowchart of another adversarial sample generation method provided for an embodiment of the present application;

[0052] Figure 10 A flowchart of a relevance gradient attack method provided for an embodiment of the present application;

[0053] Figure 11 A comparison diagram of attack success rates of adversarial samples generated at different layers of a Vgg-16 model and a Resnet-50 model on a target model provided for an embodiment of the present application;

[0054] Figure 12 A comparison diagram of attack success rates of adversarial samples constructed based on different loss functions under four source models on a normally trained model provided for an embodiment of the present application;

[0055] Figure 13 A heat map of adversarial samples generated based on four loss functions provided for an embodiment of the present application;

[0056] Figure 14 A structure schematic diagram of an adversarial sample generation device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work meet the protection scope of the present application.

[0058] It should be noted that in the embodiments of the present application, the words such as “exemplary” or “for example” are used to mean serving as an example, an instance, or an illustration. Any embodiment or design scheme described as “exemplary” or “for example” in the embodiments of the present application should not be interpreted as being more preferred or having more advantages than other embodiments or design schemes. Rather, the words “exemplary” or “for example” are used to represent the relevant concept in a specific manner.

[0059] In order to clearly describe the technical solutions in the embodiments of the present application, in the embodiments of the present application, the words “first”, “second”, and the like are used to distinguish the same or similar items or items with basically the same function and role. Those skilled in the art can understand that the words “first”, “second”, and the like are not used to limit the quantity and execution order.

[0060] Before the method for generating an adversarial sample provided by the present application is described in detail, the application scenarios and implementation environments involved in the present application are briefly introduced.

[0061] First, the application scenarios involved in the present application are briefly introduced.

[0062] As described in the background, by evaluating and testing the network model, the model can be comprehensively understood, and the deficiencies of the model can be determined, and then the model can be optimized, and the performance of the model can be improved. In the process of testing the model, the model is often attacked and tested by an adversarial sample. In the general technology, the method for generating an adversarial sample is usually to generate an adversarial sample by twisting the intermediate layer features of a source model without distinction, which is easy to fall into a local optimum specific to the source model, and thus reduces the transferability of the adversarial sample.

[0063] To solve the above problems, an adversarial sample generation method is provided in the embodiments of the present application. First, during the forward propagation of a target network model, the intermediate layer feature map corresponding to the original image is extracted, and the correlation weight of each channel in the intermediate layer feature map is obtained. The target network model is used to predict the original image input into the target network model. Then, based on the target loss function of the target network model, the gradient of the intermediate layer feature map is determined. The target loss function is used to determine the difference between the output result of the target network model and the real label. Secondly, according to the correlation weight of each channel and the gradient of the intermediate layer feature map, the correlation weighted gradient of the intermediate layer feature map is determined. Subsequently, according to the correlation weighted gradient, the adversarial loss function is determined, and based on the adversarial loss function, the target adversarial sample corresponding to the original image is iteratively generated. The target adversarial sample is used to test the model of the same type as the target network model.

[0064] As can be seen from the above, first, the present application extracts the intermediate layer feature map in the target network model to obtain the correlation weight of each channel in the intermediate layer feature map and the gradient of the intermediate layer feature map.

[0065] Then, according to the correlation weight of each channel and the gradient of the intermediate layer feature map, the gradient of the intermediate layer feature map is weighted in the channel dimension to determine the correlation weighted gradient of the intermediate layer feature map. Through the correlation weighted gradient of the intermediate layer feature map, the general key features of the model of the same category as the target network model can be captured, and the key features are assigned with a higher weight value.

[0066] Subsequently, in the process of iteratively generating the target adversarial sample through the adversarial loss function determined by the correlation weighted gradient, the most core features in the target network model can be focused on, and unnecessary disturbance to irrelevant feature regions can be avoided, thereby improving the migration ability of the target adversarial sample and realizing the generation of the target adversarial sample with higher migration ability. Moreover, the adversarial sample generated by minimizing the adversarial loss function in an iterative manner can attack and test the model through a slight disturbance that is difficult to detect by the human eye without changing the semantic information of the original image input. Thus, the model is tested, which is closer to the application of the model in a real scene, and the optimization of the model can directly improve the anti-interference ability of the model in actual application.

[0067] The implementation environment of the above adversarial sample generation method can be the adversarial sample generation system provided by the embodiments of the present application.

[0068] Figure 1 A structure schematic diagram of an adversarial sample generation system provided by the embodiments of the present application is shown. As shown in Figure 1 The adversarial sample generation system includes an adversarial sample generation device 101 and a data acquisition device 102.

[0069] The adversarial sample generation device 101 is in communication connection with the data acquisition device 102.

[0070] In actual application, the adversarial sample generation device 101 can be connected with any number of data acquisition devices 102. For the convenience of understanding, Figure 1 Taking one adversarial sample generation device 101 connected with one data acquisition device 102 as an example for description.

[0071] In the embodiment of the application, the data acquisition device 102 is used to acquire the original image as the basic input of the adversarial sample generation device 101.

[0072] The adversarial sample generation device 101 is used to acquire the original image acquired by the data acquisition device 102, and generate the adversarial sample based on the original image.

[0073] Optionally, the entity device of the data acquisition device 102 can be a camera (such as an industrial camera) or other types of electronic devices for acquiring images, and the embodiment of the application does not limit this.

[0074] Optionally, the entity device of the adversarial sample generation device 101 can be a server, a terminal, or other types of electronic devices, and the embodiment of the application does not limit this.

[0075] Optionally, the terminal can be a device that provides voice and / or data connectivity to a user, a handheld device with a wireless connection function, or other processing devices connected to a wireless modem. The wireless terminal can communicate with one or more core networks through a radio access network (RAN). The wireless terminal can be a mobile terminal, such as a mobile phone (or called "cellular" phone) and a computer with a mobile terminal, which can be a portable, pocket, handheld, built-in computer or vehicle-mounted mobile device that exchanges voice and / or data with a radio access network. For example, a mobile phone, a tablet computer, a notebook computer, a netbook, a personal digital assistant (PDA).

[0076] Optionally, the server can be one server in a server cluster (composed of multiple servers), a chip in the server, or a system on a chip in the server, or can be implemented through a virtual machine (VM) deployed on a physical machine, and the embodiment of the application does not limit this.

[0077] Optionally, the adversarial sample generation device 101 and the data acquisition device 102 can be two independent devices or integrated into the same device. When the adversarial sample generation device 101 and the data acquisition device 102 are integrated into the same device, the data acquisition device 102 can be an acquisition module (e.g., an image collector) of the adversarial sample generation device 101.

[0078] It is easy to understand that when adversarial example generation device 101 and data acquisition device 102 are integrated into the same device, the communication between them is carried out between modules within the device. In this case, the communication process between them is the same as when adversarial example generation device 101 and data acquisition device 102 are independent.

[0079] For ease of understanding, this application is described by taking the adversarial sample generation device 101 and the data acquisition device 102 as an example in which they are independent of each other.

[0080] The adversarial sample generation device in the adversarial sample generation system includes: Figure 2 The following are the components included. Figure 2 Taking the adversarial sample generation device shown in the figure as an example, the hardware structure of the adversarial sample generation device is introduced.

[0081] Figure 2 FIG2 is a schematic diagram of the hardware structure of an adversarial sample generation device provided in an embodiment of the present application. The adversarial sample generation device includes a processor 21, a memory 22, a communication interface 23, and a bus 24. The processor 21, the memory 22, and the communication interface 23 can be connected via a bus 24.

[0082] Processor 21 is the control center of the adversarial example generation device and can be a single processor or a collection of multiple processing elements. For example, processor 21 can be a general-purpose central processing unit (CPU) or other general-purpose processor. A general-purpose processor can be a microprocessor or any conventional processor.

[0083] As an embodiment, the processor 21 may include one or more CPUs, such as Figure 2 CPU0 and CPU1 are shown in the figure.

[0084] The memory 22 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0085] In a possible implementation, the memory 22 can exist independently of the processor 21, and the memory 22 can be connected to the processor 21 through the bus 24, for storing instructions or program codes. When the processor 21 invokes and executes the instructions or program codes stored in the memory 22, the method for generating an adversarial sample provided in the embodiments of the present application can be implemented.

[0086] In the embodiments of the present application, the software programs stored in the memory 22 of the adversarial sample generation apparatus are different, and therefore the functions implemented by the adversarial sample generation apparatus are different. The functions performed by each device will be described in combination with the flowcharts below.

[0087] In another possible implementation, the memory 22 can also be integrated with the processor 21.

[0088] The communication interface 23 is configured to connect the adversarial sample generation apparatus to other devices through a communication network. The communication network can be an Ethernet, a wireless access network, a wireless local area network (WLAN), etc. The communication interface 23 can include a receiving unit configured to receive data, and a sending unit configured to send data.

[0089] The bus 24 can be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 2 In the figure, only one thick line is used to represent the bus, but it does not mean that there is only one bus or only one type of bus.

[0090] It should be noted that, Figure 2 The structure shown in the figure does not constitute a limitation on the adversarial sample generation device, except Figure 2 In addition to the components shown, the adversarial sample generation device can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0091] The adversarial sample generation method provided by the embodiments of the present application will be described in detail below in conjunction with the drawings.

[0092] The adversarial sample generation method provided by the embodiments of the present application is applied to Figure 1 The adversarial sample generation device 101 in the adversarial sample generation system shown in the figure, such as Figure 3 The adversarial sample generation method provided by the embodiments of the present application includes:

[0093] S301, in the forward propagation process of the target network model, extracting the intermediate layer feature map corresponding to the original image, and obtaining the correlation weight of each channel in the intermediate layer feature map.

[0094] The target network model is used to predict the original image input into the target network model.

[0095] Specifically, the correlation weight in the intermediate layer feature map represents the correlation degree of each channel feature map and the final prediction result of the model, therefore, the correlation weight of each channel in the intermediate layer feature map needs to be obtained, so as to determine the intermediate layer correlation weighted gradient according to the correlation weight of each channel subsequently.

[0096] Optionally, the target network model can be a model based on convolutional neural network (CNN) in the image field to extract spatial features for decision making, such as image classification model, target detection model, semantic segmentation model, automatic driving vision model, defect detection and classification model or face recognition and identity verification model, etc.

[0097] Optionally, the intermediate layer feature map represents the output of the original image input into the target network model after being processed by the intermediate layers such as convolution layer and activation layer in the forward propagation process of the CNN-based target network model. Among them, the shallow layer feature map usually captures basic features such as edge and texture, and the deep layer feature map captures more abstract semantic features (such as "eye", "wheel", etc.).

[0098] Optionally, each intermediate layer feature map has multiple channels, each channel corresponding to a specific feature pattern (such as "horizontal edge" or "circular area"). The correlation weight of the channel is used to measure the relevance of the features of each channel to the final decision (such as classification or detection) of the target network model. The higher the weight, the more important the features of the channel to the decision of the target network model.

[0099] As can be seen from the above, by obtaining the correlation weight of each channel of the intermediate layer feature map, the invalid analysis of irrelevant features (such as background noise) can be avoided, and the core features (such as "ear" and "eye" channels for identifying "cat") relied on by the target network model can be focused on, providing a clear target for subsequent perturbation.

[0100] S302, determining the gradient of the intermediate layer feature map based on the target loss function of the target network model.

[0101] The target loss function is used to determine the difference between the output result of the target network model and the real label.

[0102] Specifically, the gradient of the intermediate layer feature map needs to be determined to determine the correlation weighted gradient of the intermediate layer feature map, so the gradient of the intermediate layer feature map needs to be obtained.

[0103] Optionally, the difference between the output result of the target network model and the real label can also be referred to as the loss value.

[0104] Optionally, the gradient of the intermediate layer feature map represents the influence degree of the slight change of the intermediate layer feature map on the loss value of the target network model. That is, which position in the intermediate layer feature map can most effectively change the output of the target network model through slight perturbation.

[0105] As can be seen from the above, by obtaining the gradient of the intermediate layer feature map, a basis for the perturbation direction of the adversarial sample can be provided, that is, how to most effectively increase the loss value of the target network model by slightly changing the feature map, thereby avoiding blind addition of perturbation and only adding perturbation to the area that most significantly affects the loss value of the target network model, improving the efficiency of the perturbation.

[0106] S303, determining the correlation weighted gradient of the intermediate layer feature map according to the correlation weight of each channel and the gradient of the intermediate layer feature map.

[0107] Specifically, the correlation weighted gradient of the intermediate layer feature map determined according to the correlation weight of each channel and the gradient of the intermediate layer feature map can ensure that the perturbation focuses on the core features of the target network model, which can effectively mislead the model and control the amplitude of the perturbation.

[0108] As can be seen from the above, the correlation-weighted gradient of the intermediate layer feature map fuses the composite index of "feature channel importance" and "gradient sensitivity", quantifies the comprehensive influence of the feature at each position in the intermediate layer feature map f on the model decision, and reflects whether the feature is the key of the target network model (determined by the correlation weight) and whether the slight change of the feature will significantly affect the prediction result of the target network model (determined by the gradient of the intermediate layer feature map).

[0109] S304, determining an adversarial loss function according to the correlation-weighted gradient, and iteratively generating a target adversarial sample corresponding to the original image based on the adversarial loss function.

[0110] The target adversarial sample is used to test a model of the same type as the target network model.

[0111] Specifically, determining the adversarial loss function according to the correlation-weighted gradient can ensure that the generated adversarial sample maximizes the loss of the target network model under a slight perturbation, and ensure that the adversarial sample generated based on the adversarial loss function is used to test a model of the same type as the target network model.

[0112] Optionally, the adversarial loss function represents the difference between the adversarial sample and the original image, and is used to guide the generation of the adversarial sample. Based on the adversarial loss function determined by the correlation-weighted gradient, an adversarial sample that can stably mislead the model and is difficult for the human eye to detect is generated, and it is ensured that the sample can be migrated to a model of the same type as the target network model.

[0113] In some embodiments, the correlation-weighted gradient of the intermediate layer feature map is determined according to the correlation weight of each channel and the gradient of the intermediate layer feature map, and includes:

[0114] The correlation-weighted gradient satisfies the following formula:

[0115]

[0116] wherein, is the correlation-weighted gradient; is the correlation weight of the i th channel of the intermediate layer feature map; c is the classification of the target network model on the original image; k is the k th intermediate layer of the target network model; i is the i th channel of the intermediate layer feature map; is the gradient of the intermediate layer feature map.

[0117] As can be seen from the above formula, by multiplying the correlation weights of each channel and the gradient of the intermediate layer feature map, the correlation weighted gradient of the intermediate layer feature map is determined, and the gradient of the intermediate layer feature map is weighted in the channel dimension, a more robust correlation weighted gradient of the intermediate layer feature map can be obtained. The correlation weighted gradient determined in this way binds the importance of the channel to the sensitivity of the position, so that the subsequently generated perturbations focus on the core features of the model. For example, if a certain channel is the core of the model decision, and the target edge position in the channel is extremely sensitive to the change in loss, the weighted gradient of this position will be significantly amplified and become the core target of the subsequent perturbations. This reduces invalid perturbations (such as perturbations to the background area), allowing the subsequently generated adversarial samples to effectively mislead the model with small perturbations.

[0118] In some embodiments, combined Figure 3 ,like Figure 4 As shown, in the above S301, obtaining the correlation weight of each channel in the intermediate layer feature map includes:

[0119] S401. Extract the activation map of each channel in the intermediate layer feature map.

[0120] Specifically, the response distribution of each channel to the original image in the intermediate layer feature map is obtained, that is, the image area that each channel focuses on (that is, the activation map of each channel), so as to subsequently obtain the correlation score of each channel.

[0121] Optionally, the intermediate layer feature map consists of multiple channels, each channel corresponding to a specific feature pattern (such as "edge", "texture", "semantic component").

[0122] Optionally, the activation map represents the feature map output of each channel, and its pixel value represents the response intensity of the position to the corresponding feature of the channel (the higher the value, the more consistent the position is with the characteristic pattern of the channel).

[0123] S402: Determine the correlation scores of the channels corresponding to the activation map to obtain the correlation scores of the channels.

[0124] Specifically, the correlation scores of the channels corresponding to the activation maps are calculated through inter-layer correlation propagation to obtain the correlation scores of each channel.

[0125] Optionally, the correlation score of each channel represents the degree of correlation between the activation map of each channel and the final decision of the target network model (such as classification result, detection confidence).

[0126] S403: Determine the correlation weight of each channel according to the correlation score of each channel.

[0127] The correlation weight of each channel satisfies the following formula:

[0128]

[0129] wherein, is the relevance weight of the ith channel; c is the classification of the original image by the target network model; k is the kth intermediate layer of the target network model; i is the ith channel of the intermediate layer feature map; is the relevance score of the ith channel; (x, y) is the pixel position in the ith channel.

[0130] For example, according to the Relevance-Class Activation Mapping (Relevance-CAM) method, the relevance weight of each channel is determined.

[0131] First, during the forward propagation of the target network model, the activation map (also referred to as the feature map) is extracted, and the relevance score is calculated by the relevance propagation rule. The relevance propagation rule is the z rule, which is based on the theoretical basis of deep Taylor decomposition, and the relevance score (the contribution degree to the decision of the target network model) of the high-level neuron is back-propagated to the low-level neuron, and then the key area of the input data (such as the original image) to the decision of the target network model is located. The relevant formula is as follows:

[0132]

[0133] wherein, R i and R j respectively represent the relevance score of the ith layer of the target network model and the relevance score of the jth layer; x i represents the activation output of the ith layer; represents the forward contribution component of the ith layer neuron to the jth layer neuron; w + ij represents the positive part of the weight between the ith layer and the jth layer of the target network model.

[0134] Then, based on the relevance score of each channel, the relevance weight of each channel is determined according to the above-mentioned determination method of the relevance score of each channel. As can be seen from the above relevance weight formula of each channel, the relevance score of each channel is multiplied by the relevance weight of each channel to obtain the relevance weight of each channel. The global average pooling calculates the relevance weight of all pixel points (i.e. pixel point (x, y)) in the channel dimension

[0135] As can be seen from the above, the Relevance-CAM method obtains the relevance score through layer-wise relevance propagation (LRP), and provides basis for the channel weight corresponding to the class activation map, and then generates a visual heat map reflecting the attention area of the model to a specific class. The formula of the Relevance-CAM class activation map (also referred to as a heat map) is shown in the following formula (1) :

[0136]

[0137] wherein, is the Relevance-CAM class activation map generated for class c at the k-th intermediate layer, used for visualizing the spatial region to which the model pays attention when judging that the input belongs to class c. is the relevance weight of the i-th channel in the k-th layer for class c. is the activation map of the i-th channel in the k-th layer for class c.

[0138] As can be seen from the above, the model can determine the Relevance-CAM class activation map and the relevance weight of each channel through the activation map and the relevance score through one forward propagation and one backward propagation.

[0139] In some embodiments, based on a target loss function of a target network model, a gradient of an intermediate layer feature map is determined, including:

[0140] The gradient of the intermediate layer feature map satisfies the following formula:

[0141]

[0142] wherein, is the gradient of the intermediate layer feature map; k is the k-th intermediate layer of the target network model; x is the original image; l(x, t) is the target loss function, representing the loss between the prediction result of the target network model and the true label t; f k (x) is the intermediate layer feature map.

[0143] As can be seen from the above formula, the partial derivative of the target loss function l(x, t) of the target network model with respect to the intermediate layer feature map f k (x) is taken to determine which is used to represent the influence degree of the slight change of the intermediate layer feature map on the loss of the model, that is, the greater , the more obvious the change of the loss (prediction error) of the target network model when the pixels of the feature image at this position are slightly changed.

[0144] As shown in the following formula (2), for example, Figure 5 Figure 5 ​Gradient visualizations of the intermediate layer feature maps of the original image on the Resnet-50 and Vgg-16 models are respectively shown. Among them, the last layer of block1 is selected as the intermediate layer feature map in the Resnet-50 model, and the Conv3_3 layer is selected as the intermediate layer feature map on the Vgg-16. It can be found that although part of the gradient peak focuses on the target object body, there are still many visual noises in the gradient map of the original image, that is, the peak value scattered on the background area of the non-object body, which may be caused by the specific solution space of the model (that is, there are inherent differences in feature extraction and weight allocation of different models). In order to avoid the interference of model-specific information, the application proposes a relevance weighted gradient, and the relevance weight and Relevance-CAM heat map of each channel in the intermediate layer feature map of any layer in the network model can be obtained by the Relevance-CAM method.

[0145] Figure 6 A Relevance-CAM heat map generated by the Relevance-CAM method provided by the embodiment of the application in the Resnet-50 and Vgg-16 models is provided. As shown in Figure 6 , by observing the Relevance-CAM heat map generated by the original image in the Resnet-50 and Vgg-16 models, it can be found that the attention distribution presented by the models with different architectures on the Relevance-CAM heat map has high similarity (that is, the core regions focused by the models are highly overlapped), so that the similarity between different network models can be extracted, thereby obtaining more robust feature importance (that is., obtain the core features that are not disturbed by model-specific information and can stably reflect the essence of the model task).

[0146] Figure 7 A schematic diagram of the network intermediate layer relevance weighted gradient visualization of the original image on the Resnet-50 and Vgg-16 models provided by the embodiment of the application is provided. In combination with Figure 5 , as shown in Figure 7 , Figure 7 , compared with the original intermediate layer gradient of Figure 5 , Figure 7 , the relevance weighted gradient in becomes clearer, and the interference of the non-object area that may cause local optimum is weakened, while the object-related feature area is highlighted. These object-related feature areas can provide better feature importance reference and guide the generation of more transferable adversarial samples (that is., the generated adversarial samples can be used for models with the same task type).

[0147] In some embodiments, in combination with Figure 3 , as shown in Figure 8As shown in the above S304, the target adversarial sample corresponding to the original image is iteratively generated based on the adversarial loss function, including:

[0148] S801, in the current iteration, the current adversarial sample is obtained.

[0149] Wherein, when the current iteration is the first iteration, the current adversarial sample is the original image. The current adversarial sample is the adversarial sample generated by the last iteration.

[0150] Specifically, when the first iteration, is through the original image to increase the disturbance to achieve the generation of adversarial samples, and in each iteration process, the last generated adversarial sample is iteratively updated, therefore, before each iteration, the current adversarial sample needs to be obtained.

[0151] S802, according to the current adversarial sample, the gradient of the intermediate layer feature map is updated to obtain the updated gradient of the intermediate layer feature map.

[0152] Specifically, the gradient of the intermediate layer feature map depends on the input sample, and in the iteration process, the input sample of each iteration is different (i.e. the generated adversarial sample is different), therefore, the gradient of the intermediate layer feature map needs to be updated according to the current adversarial sample, to obtain the updated gradient of the intermediate layer feature map.

[0153] Optionally, according to the current adversarial sample, the gradient of the intermediate layer feature map is updated, and the updated gradient of the intermediate layer feature map satisfies the following formula:

[0154]

[0155] Wherein, is the updated gradient of the intermediate layer feature map; is the current adversarial sample; is the target loss function; is the intermediate layer feature map.

[0156] S803, according to the correlation weight of each channel and the updated gradient of the intermediate layer feature map, the correlation weighted gradient of the intermediate layer feature map is updated to obtain the updated correlation weighted gradient.

[0157] Specifically, when the gradient of the intermediate layer feature map changes, the corresponding correlation weighted gradient of the intermediate layer feature map will also change, therefore, according to the correlation weight of each channel and the updated gradient of the intermediate layer feature map, the correlation weighted gradient of the intermediate layer feature map is updated to obtain the updated correlation weighted gradient.

[0158] S804, according to the updated correlation weighted gradient, the adversarial loss function is updated to obtain the updated adversarial loss function.

[0159] Specifically, when the correlation weighted gradient changes, because the adversarial loss function is determined based on the correlation weighted gradient, therefore, it is necessary to update the adversarial loss function according to the updated correlation weighted gradient to obtain an updated adversarial loss function.

[0160] In some embodiments, the updated adversarial loss function satisfies the following formula:

[0161]

[0162] wherein, is the updated adversarial loss function; x adv is the current adversarial sample; is the updated correlation weighted gradient; k is the kth intermediate layer of the target network model; ⊙ represents element multiplication; f k (x adv ) is the feature map of the adversarial sample at the kth layer of the target network model.

[0163] As can be seen from the above formula, the adversarial loss function determined based on the correlation weighted gradient can guide the perturbation to focus on the key feature area of the model. The correlation weighted gradient has filtered the model-specific noise and only retains the sensitive area of the task core feature. After multiplying the intermediate layer feature map gradient, the loss value will be amplified in the area where the feature response is strong and the model is sensitive (such as the target subject), guiding the perturbation to preferentially adjust these areas. Compared with the traditional method of relying only on the output layer loss (such as cross-entropy), this formula directly acts on the intermediate layer features, which can more accurately attack the feature extraction logic of the test model, so that a small perturbation can significantly reduce the model performance (attack test efficiency is higher).

[0164] S805, determine the gradient of the current adversarial sample based on the updated adversarial loss function.

[0165] Specifically, the gradient of the current adversarial sample is determined by taking the partial derivative of the current adversarial sample with respect to the updated adversarial loss function.

[0166] Optionally, the gradient of the current adversarial sample is used to indicate the direction of image pixel perturbation.

[0167] S806, determine the current momentum gradient according to the gradient of the current adversarial sample.

[0168] Optionally, the momentum gradient generally represents the weighted sum of the gradient of the current adversarial sample and the historical momentum gradient, in order to synchronize with the previous optimization direction, avoid being disturbed by local noise, and make the sample adjust to the effective area faster.

[0169] In some embodiments, the current momentum gradient satisfies the following formula:

[0170]

[0171] wherein g t+1 current momentum gradient; t is the current iteration number; μ is a momentum factor; g t is the momentum gradient of the t-th iteration, and represents the momentum gradient accumulation result of the previous t iterations; is the updated adversarial loss function; is the gradient of the current adversarial sample, is the current adversarial sample; is the L1 norm of the gradient of the current adversarial sample; vector operator, representing the gradient of the updated adversarial loss function with respect to the pixel value of the current adversarial sample.

[0172] As can be seen from the above formula, the application solves the problems of large gradient fluctuation and unstable optimization direction in the generation of adversarial samples by weighted fusion of historical momentum and normalized current gradient. In the process of generating adversarial samples, the gradient of the current adversarial sample may suddenly change (for example, the gradient points to the target head in the last iteration, and suddenly points to the background due to noise in this iteration) due to noise in the intermediate layer feature map (such as high activation in the background area). The formula retains the direction of the historical momentum, which can weaken the interference of the current noise gradient, and ensures that the disturbance always focuses on the core area sensitive to the model, avoiding repeated adjustment of the sample update in irrelevant areas, improving the iteration stability and the transferability of the adversarial sample.

[0173] S807, based on the current momentum gradient, updating the current adversarial sample to obtain an adversarial sample generated by the current iteration.

[0174] Specifically, based on the current momentum gradient, updating the current adversarial sample can adjust the sample pixels in the direction of the momentum gradient to ensure the generation of a better adversarial sample, while maintaining a small disturbance and enhancing the attack test effect of the adversarial sample.

[0175] In some embodiments, the current adversarial sample satisfies the following formula:

[0176]

[0177] wherein, is the updated current adversarial sample; Clip is a clipping symbol; ε is the maximum disturbance range; α is the step size, α = ε / T; t is the current iteration number; T is the total number of iterations; and Sign is a sign function.

[0178] As can be seen from the above formula, Clip x,εThe restriction that the adversarial sample is in the ε neighborhood of the original sample x limits the maximum amplitude of the perturbation. It ensures that the adversarial sample is highly similar to the original sample in vision or semantics (for example, the pixel change in the image is slight, and the character replacement in the text does not affect the readability), so that when the model is attacked and tested based on the adversarial sample later, the real environment can be better simulated.

[0179] S808, repeat the above iteration process until the current iteration number reaches the preset total number of iterations, and determine the adversarial sample generated by the current iteration as the target adversarial sample.

[0180] Optionally, by multiple iterations, the adversarial loss function is minimized to ensure that the generated adversarial sample satisfies the ability to be disturbed based on the core area of the model, so that other models with the same task can also be attacked and tested, and the transferability of the adversarial sample is realized.

[0181] Exemplarily, Figure 9 Another flowchart of an adversarial sample generation method provided by an embodiment of the present application is shown in FIG. 8. As shown in the figure, the adversarial sample generation method includes the following steps. Figure 9

[0182] S901, input initialization.

[0183] Exemplarily, the input initialization includes input parameter initialization and initialization variable.

[0184] The input parameters include: an original clean image x, a true label c, a source model f, a specified intermediate layer k, a maximum perturbation range ε, and an iteration number T.

[0185] The initialization variables include: the perturbation is initialized as a zero vector to ensure that the initial adversarial sample is the original image x 0 = x. The momentum gradient g is initialized as zero to accumulate historical gradient information to stabilize the optimization direction. The step size α is set as ε / T.

[0186] S902, forward propagation and gradient calculation.

[0187] Exemplarily, in the t-th iteration (0 <= t < T), the current adversarial sample x t adv The feature map of the intermediate layer k is extracted as the intermediate layer feature map in the process of forward propagation in the source model f.

[0188] The target loss function l(x t adv , c) for calculating the loss value between the source model and the true label c is determined, and the gradient of the output on the intermediate layer feature map is calculated in the process of back propagation through the source model. The gradient dimension is the same as the feature Figure 1 ​The direction and strength of the influence of each feature unit on the classification result are reflected.

[0189] S903, correlation weight calculation.

[0190] For example, the correlation score of the intermediate layer k is calculated by the LRP method which is the core method of the Relevance-CAM method. represents the correlation score of the i-th channel of the k-th layer feature map for the target class c obtained by the LRP method.

[0191] The correlation scores of each channel are channel-level global average pooled to obtain the correlation weight in the channel dimension. The higher the weight value, the stronger the importance of the corresponding channel to the classification.

[0192] S904, constructing a correlation weighted gradient.

[0193] For example, the intermediate layer gradient is multiplied by the correlation weight element by element to obtain the correlation weighted gradient. Through the correlation weight, the gradient noise of the non-object region is suppressed, and the feature gradient related to the target object is highlighted, so that the perturbation is more focused on the key features across the model.

[0194] S905, loss function definition and image gradient calculation.

[0195] For example, the loss function is defined as the dot product of the correlation weighted gradient and the feature map. Minimizing this loss function can force the source model to reduce the response to the key features. By affecting the source model's response to key features, the attack on the source model is completed, and the robustness of the test source model is tested.

[0196] The gradient of the loss function with respect to the current adversarial sample x t adv This gradient indicates the direction of the image pixel perturbation.

[0197] S906, momentum iterative update perturbation.

[0198] For example, the momentum iterative fast gradient sign method (MI-FGSM) is used to iteratively update and obtain the momentum gradient accumulation. By fusing historical gradients to smooth the update direction, it is avoided to fall into local optimum. The gradient is L1 normalized to eliminate the influence of different scales.

[0199] Update the adversarial sample to ensure that the difference between the adversarial sample and the original image is visually and semantically imperceptible.

[0200] S907, iteration termination and result output.

[0201] ​The above S902 and S907 processes are repeated for T iterations, and the perturbation direction and strength are gradually optimized.

[0202] After the iteration is completed, the target adversarial sample is output, which can realize effective attack and test on the source model and the model with the same task type as the source model by perturbing the key features of the source model.

[0203] Exemplarily, Figure 10 A schematic diagram of a relevance gradient attack method provided by an embodiment of the present application is shown in Figure 10 The relevance gradient attack method (Relevance-based Gradient Attack, RGA) includes:

[0204] First, the original image is input to the source model, and the intermediate layer feature map corresponding to the original image is extracted from the intermediate layer of the source model.

[0205] Then, the channel-level correlation score is obtained by the correlation propagation rule in the process of reverse propagation of the source model from the output to the intermediate layer feature map, and the intermediate layer gradient is obtained by gradient reverse propagation. The correlation weighted gradient obtained by multiplying the channel-level correlation score and the intermediate layer gradient is used as the feature importance.

[0206] Secondly, the intermediate layer feature map and the correlation weighted gradient are multiplied to construct an adversarial attack loss function to determine the weighted feature map.

[0207] Subsequently, the target is optimized to generate an adversarial sample, and the generated adversarial sample is used to attack and test other models with the same task as the source model.

[0208] Exemplarily, in order to verify the effectiveness of the relevance gradient attack method provided by the present application, a large number of experiments are carried out on the RGA method provided by the present application and other excellent attack methods, the attack test is carried out on different models, the transfer attack performance of the adversarial sample generated in the RGA method provided by the present application on the normal training model and the defense model is explored, and the influence of different intermediate layers on the attack and test model results of the adversarial sample is explored. In addition, several loss functions are designed to use ablation experiments to discuss the contribution of the correlation weighted gradient to the RGA method provided by the present application.

[0209] 1. Experimental setup

[0210] (1) Dataset

[0211] Experiments are conducted on the ImageNet-compatible dataset used in the NIPS-2017 adversarial competition, which contains 1000 images. It can be determined that the classification success rate of all network models used for experiments on this dataset is almost close to 100%.

[0212] (2) Network model

[0213] Twelve classification models are selected as target models for evaluating the effects of different attacks, including seven normally trained models and five adversarially trained models (i.e., defense models). The normally trained models include Inception-V3 (Inc-v3), Inception-V4 (Inc-v4), Inception-ResNet-V2 (IncRes-v2), ResNet-V1-50 (Res-50), ResNet-V1-152 (Res-152), VGG16 (Vgg-16), and VGG19 (Vgg-19). The defense models include Adv-Inc-v3, Adv-IncRes-v2, Ens3-Inc-v3, Ens4-Inc-v3, and Ens-IncRes-v2.

[0214] In order to quantitatively compare the transferability between RGA and other attack methods, Vgg-16, Vgg-19, Res-50, and Res-152 are selected as source models to attack other normally trained models and defense models.

[0215] (3) Comparison method

[0216] In order to prove the effectiveness of the RGA method, experiments are compared with various advanced attack methods, including the Momentum Iterative Method (MIM) and the Patch-wise Iterative Method (PIM). In addition, other feature-level attack methods are also included in the comparison, such as the Feature Disruptive Attack (FDA) and the Neural Representation Distortion Method (NRDM).

[0217] (4) Evaluation index

[0218] Attack success rate refers to the ratio of adversarial samples that successfully mislead the target model to all generated adversarial samples. Therefore, the attack success rate of adversarial samples on the target model is used to evaluate the attack performance.

[0219] (5) Parameter setting

[0220] For parameter settings, the maximum perturbation e = 16, the number of iterations T = 10, and the step size a = e / T = 1.6 are set in all experiments. For the comparative methods, the experiments will use their default settings. For MIM, the decay factor m = 1.0; for PIM, the settings vary for different target models, which will be specified in detail in each related experiment (i.e. the amplification factor b, the projection factor g, the projection kernel size k w ) below. For the feature-level attack method NRDM, the FDA attack layer selection is set to the Conv3_3 layer in Vgg-16, the Conv3_4 layer in Vgg-19, the last layer of the second block in Res-50, and the last layer of the second block in Res-152.

[0221] 2. Transfer attack experiments on normally trained models

[0222] In the transfer attack experiments on normally trained models, b = 10, g = 16, and the projection kernel size k w = 3 in PIM. The intermediate layer selection for the RGA method attack is as follows: Conv3_3 in Vgg-16; Conv3_4 in Vgg-19; the last layer of block3 in Res-50; the last layer of block3 in Res-152. Tables 1 and 2 show the experimental results of attacking various normally trained models with Vgg and Resnet networks as source models, respectively.

[0223] RGA is significantly better than other attack methods that blindly generate perturbations, with an average of 8.7% improvement in attack success rate compared to these methods, which verifies the strong transfer attack ability of the RGA method.

[0224] Table 1 Attack success rates of different attack methods on normally trained models when the Vgg network is used as the source model. The first column is the source model, and the first row lists the target models. “*” indicates a white-box attack because the target model is the source model. The best result is highlighted in bold, and the second best result is underlined.

[0225]

[0226]

[0227] Table 2 Attack success rates of different attack methods on normally trained models when the Resnet network is used as the source model. The first column is the source model, and the first row lists the target models. “*” indicates a white-box attack because the target model is the source model. The best result is highlighted in bold, and the second best result is underlined.

[0228]

[0229] 3. Migration attack experiment on defense model

[0230] In the migration attack experiment on defense model, β = 10, γ = 16 in PIM, the projection kernel size k = 7, and the momentum term is removed because it can affect the performance of attacking the defense model. The middle layers selected for RGA method attack are as follows: Conv3_3 in Vgg-16; Conv3_4 in Vgg-19; the last layer of block3 in Res-50; the last layer of block3 in Res-152. w

[0231] Tables 3 and 4 respectively show the experimental results of attacking various defense models using Vgg and Resnet networks as source models. It can be found that compared with the experiment on the normally trained model, the attack success rate of various attack methods has decreased, because the defense model has stronger robustness to adversarial samples. The RGA method proposed in the present case is still significantly better than other advanced attack methods, with an average increase of 15.6% in attack success rate.

[0232] Table 3 Attack success rate of different attack methods on defense model when Vgg network is used as source model. The first column shows the source model, and the first row lists the target model. The best result is highlighted in bold, and the second best result is marked with an underline

[0233]

[0234]

[0235] Table 4 Attack success rate of different attack methods on defense model when Resnet network is used as source model. The first column shows the source model, and the first row lists the target model. The best result is highlighted in bold, and the second best result is marked with an underline

[0236]

[0237] 4. Attack layer selection experiment

[0238] An important factor affecting the performance of the RGA method provided in the present application is the target middle layer selected for attack. In order to determine which layer (shallow, middle or deep) can generate highly transferable adversarial samples, experiments are designed to use the RGA method to attack different middle layers to generate adversarial samples and observe their transferability.

[0239] The experiments respectively select Vgg-16 and Resnet-50 as source models, as shown in Figure 11 and select different target middle layers to generate adversarial samples to attack other normally trained models according to their network structures. ​

[0240] For Vgg-16 model, the last convolutional layer of each stage is selected as the target intermediate layer for attack, i.e. Conv1_2, Conv2_2, Conv3_3, Conv4_3 and Conv5_3, as shown in Figure 11 As shown in (a), when attacking the source model or the Vgg-19 model with similar network structure, the attack success rate of the adversarial samples generated on different intermediate layers is close to 100%, while when attacking other models, the best transfer attack effect can be achieved by attacking on the Conv3_3 layer.

[0241] For Resnet-50 model, the last layer of each Block, denoted as Layer1, Layer2, Layer3 and Layer4, is selected as the target intermediate layer for attack. As shown in (b), when performing white-box attack or attacking the Resnet-152 model with similar network structure, the attack success rate of the experimental structure in this paper is very high, while when attacking other models, the best transfer attack effect can be achieved by attacking on the Layer3 layer. For other source models, similar experiments are also performed for comparison, and finally the appropriate intermediate layer is selected as the final attack layer. Figure 11

[0242] Such experimental results can be explained as follows: the low-level features contained in the shallow layer have less impact on the output, while the deep layer contains high-level features, but attacking the deep layer overfits the source model and fails to produce highly transferable adversarial samples. Therefore, attacking the intermediate feature layer can achieve the best attack performance.

[0243] 6. Ablation experiment

[0244] The RGA method uses the relevance weighted gradient as the feature importance to guide the attack, and generates highly transferable adversarial samples by perturbing the important robust features of the image. In order to highlight the important role of the relevance weighted gradient, the following four loss functions are designed for ablation experiment to test the contribution of the relevance weighted gradient to the attack.

[0245] Among them Like most general methods, the feature perturbation is optimized without constraints. Directly use the gradient of the original sample in the intermediate feature layer to guide the attack. The relevance weight R k is used to guide the attack, i.e. the attack strategy of directly attacking the Relevance-CAM heat map is adopted. The method proposed in the application uses the relevance weighted gradient as the feature importance.​

[0246]

[0247] The application uses four objective functions respectively to perform attack experiments on four models of Vgg-16, Vgg-19, Resnet-50 and Resnet-152. As shown in Figs. 1(a), 1(b), 1(c) and 1(d), the attack success rates of the adversarial samples constructed based on different loss functions attacking the normally trained models are respectively shown. Figure 12

[0248] It can be found that the performance of the RGA method is always the best among the four losses, the performance of the RGA method based on the intermediate layer gradient of the original sample is the second, and the performance of the RGA method based on the correlation weight is the third, and the performance of the RGA method based on the correlation weight is the worst. It does not have any constraints to generate adversarial samples blindly, which shows that the correlation weighted gradient proposed by the RGA method is very effective for improving the migration performance of the attack.

[0249] For the same image, four loss functions are respectively applied to generate adversarial samples of the image with Vgg16 as the source model, Figure 13 and the heat maps of the adversarial samples are respectively shown. By comparison, it can be found that the heat map of the adversarial sample generated by taking the RGA method as the loss function is completely transferred to the target irrelevant background area, the adversarial sample generated by taking the RGA method based on the intermediate layer gradient of the original sample as the loss function fails to effectively disturb the judgment of the heat map, and the heat map still focuses on the target object main body, and the heat maps of the adversarial samples generated by the other two loss functions also have some areas remaining in the target related area. Through such intuitive comparison, it can also be shown that the correlation gradient attack method proposed in this chapter can effectively disturb the important robust features, so as to obtain highly migratable adversarial samples, which is also consistent with the experimental data results of the RGA method. Figure 12

[0250] The above mainly introduces the scheme provided by the embodiments of the application from the perspective of the method. In order to realize the above functions, it contains the hardware structure and / or software module corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of the examples described in the embodiments disclosed in the present text, the application can be realized in the form of hardware or the combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. The professional technicians can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the application. ​​​​​​​

[0251] The embodiments of the present application can divide the functional modules of the adversarial sample generation apparatus according to the above method examples. For example, each functional module can be divided according to each function, or two or more functions can be integrated in one processing module. The above integrated module can be realized in the form of hardware or in the form of a software functional module. Optionally, the division of the modules in the embodiments of the present application is illustrative, and is only a logical functional division. When actually implemented, another division mode can be used.

[0252] Figure 14 A structure diagram of an adversarial sample generation apparatus provided by an embodiment of the present application is shown. As shown in Figure 14 The adversarial sample generation apparatus includes a communication unit 1401 and a processing unit 1402.

[0253] The communication unit 1401 is configured to extract an intermediate layer feature map corresponding to an original image in a forward propagation process of a target network model, and obtain a correlation weight of each channel in the intermediate layer feature map.

[0254] The target network model is configured to perform prediction on the original image input into the target network model.

[0255] The processing unit 1402 is configured to determine a gradient of the intermediate layer feature map based on a target loss function of the target network model.

[0256] The target loss function is configured to determine a difference between an output result of the target network model and a real label.

[0257] The processing unit 1402 is further configured to determine a correlation weighted gradient of the intermediate layer feature map according to the correlation weight of each channel and the gradient of the intermediate layer feature map.

[0258] The processing unit 1402 is further configured to determine an adversarial loss function according to the correlation weighted gradient, and iteratively generate a target adversarial sample corresponding to the original image based on the adversarial loss function.

[0259] The target adversarial sample is configured to test a model of the same type as the target network model.

[0260] Optionally, the correlation weighted gradient satisfies the following formula:

[0261]

[0262] wherein, is the correlation weighted gradient; is the correlation weight of the i th channel of the intermediate layer feature map; c is a classification of the original image by the target network model; k is the k th intermediate layer of the target network model; i is the i th channel of the intermediate layer feature map; is a gradient of the intermediate layer feature map.

[0263] Optionally, the communication unit 1401 is specifically configured to: extract activation maps of each channel in the intermediate layer feature map; determine a relevance score of a channel corresponding to the activation map to obtain the relevance score of each channel; and determine the relevance weight of each channel according to the relevance score of each channel.

[0264] wherein the relevance weight of each channel satisfies the following formula:

[0265]

[0266] wherein, is the relevance weight of the i th channel; c is a classification of the original image by the target network model; k is the k th intermediate layer of the target network model; and i is the i th channel of the intermediate layer feature map. is the relevance score of the i th channel; and (x, y) is a pixel position in the i th channel.

[0267] Optionally, the gradient of the intermediate layer feature map satisfies the following formula:

[0268]

[0269] wherein, is the gradient of the intermediate layer feature map; k is the k th intermediate layer of the target network model; x is the original image; l(x, t) is a target loss function, representing a loss between a prediction result of the target network model and a real label t; and f k (x) is the intermediate layer feature map.

[0270] The processing unit 1402 is specifically configured to:

[0271] In the current iteration, the current adversarial sample is obtained.

[0272] wherein, when the current iteration is the first iteration, the current adversarial sample is the original image; and the current adversarial sample is the adversarial sample generated in the last iteration.

[0273] According to the current adversarial sample, the gradient of the intermediate layer feature map is updated to obtain an updated gradient of the intermediate layer feature map.

[0274] According to the relevance weight of each channel and the updated gradient of the intermediate layer feature map, the relevance weighted gradient of the intermediate layer feature map is updated to obtain an updated relevance weighted gradient.

[0275] According to the updated relevance weighted gradient, the adversarial loss function is updated to obtain an updated adversarial loss function.

[0276] Determine the gradient of the current adversarial sample based on the updated adversarial loss function.

[0277] Determine the current momentum gradient based on the gradient of the current adversarial sample.

[0278] Update the current adversarial sample based on the current momentum gradient to obtain the adversarial sample generated by the current iteration.

[0279] Repeat the above iteration process until the current iteration number reaches the preset total number of iterations, and determine the adversarial sample generated by the current iteration as the target adversarial sample.

[0280] Optionally, the current momentum gradient satisfies the following formula:

[0281]

[0282] wherein g t+1 The current momentum gradient; t is the current iteration number; μ is the momentum factor; g t is the momentum gradient of the tth iteration, and represents the cumulative result of the momentum gradient of the previous t iterations; is the updated adversarial loss function; is the gradient of the current adversarial sample, is the current adversarial sample; is the L1 norm of the gradient of the current adversarial sample; is the vector operator, which represents the gradient of the updated adversarial loss function with respect to the pixel value of the current adversarial sample.

[0283] Optionally, the current adversarial sample satisfies the following formula:

[0284]

[0285] wherein, is the updated current adversarial sample; Clip is the clipping symbol; ε is the maximum perturbation range; α is the step size, α = ε / T; t is the current iteration number; T is the total number of iterations; Sign is the sign function.

[0286] Optionally, the updated adversarial loss function satisfies the following formula:

[0287]

[0288] wherein, is the updated adversarial loss function; x adv is the current adversarial sample; is the updated correlation weighted gradient; k is the kth intermediate layer of the target network model; represents element multiplication; f k (x adv) is a feature map of the target network model at the kth layer.

[0289] The embodiment of the present application further provides a computer readable storage medium, which comprises computer execution instructions, and when the computer execution instructions run on a computer, the computer execution instructions make the computer execute the adversarial sample generation method provided in the above embodiment.

[0290] The embodiment of the present application further provides a computer program, which can be directly loaded into a memory and contains software codes, and the computer program can realize the adversarial sample generation method provided in the above embodiment after being loaded and executed by a computer.

[0291] Those skilled in the art can understand that, in one or more examples described above, the functions described in the present application can be realized by hardware, software, firmware or any combination thereof. When realized by software, the functions can be stored in a computer readable medium or transmitted as one or more instructions or codes on a computer readable medium. The computer readable medium includes a computer readable storage medium and a communication medium, wherein the communication medium includes any medium that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0292] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the convenience and brevity of description, only the above division of functional modules is taken as an example, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above.

[0293] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only illustrative, for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another device, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed units can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms. The units described as separate components can be or can not be physically separated, and the components displayed as units can be one physical unit or a plurality of physical units, that is, can be located in one place, or can be distributed to a plurality of different places. According to actual needs, part or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0294] In addition, each of the functional units in the embodiments of the present application can be integrated in one processing unit, or each unit can exist alone physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware or software function units. When the integrated unit is implemented in the form of software function units and sold or used as an independent product, it can be stored in a readable storage medium. Based on such an understanding, the technical solutions of the embodiments of the present application essentially, or the part that contributes to the prior art, or all or a part of the technical solutions can be embodied in the form of a software product. The software product is stored in a storage medium, including a number of instructions to make a device (which can be a single chip, chip, etc.) or a processor execute all or a part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, and various other storage media that can store program codes.

[0295] The above describes only specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for generating adversarial samples, characterized in that: include: During the forward propagation of the target network model, the intermediate layer feature map corresponding to the original image is extracted, and the correlation weight of each channel in the intermediate layer feature map is obtained; The target network model is used to predict the original image input into the target network model; Determining the gradient of the intermediate layer feature map based on the target loss function of the target network model; the target loss function is used to determine the difference between the output result of the target network model and the true label; Determining a correlation weighted gradient of the intermediate layer feature map according to the correlation weights of the channels and the gradient of the intermediate layer feature map; According to the correlation weighted gradient, an adversarial loss function is determined, and based on the adversarial loss function, a target adversarial sample corresponding to the original image is iteratively generated; the target adversarial sample is used to test a model with the same task type as the target network model.

2. The method according to claim 1, characterized in that Determining the correlation weighted gradient of the intermediate layer feature map according to the correlation weights of the channels and the gradient of the intermediate layer feature map includes: The relevance weighted gradient satisfies the following formula: in, weighting the gradient of the correlation; is the correlation weight of the i-th channel of the intermediate layer feature map; c is the classification of the original image by the target network model; k is the k-th intermediate layer of the target network model; i is the i-th channel of the intermediate layer feature map; is the gradient of the intermediate layer feature map.

3. The method according to claim 1, characterized in that The obtaining of the correlation weight of each channel in the intermediate layer feature map includes: Extracting activation maps of each channel in the intermediate layer feature map; Determining the correlation scores of the channels corresponding to the activation map to obtain the correlation scores of the channels; Determine the correlation weight of each channel according to the correlation score of each channel; the correlation weight of each channel satisfies the following formula: in, is the correlation weight of the i-th channel; c is the classification of the original image by the target network model; k is the k-th intermediate layer of the target network model; i is the i-th channel of the intermediate layer feature map; is the correlation score of the i-th channel; (x, y) is the pixel position in the i-th channel.

4. The method according to claim 1, wherein The determining the gradient of the intermediate layer feature map based on the target loss function of the target network model includes: The gradient of the intermediate layer feature map satisfies the following formula: in, is the gradient of the intermediate layer feature map; k is the kth intermediate layer of the target network model; x is the original image; l(x, t) is the target loss function, which represents the loss between the prediction result of the target network model and the true label t; f k (x) is the intermediate layer feature map.

5. The method according to claim 1, wherein The iteratively generating a target adversarial sample corresponding to the original image based on the adversarial loss function includes: In the current iteration, a current adversarial sample is obtained; when the current iteration is the first iteration, the current adversarial sample is the original image; the current adversarial sample is the adversarial sample generated in the previous iteration; According to the current adversarial example, updating the gradient of the intermediate layer feature map to obtain the gradient of the updated intermediate layer feature map; Updating the correlation weighted gradient of the intermediate layer feature map according to the correlation weights of the channels and the gradient of the updated intermediate layer feature map to obtain an updated correlation weighted gradient; Updating the adversarial loss function according to the updated relevance weighted gradient to obtain an updated adversarial loss function; Determining the gradient of the current adversarial example based on the updated adversarial loss function; Determining a current momentum gradient according to the gradient of the current adversarial sample; Based on the current momentum gradient, update the current adversarial sample to obtain the adversarial sample generated in the current iteration; Repeat the above iterative process until the current number of iterations reaches the preset total number of iterations, and determine the adversarial sample generated by the current iteration as the target adversarial sample.

6. The method according to claim 5, characterized in that The current momentum gradient satisfies the following formula: Among them, g t+1 Current momentum gradient; t is the current iteration number; μ is the momentum factor; g t is the momentum gradient of the tth iteration, which represents the cumulative result of the momentum gradient of the first t iterations; is the updated adversarial loss function; is the gradient of the current adversarial sample, is the current adversarial sample; is the L1 norm of the gradient of the current adversarial sample; A vector operator representing the gradient of the updated adversarial loss function with respect to the pixel value of the current adversarial sample.

7. The method according to claim 5, characterized in that The current adversarial sample satisfies the following formula: in, is the updated current adversarial sample; Clip is the clipping symbol; ε is the maximum perturbation range; α is the step size, α = ε / T; t is the current number of iterations; T is the total number of iterations; Sign is the sign function.

8. The method according to claim 5, characterized in that The updated adversarial loss function satisfies the following formula: in, is the updated adversarial loss function; x adv is the current adversarial sample; is the updated correlation weighted gradient; k is the kth intermediate layer of the target network model; ⊙ represents element multiplication; f k (x adv ) is the feature map of the adversarial sample in the k-th layer of the target network model.

9. A device for generating adversarial samples, characterized in that: The device includes: a communication unit and a processing unit; The communication unit is used to extract the intermediate layer feature map corresponding to the original image and obtain the correlation weight of each channel in the intermediate layer feature map during the forward propagation process of the target network model; the target network model is used to predict the original image input into the target network model; The processing unit is used to determine the gradient of the intermediate layer feature map based on the target loss function of the target network model; the target loss function is used to determine the difference between the output result of the target network model and the true label; The processing unit is further configured to determine a correlation weighted gradient of the intermediate layer feature map based on the correlation weights of the channels and the gradient of the intermediate layer feature map; The processing unit is further used to determine an adversarial loss function according to the correlation weighted gradient, and iteratively generate a target adversarial sample corresponding to the original image based on the adversarial loss function; the target adversarial sample is used to test a model with the same task type as the target network model.

10. A device for generating adversarial samples, characterized in that: include: A processor and a memory; wherein the memory is used to store one or more programs, and the one or more programs include computer-executable instructions. When the device is running, the processor executes the computer-executable instructions stored in the memory to enable the device to perform the method according to any one of claims 1 to 8.

Images