Method and apparatus for generating image adversarial sample, method and apparatus for training image processing model, image processing method and apparatus, and device and medium

Abstract

Provided in the present application are a method and apparatus for generating an image adversarial sample, a method and apparatus for training an image processing model, an image processing method and apparatus, and a device and a medium. The method for generating an image adversarial sample comprises: acquiring an original image sample, and acquiring a feature vector map corresponding to the original image sample (S110); performing image scaling processing on the feature vector map according to an image scale of the original image sample, so as to obtain a standard-scale feature map (S120); using an attention mechanism network of a target type to process the standard-scale feature map, so as to obtain an attention influence map (S130); and on the basis of the attention influence map, adding a disturbance to the original image sample, so as to obtain an image adversarial sample corresponding to the original image sample (S140).

Description

Image adversarial sample generation method and device, image processing model training method and device, image processing method and device, equipment, medium

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 20, 2023, with application number 202311370877.5, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of image processing technology, in particular to the field of deep learning, for example, to a method for generating image adversarial samples, a method for training image processing models, an image processing method, a device for generating image adversarial samples, a device for training image processing models, an image processing device, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0003] In many model training scenarios where there is a lack of training data, the accuracy, generalization, and adversarial performance of the model can be improved by constructing adversarial samples.

[0004] Adversarial examples are input samples created by adding subtle perturbations to a dataset, designed to mislead the model into producing an incorrect output with high confidence. Using adversarial training in a regularized context can reduce the error rate of an independent and identically distributed test set. Related techniques primarily utilize the training gradients of the original image to generate corresponding perturbations on the original image.

[0005] The main drawback of the relevant technology is that adversarial samples are generally given a discrete value related to the gradient (a fixed hyperparameter of the model) based on the original image, and the options for this discrete value are only 0 and 1. The adversarial samples generated in this way lack flexibility and cannot cope with a variety of image processing scenarios; in addition, since the training gradient of the original image needs to be introduced for calculation, the amount of calculation is generally large.

[0006] Summary of the Invention

[0007] The present application provides a method for generating image adversarial samples, a method for training an image processing model, an image processing method, a device for generating image adversarial samples, a device for training an image processing model, an image processing device, an electronic device, and a non-transitory computer-readable storage medium.

[0008] According to one aspect of an embodiment of the present application, a method for generating an image adversarial sample is provided, including: obtaining an original image sample and obtaining a feature vector map corresponding to the original image sample; performing image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard-scale feature map; using a target-type attention mechanism network to process the standard-scale feature map to obtain an attention influence map; and adding perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample.

[0009] According to another aspect of an embodiment of the present application, a training method for an image processing model is also provided, including: obtaining multiple original image samples, and generating multiple image adversarial samples corresponding to the multiple original image samples through the image adversarial sample generation method as described in any embodiment of the present application; using the original image samples and the image adversarial samples corresponding to the original image samples to train a preset deep learning model to obtain a trained image processing model.

[0010] According to another aspect of an embodiment of the present application, an image processing method is also provided, including: obtaining an image to be processed; inputting the image to be processed into the image processing model trained by the training method of the image processing model described in any embodiment of the present application, to obtain a processing result of the image to be processed.

[0011] According to another aspect of an embodiment of the present application, a device for generating an image adversarial sample is also provided, including: a feature vector map acquisition module, configured to acquire an original image sample and acquire a feature vector map corresponding to the original image sample; a standard scale feature map generation module, configured to perform image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; an attention influence map generation module, configured to use a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map; an image adversarial sample generation module, configured to add perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample.

[0012] According to another aspect of an embodiment of the present application, a training device for an image processing model is also provided, including: an image adversarial sample generation module, configured to obtain multiple original image samples, and generate multiple image adversarial samples corresponding to the multiple original image samples through the above-mentioned image adversarial sample generation device; an image processing model training module, configured to use the original image samples and the image adversarial samples corresponding to the original image samples to train a preset deep learning model to obtain a trained image processing model.

[0013] According to another aspect of an embodiment of the present application, an image processing device is also provided, including: an image acquisition module, configured to acquire an image to be processed; a processing result generation module, configured to input the image to be processed into the image processing model trained by the above-mentioned image processing model training device to obtain a processing result of the image to be processed.

[0014] According to another aspect of an embodiment of the present application, an electronic device is also provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above-mentioned method for generating image adversarial samples, execute the above-mentioned method for training an image processing model, or execute the above-mentioned image processing method.

[0015] According to another aspect of an embodiment of the present application, a non-transitory computer-readable storage medium storing computer instructions is also provided, wherein the computer instructions are used to enable a computer to execute the above-mentioned method for generating image adversarial samples, execute the above-mentioned method for training the image processing model, or execute the above-mentioned image processing method.

[0016] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, implements the above-mentioned method for generating image adversarial samples, implements the above-mentioned method for training the image processing model, or implements the above-mentioned image processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] FIG1 is a schematic diagram of a method for generating an image adversarial sample provided by an embodiment of the present application;

[0018] FIG2 is a schematic diagram of another method for generating image adversarial samples provided in an embodiment of the present application;

[0019] FIG3 is a schematic diagram of the structure of a channel attention mechanism network applicable to an embodiment of the present application;

[0020] FIG4 is a schematic diagram of a complete process for generating an image adversarial sample applicable to an embodiment of the present application;

[0021] FIG5 is a schematic diagram of a training method for an image processing model provided in an embodiment of the present application;

[0022] FIG6 is a schematic diagram of an image processing method provided in an embodiment of the present application;

[0023] FIG7 is a schematic diagram of a device for generating an image adversarial sample according to an embodiment of the present application;

[0024] FIG8 is a schematic diagram of an attention influence graph generation module provided in an embodiment of the present application;

[0025] FIG9 is a schematic diagram of another apparatus for generating an image adversarial sample according to an embodiment of the present application;

[0026] FIG10 is a schematic diagram of a training device for an image processing model provided in an embodiment of the present application;

[0027] FIG11 is a schematic diagram of an image processing model training module provided in an embodiment of the present application;

[0028] FIG12 is a schematic diagram of an image processing device provided in an embodiment of the present application;

[0029] Figure 13 is a block diagram of an electronic device used to implement the generation of image adversarial samples, training of image processing models, or image processing methods in an embodiment of the present application. DETAILED DESCRIPTION

[0030] The following describes exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding. They should be considered as merely exemplary. For the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0031] Figure 1 is a schematic diagram of a method for generating image adversarial samples provided by an embodiment of the present application. This embodiment of the present application is applicable to situations where image adversarial samples corresponding to original image samples are efficiently and flexibly generated. This method can be performed by an image adversarial sample generation device, which can be implemented in hardware and / or software and is generally integrated into a terminal or server with data processing capabilities.

[0032] As shown in FIG1 , the method may include:

[0033] S110: Obtain an original image sample, and obtain a feature vector map corresponding to the original image sample.

[0034] The original image samples are sample images used to train the deep learning model. The type of the original image samples matches the training target of the deep learning model. For example, the type of the original image samples can be a face type or a multi-object type, etc. This embodiment does not limit this.

[0035] The feature vector graph can be understood as the result obtained by extracting multi-dimensional image features from the original image sample and arranging the multi-dimensional image features according to the image organization form.

[0036] In one embodiment, an image feature extraction model can be pre-trained, where the input of the model is a given image and the output is an image feature map of the image. Alternatively, one or more image feature extraction layers can be obtained from an existing image processing model, such as an image classification model or an image detection model, and the parameters of the image feature extraction layer can be fixed before use as the aforementioned image feature extraction model.

[0037] Generally speaking, the image scale of the feature vector map is much smaller than the image scale of the original image sample. For example, after feature extraction of a 1024*768 original image sample, a 32*32 feature vector map may be obtained.

[0038] S120 , performing image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard-scale feature map.

[0039] As mentioned above, since the feature vector map is generally inconsistent with the image scale of the original image sample, in order to facilitate subsequent processing, the feature vector map needs to be first converted into a standard scale feature map that is consistent with the image scale of the original image sample.

[0040] Continuing with the previous example, after obtaining a 32*32 feature vector map, it is necessary to first convert the 32*32 feature vector map into a standard scale feature map of 1024*768.

[0041] That is, the standard scale feature map and the original image sample have the same image scale, but each pixel position in the standard scale feature map carries not the pixel value of the pixel point, but the feature value of the image at the set dimension.

[0042] In one embodiment, the image scaling process can be performed on the feature vector image by directly calling the image scaling function resize(.). Alternatively, the image scaling process can be performed on the feature vector image by customizing the scaling mapping relationship, which is not limited in this embodiment.

[0043] S130. Use a target-type attention mechanism network to process the standard-scale feature map to obtain an attention influence map.

[0044] In this embodiment, the multiple disadvantages of using the training gradient of each pixel in the original image sample to construct the image perturbation are comprehensively considered. Instead of obtaining the training gradient as in the related art, a new method of constructing the image perturbation by applying the attention mechanism is proposed.

[0045] The attention mechanism is a mechanism that performs weighted processing on different parts of the input data in order to focus on important information more accurately. The types of attention mechanisms mainly include: channel attention mechanism, spatial attention mechanism, and hybrid attention mechanism. Through multiple experimental studies, it is found that when an attention mechanism network is introduced to perform attention processing on an image, especially the image features of an image, the obtained processing result graph can not only reflect the weight enhancement of important image features, but also the enhanced weight value will not have obvious mutations. Based on this, the present application proposes to directly use the attention mechanism network to extract the tiny attention differences of the image features of the original image sample under a specific attention mechanism, and use the tiny attention differences (that is, the attention influence graph) as perturbations to finally generate image adversarial samples.

[0046] In one embodiment, the attention result map output by the target type's attention mechanism network can be directly used as the attention influence map, or the data at each pixel position in the attention result map can be normalized to limit the data at each pixel position to a set numerical range, for example, between [0, 1]. This embodiment does not impose any restrictions on this.

[0047] S140. Add perturbations to the original image sample according to the attention influence map to obtain image adversarial samples corresponding to the original image sample.

[0048] Since the standard-scale feature map is already consistent with the image scale of the original image sample, the image scale of the attention influence map is also consistent with the image scale of the original image sample. Furthermore, the attention influence map can be directly used to add perturbations to the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0049] With the above settings, the training gradients of the pixels in the original image sample are not used at all. By simply using an attention mechanism network, the perturbation information of the original image sample can be flexibly and variably obtained.

[0050] The technical solution of the embodiment of the present application obtains a feature vector map corresponding to the original image sample; performs image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; uses a target type attention mechanism network to process the standard scale feature map to obtain an attention influence map; adds perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample, providing a new way to construct an image adversarial sample without using the training gradient of the original image sample. This operation not only avoids the introduction of excessive computational complexity of obtaining pixel training gradients, but also makes the generated image adversarial samples more diverse and more flexible; at the same time, by using the image features of the original image sample to generate image adversarial samples, the generation effect of the image adversarial sample is more ideal. Furthermore, the training effect of the subsequent image processing model can be improved to a certain extent.

[0051] In one implementation of this embodiment, adding perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample may include: pixel-wise superposition of the attention influence map and the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0052] As mentioned above, since the attention influence map and the original image sample have the same image scale, and the value of each pixel position in the attention influence map is generally small, it is possible to directly add the attention influence map to the data at the same pixel position in the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0053] Through the above settings, image adversarial samples corresponding to the original image samples can be obtained simply and conveniently, which saves the computational complexity of the generation process while ensuring the generation diversity and generation effect of the adversarial image samples.

[0054] Figure 2 is a schematic diagram of another method for generating image adversarial samples provided by an embodiment of the present application. This embodiment is explained based on the above embodiment. In this embodiment, the target type of attention mechanism network is determined to be a channel attention mechanism network, and the structure of the channel attention mechanism network is limited.

[0055] As shown in FIG2 , the method may include:

[0056] S210: Obtain an original image sample, and obtain a feature vector map corresponding to the original image sample.

[0057] S220 , performing image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard-scale feature map.

[0058] S230: Input the standard scale feature map into a pre-built channel attention mechanism network to obtain an attention result map output by the channel attention mechanism network.

[0059] As mentioned above, in this embodiment, the target type of attention mechanism network is a channel attention mechanism network. The core idea of ​​the channel attention mechanism network is to extract features from each convolution kernel channel, strengthen some interesting channel features, and weaken some uninteresting channel features.

[0060] Figure 3 is a schematic diagram of the structure of a channel attention mechanism network applicable to an embodiment of the present application. As shown in Figure 3, the channel attention mechanism network includes: a first convolution kernel module, a normalization module, and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module, and an adder connected in sequence.

[0061] The input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network.

[0062] In one embodiment, the first convolution kernel module, the second convolution kernel module, and the third convolution kernel module can all be 1*1 convolution kernels, which only compress or expand channel features. The normalization module can be a module based on the Softmax activation function, which is used to convert the input channel features into probability values ​​between [0, 1]. The layer normalization module can be a module based on the ReLU activation function, which is a linear rectifier function, referring to a mathematical ramp function.

[0063] The core idea of ​​the channel attention mechanism network provided in the embodiments of this application is to extract the channel relationship information between different channels in the input standard-scale feature map by using multiple convolution kernels for calculation. Finally, using the extracted channel relationship information, the standard-scale feature map is slightly modified to obtain the attention result map. By using the above structure to process the standard-scale feature map, the difference between the attention result map and the standard-scale feature map can be minimized while ensuring that the difference is slightly different from the standard-scale feature map. At the same time, the location of the difference also has a certain degree of randomness, thereby making the final attention result map more flexible.

[0064] S240: Input the attention result graph into a logistic regression module to obtain the attention influence graph output by the logistic regression module.

[0065] Although the difference between the attention result map and the input standard scale feature map is not large, in order to ensure that the above difference value is within a controllable numerical range, the attention result map is input into the logistic regression module to ensure that the data of each pixel position in the attention result map is within a limited numerical range.

[0066] The logistic regression module is a module based on the Sigmoid activation function. After the attention influence map is obtained through processing by the logistic regression module, it can be ensured that the value of each pixel position in the attention influence map falls within the range of (0,1).

[0067] S250: Superimpose the attention influence map and the original image sample pixel by pixel to obtain an image adversarial sample corresponding to the original image sample.

[0068] The technical solution of the embodiment of the present application generates an attention influence map by using a channel attention mechanism network with a specific structure in combination with a logistic regression module, and uses the attention influence map as a perturbation to generate an image adversarial sample corresponding to the original image sample. This can effectively ensure the diversity and generation effect of the image adversarial sample and meet different application requirements of image adversarial samples.

[0069] On the basis of the above embodiment, before inputting the standard-scale feature map into the pre-constructed channel attention mechanism network, it may also include: expanding the channel dimension of the standard-scale feature map according to the number of processing channels of the channel attention mechanism network; correspondingly, after inputting the attention result map into the logistic regression module and obtaining the attention influence map output by the logistic regression module, it may also include: reducing the channel dimension of the attention influence map to the channel dimension state before expansion.

[0070] In this embodiment, considering that when using the channel attention mechanism network to correct the standard-scale feature map, the fewer the number of channels, the larger the correction value in a single channel may be, and the more the number of channels, the smaller the correction value in a single channel will be. Furthermore, in order to ensure the actual required slight jitter so that the image features in the standard-scale feature map are not excessively corrected by the channel attention mechanism network, the channel dimension of the standard-scale feature map can first be expanded according to the number of processing channels of the channel attention mechanism network (which can be understood here as the maximum number of channels that the channel attention mechanism network can correct). After obtaining the attention influence map output by the logistic regression module, the channel dimension of the attention influence map is reduced to the channel dimension state before the expansion to ensure that there is only a small and random difference between the attention influence map and the standard-scale feature map.

[0071] In order to clearly describe the technical solution of the embodiment of the present application, Figure 4 is a schematic diagram of a complete image adversarial sample generation process applicable to the embodiment of the present application. As shown in Figure 4, after obtaining the feature vector map corresponding to the original image sample, the feature vector map is first converted into a standard scale feature map consistent with the image scale of the original image sample through the image scaling module. Afterwards, a 1*1 convolution kernel can be used as a channel dimension expansion module to expand the maximum channel dimension of the standard scale feature map, and the standard scale feature map after the channel expansion is input into the channel attention mechanism network to obtain an attention result map, and then, a 1*1 convolution kernel is continued as a channel dimension reduction module to restore the channel dimension of the attention result map, so that the channel dimension of the attention result map is consistent with the channel dimension of the standard scale feature map. Finally, the data of each pixel position in the attention result map is normalized to an output result of (0,1) through the logistic regression module to obtain an attention influence map.

[0072] Figure 5 is a schematic diagram of a method for training an image processing model provided by an embodiment of the present application. The embodiment of the present application is applicable to the case where an image processing model is trained using both original image samples and image adversarial samples corresponding to the original image samples. The method can be performed by a training device for an image processing model, which can be implemented in hardware and / or software and can generally be integrated into a terminal or server with data processing capabilities.

[0073] As shown in FIG5 , the method may include:

[0074] S510: Acquire multiple original image samples, and generate image adversarial samples corresponding to the multiple original image samples using the method for generating image adversarial samples as described in any embodiment of the present application.

[0075] As mentioned above, in many model training scenarios where training data is lacking, the accuracy, generalization, and adversarial performance of the model can be improved by constructing adversarial samples. The embodiments of the present application are applied to the above application scenarios where original image samples are lacking.

[0076] Furthermore, the technical solutions of the aforementioned embodiments can be used to generate image adversarial samples corresponding to each original image sample.

[0077] S520: Use the original image sample and the image adversarial sample corresponding to the original image sample to train a preset deep learning model to obtain a trained image processing model.

[0078] In one embodiment, the method of training a preset deep learning model using the original image sample and the image adversarial sample corresponding to the original image sample may be:

[0079] The original image sample and the adversarial image sample are spliced ​​to obtain a spliced ​​image sample; the spliced ​​image sample is input into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample; the loss of the original image sample and the loss of the adversarial image sample are determined according to the output result of the original image sample and the output result of the adversarial image sample; the parameters of the deep learning model are adjusted according to the loss of the original image sample and the loss of the adversarial image sample.

[0080] The original image samples and image adversarial samples have different data distributions. The loss function for hybrid training of the original image samples and image adversarial samples is shown in the following formula.

[0081] Among them, L(θ, x, y) represents the loss of the original image sample, Denotes the loss of the image adversarial sample, and E represents the combined operation of the loss of the original image sample and the loss of the image adversarial sample, such as averaging or weighted summation. ∈ is a preset parameter, and θ is a parameter of the deep learning model.

[0082] This loss essentially consists of two parts: the loss of the original image sample and the loss of the image adversarial sample. Hybrid training essentially involves combined training under two different distributions. In order to decompose this mixed distribution into two simpler distributions, the structure of the deep learning model can be improved.

[0083] The deep learning model includes at least one statistical layer, and each statistical layer includes an original statistical sublayer and an auxiliary statistical sublayer; accordingly, inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample may include: for each statistical layer, in response to the feature of the spliced ​​image sample being input into the statistical layer, splitting the feature of the spliced ​​image sample into the feature of the original image sample and the feature of the adversarial image sample; inputting the feature of the original image sample into the original statistical sublayer to obtain the feature distribution information of the original image sample; inputting the feature of the adversarial image sample into the auxiliary statistical sublayer to obtain the feature distribution information of the adversarial image sample; splicing the feature distribution information of the original image sample and the feature distribution information of the adversarial image sample to obtain the output result of the statistical layer; and determining the output result of the original image sample and the output result of the adversarial image sample based on the output result of the statistical layer.

[0084] This embodiment inserts an auxiliary statistical sublayer in parallel with the original statistical sublayer. The features of the original image samples are input into the original statistical sublayer, and the features of the image adversarial samples are input into the auxiliary statistical sublayer. The original statistical sublayer calculates the feature distribution information of the original image samples, while the auxiliary statistical sublayer calculates the feature distribution information of the image adversarial samples. This allows the deep learning model to calculate the feature distribution information of both real images and simulated images, thus expanding the deep learning model's knowledge domain. For newly received real images, the original statistical sublayer can be selected for feature statistics, avoiding the inclusion of features in the incorrect distribution curve and ensuring the accuracy of the deep learning model in identifying the target object.

[0085] The technical solution of the embodiment of the present application uses image adversarial samples generated based on the attention mechanism network, combined with original image samples to train the deep learning model. Since the generated image adversarial samples are more diverse and flexible, the training effect of the subsequent image processing model can be improved to a certain extent. In the model training scenario where the original image samples are lacking, the accuracy, generalization and adversarial ability of the model can be effectively improved.

[0086] Figure 6 is a schematic diagram of an image processing method provided in an embodiment of the present application. This embodiment of the present application is applicable to generating image processing results for specific scenarios using an image processing model for generating adversarial image samples constructed using an attention mechanism network. The method can be performed by an image processing device, which can be implemented in hardware and / or software and can generally be integrated into a terminal or server with data processing capabilities.

[0087] As shown in FIG6 , the method may include:

[0088] S610: Acquire an image to be processed.

[0089] The image to be processed can be understood as an image that needs to be processed in a set image processing scenario, for example, an image detection scenario or an object recognition scenario.

[0090] S620. Input the image to be processed into the image processing model trained by the image processing model training method described in any embodiment of the present application to obtain a processing result of the image to be processed.

[0091] The technical solution of the embodiment of the present application uses an image processing model for generating image adversarial samples constructed by an attention mechanism network to generate image processing results that are the same as the image to be processed. In scenarios where the number of original image samples of the image processing model is insufficient, efficient and accurate training can be used to obtain an image processing model that meets the requirements of accuracy, generalization and adversarial properties, thereby meeting the image processing needs in multiple scenarios.

[0092] As an implementation of the above-mentioned method for generating image adversarial samples, the present application also provides an embodiment of an execution device for implementing the above-mentioned method for generating image adversarial samples.

[0093] FIG7 is a schematic diagram of an image adversarial sample generation device provided by an embodiment of the present application. As shown in FIG7, the image adversarial sample generation device includes: a feature vector map acquisition module 710, a standard scale feature map generation module 720, an attention influence map generation module 730, and an image adversarial sample generation module 740, wherein: the feature vector map acquisition module 710 is configured to acquire an original image sample and acquire a feature vector map corresponding to the original image sample; the standard scale feature map generation module 720 is configured to perform image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; the attention influence map generation module 730 is configured to use a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map; and the image adversarial sample generation module 740 is configured to add perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample.

[0094] The technical solution of the embodiment of the present application obtains a feature vector map corresponding to the original image sample; performs image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; uses a target type attention mechanism network to process the standard scale feature map to obtain an attention influence map; adds perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample, providing a new way to construct an image adversarial sample without using the training gradient of the original image sample. This operation not only avoids the introduction of excessive computational complexity of obtaining pixel training gradients, but also makes the generated image adversarial samples more diverse and more flexible; at the same time, by using the image features of the original image sample to generate image adversarial samples, the generation effect of the image adversarial sample is more ideal. Furthermore, the training effect of the subsequent image processing model can be improved to a certain extent.

[0095] Based on the above embodiment, Figure 8 is a schematic diagram of an attention influence map generation module provided in an embodiment of the present application. The attention influence map generation module 730 may include: an attention result map generation unit 731, configured to input the standard scale feature map into a pre-built channel attention mechanism network to obtain the attention result map output by the channel attention mechanism network; and a logistic regression processing unit 732, configured to input the attention result map into a logistic regression module to obtain the attention influence map output by the logistic regression module.

[0096] Based on the above embodiments, the channel attention mechanism network may include: a first convolution kernel module, a normalization module and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module and an adder connected in sequence; the input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network.

[0097] Based on the above embodiment, FIG9 is a schematic diagram of another apparatus for generating image adversarial samples provided by an embodiment of the present application. The apparatus for generating image adversarial samples may further include a channel dimension expansion unit 733 configured to expand the channel dimension of the standard-scale feature map according to the number of processing channels of the channel attention mechanism network before inputting the standard-scale feature map into the pre-built channel attention mechanism network; and a channel dimension reduction unit 734 configured to reduce the channel dimension of the attention influence map to the pre-expansion channel dimension state after inputting the attention result map into the logistic regression module and obtaining the attention influence map output by the logistic regression module.

[0098] On the basis of the above embodiment, the image adversarial sample generation module 740 may be configured to: perform pixel superposition of the attention influence map and the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0099] The above-mentioned product can execute the method provided in any embodiment of this application, and has the corresponding functional modules and effects of the execution method.

[0100] As an implementation of the training method for the above-mentioned image processing model, the present application also provides an embodiment of an execution device for implementing the training method for the above-mentioned image processing model.

[0101] FIG10 is a schematic diagram of a training device for an image processing model provided in an embodiment of the present application. As shown in FIG10 , the training device for the image processing model includes: an image adversarial sample generation module 810 and an image processing model training module 820, wherein: the image adversarial sample generation module 810 is configured to obtain a plurality of original image samples and generate image adversarial samples corresponding to the plurality of original image samples through the image adversarial sample generation device as described in any embodiment of the present application; and the image processing model training module 820 is configured to use the original image samples and the image adversarial samples corresponding to the original image samples to train a preset deep learning model to obtain a trained image processing model.

[0102] The technical solution of the embodiment of the present application uses image adversarial samples generated based on the attention mechanism network, combined with original image samples to train the deep learning model. Since the generated image adversarial samples are more diverse and flexible, the training effect of the subsequent image processing model can be improved to a certain extent. In the model training scenario where the original image samples are lacking, the accuracy, generalization and adversarial ability of the model can be effectively improved.

[0103] Based on the above embodiment, FIG11 is a schematic diagram of an image processing model training module provided by an embodiment of the present application. The image processing model training module 820 may include: a splicing image sample generation unit 821, configured to splice the original image sample and the adversarial image sample to obtain a spliced ​​image sample; an output result acquisition unit 822, configured to input the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample; a loss determination unit 823, configured to determine the loss of the original image sample and the loss of the adversarial image sample based on the output result of the original image sample and the output result of the adversarial image sample; and a parameter adjustment unit 824, configured to adjust the parameters of the deep learning model based on the loss of the original image sample and the loss of the adversarial image sample.

[0104] Based on the above embodiments, the deep learning model may include at least one statistical layer, each statistical layer including an original statistical sublayer and an auxiliary statistical sublayer; the output result acquisition unit 822 may be configured to: for each of the statistical layers, in response to the feature of the spliced ​​image sample being input into the statistical layer, split the feature of the spliced ​​image sample into the feature of the original image sample and the feature of the adversarial image sample; input the feature of the original image sample into the original statistical sublayer to obtain the feature distribution information of the original image sample; input the feature of the adversarial image sample into the auxiliary statistical sublayer to obtain the feature distribution information of the adversarial image sample; splice the feature distribution information of the original image sample and the feature distribution information of the adversarial image sample to obtain the output result of the statistical layer; and determine the output result of the original image sample and the output result of the adversarial image sample based on the output result of the statistical layer.

[0105] The above-mentioned product can execute the method provided in any embodiment of this application, and has the corresponding functional modules and effects of the execution method.

[0106] As an implementation of the above-mentioned image processing method, the present application also provides an embodiment of an execution device for implementing the above-mentioned image processing method.

[0107] Figure 12 is a schematic diagram of an image processing device provided in an embodiment of the present application. As shown in Figure 12, the image processing device includes: an image acquisition module 910 and a processing result generation module 920, wherein: the image acquisition module 910 is configured to acquire an image to be processed; the processing result generation module 920 is configured to input the image to be processed into the image processing model trained by the image processing model training device described in any embodiment of the present application, to obtain a processing result for the image to be processed.

[0108] The technical solution of the embodiment of the present application uses an image processing model for generating image adversarial samples constructed by an attention mechanism network to generate image processing results that are the same as the image to be processed. In scenarios where the number of original image samples of the image processing model is insufficient, efficient and accurate training can be used to obtain an image processing model that meets the requirements of accuracy, generalization and adversarial properties, thereby meeting the image processing needs in multiple scenarios.

[0109] The above-mentioned product can execute the method provided in any embodiment of this application, and has the corresponding functional modules and effects of the execution method.

[0110] In the technical solution of this application, the collection, storage, use, processing, transmission, provision and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0111] According to an embodiment of the present application, the present application also provides an electronic device, a readable storage medium and a computer program product.

[0112] FIG13 shows a schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0113] As shown in FIG13 , the device 1000 includes a computing unit 1001 that can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. Various programs and data required for the operation of the device 1000 can also be stored in the RAM 1003. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0114] Various components in device 1000 are connected to I / O interface 1005, including: an input unit 1006, such as a keyboard, mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a magnetic disk, optical disk, etc.; and a communication unit 1009, such as a network card, modem, wireless communication transceiver, etc. The communication unit 1009 allows device 1000 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0115] The computing unit 1001 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include a central processing unit (CPU), a graphics processing unit (GPU), a variety of dedicated artificial intelligence (AI) computing chips, a variety of computing units that run machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 1001 performs the multiple methods and processes described above, such as performing the generation of image adversarial samples, the training of image processing models, or the image processing method as described in any embodiment of the present application. For example, in some embodiments, the generation of image adversarial samples, the training of image processing models, or the image processing method as described in any embodiment of the present application can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the above-described generation of image adversarial samples, training of image processing models, or image processing methods as described in any embodiment of the present application may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the generation of image adversarial samples, training of image processing models, or image processing methods as described in any embodiment of the present application by any other appropriate means (e.g., by means of firmware).

[0116] The method for generating an image adversarial sample includes: obtaining an original image sample and obtaining a feature vector map corresponding to the original image sample; performing image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard-scale feature map; using a target-type attention mechanism network to process the standard-scale feature map to obtain an attention influence map; and adding perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample.

[0117] And, the training method of the image processing model includes: obtaining multiple original image samples, and generating image adversarial samples corresponding to the multiple original image samples through the image adversarial sample generation method as described in any embodiment of the present application; using the original image samples and the image adversarial samples corresponding to the original image samples to train a preset deep learning model to obtain a trained image processing model.

[0118] At the same time, the image processing method includes: obtaining an image to be processed; inputting the image to be processed into the image processing model trained by the image processing model training method described in any embodiment of the present application to obtain a processing result of the image to be processed.

[0119] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0120] The program code for implementing the methods of the present application can be written in any combination of one or more programming languages. Such program code can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the program code is executed by the processor or controller, the functions / operations specified in the flow charts and / or block diagrams are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0121] In the context of the present application, machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. Machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable medium can include electronic, magnetic, optical, electromagnetic, infrared or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Examples of machine-readable storage media can include electrical connections based on one or more lines, portable computer disks, hard disks, RAM, ROM, erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0122] To provide for user interaction, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or a liquid crystal display (LCD) monitor) configured to display information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be configured to provide for user interaction; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic input, voice input, or tactile input.

[0123] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0124] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact via a communication network. This client-server relationship is established by computer programs running on the respective computers, creating a client-server relationship. The server may be a cloud server, also known as a cloud computing server or cloud host, a host product within a cloud computing service ecosystem that addresses the management difficulties and limited scalability of traditional physical hosts and virtual private servers (VPS) services. The server may also be a server in a distributed system or a server integrated with blockchain.

[0125] Artificial intelligence (AI) is the study of how computers can simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, and planning). It encompasses both hardware and software technologies. AI hardware technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, and big data processing. AI software technologies primarily encompass computer vision, speech recognition, natural language processing, machine learning / deep learning, big data processing, and knowledge graphs.

[0126] Cloud computing refers to a technology system that provides network access to elastically scalable shared pools of physical or virtual resources. These resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on-demand in a self-service manner. Cloud computing technology provides efficient and powerful data processing capabilities for the application of technologies such as artificial intelligence and blockchain, as well as for model training.

[0127] The various forms of processes shown above can be used to reorder, add, or delete steps. For example, the multiple steps described in this application can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions provided in this application can be achieved. This document is not limited here.

Claims

1. A method for generating an image adversarial sample, comprising: Obtaining an original image sample, and obtaining a feature vector map corresponding to the original image sample; According to the image scale of the original image sample, the feature vector map is subjected to image scaling processing to obtain a standard scale feature map; Using a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map; According to the attention influence map, perturbations are added to the original image samples to obtain image adversarial samples corresponding to the original image samples.

2. The method according to claim 1, wherein: The target type attention mechanism network is used to process the standard scale feature map to obtain an attention influence map, including: Input the standard scale feature map into a pre-built channel attention mechanism network to obtain an attention result map output by the channel attention mechanism network; The attention result map is input into the logistic regression module to obtain the attention influence map output by the logistic regression module.

3. The method according to claim 2, wherein: The channel attention mechanism network includes: a first convolution kernel module, a normalization module and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module and an adder connected in sequence; The input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network.

4. The method according to claim 2 or 3, before inputting the standard scale feature map into the pre-built channel attention mechanism network, further comprising: Expanding the channel dimension of the standard scale feature map according to the number of processing channels of the channel attention mechanism network; After inputting the attention result graph into the logistic regression module and obtaining the attention influence graph output by the logistic regression module, the method further includes: The channel dimension of the attention influence map is reduced to the channel dimension state before expansion.

5. The method according to claim 1, wherein: The step of adding a disturbance to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample includes: The attention influence map is pixel-superimposed with the original image sample to obtain an image adversarial sample corresponding to the original image sample.

6. A method for training an image processing model, comprising: Acquire a plurality of original image samples, and generate a plurality of image adversarial samples corresponding to the plurality of original image samples by the method according to any one of claims 1 to 5; The original image samples and the image adversarial samples corresponding to the original image samples are used to train a preset deep learning model to obtain a trained image processing model.

7. The method according to claim 6, wherein: The using the original image sample and the image adversarial sample corresponding to the original image sample to train a preset deep learning model includes: Splicing the original image sample and the adversarial image sample to obtain a spliced ​​image sample; Inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample; Determine the loss of the original image sample and the loss of the adversarial image sample according to the output result of the original image sample and the output result of the adversarial image sample; Adjusting parameters of the deep learning model according to the loss of the original image sample and the loss of the adversarial image sample.

8. The method according to claim 7, wherein: The deep learning model includes at least one statistical layer, each statistical layer includes an original statistical sublayer and an auxiliary statistical sublayer; The step of inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample includes: For each statistical layer, in response to the feature of the spliced ​​image sample being input into the statistical layer, the feature of the spliced ​​image sample is split into the feature of the original image sample and the feature of the adversarial image sample; Inputting the features of the original image samples into the original statistical sublayer of the statistical layer to obtain feature distribution information of the original image samples; Inputting the features of the adversarial image samples into the auxiliary statistical sublayer of the statistical layer to obtain feature distribution information of the adversarial image samples; The feature distribution information of the original image sample and the feature distribution information of the adversarial image sample are concatenated to obtain the output result of the statistical layer; According to the output result of the statistical layer, the output result of the original image sample and the output result of the adversarial image sample are determined.

9. An image processing method, comprising: Get the image to be processed; The image to be processed is input into the image processing model trained by the method according to any one of claims 6 to 8 to obtain a processing result of the image to be processed.

10. A device for generating an image adversarial sample, comprising: A feature vector map acquisition module, configured to acquire an original image sample and acquire a feature vector map corresponding to the original image sample; A standard scale feature map generation module is configured to perform image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; An attention influence map generation module is configured to use a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map; The image adversarial sample generation module is configured to add perturbations to the original image samples according to the attention influence map to obtain image adversarial samples corresponding to the original image samples.

11. The device according to claim 10, wherein: The attention influence graph generation module includes: An attention result map generating unit is configured to input the standard scale feature map into a pre-constructed channel attention mechanism network to obtain an attention result map output by the channel attention mechanism network; The logistic regression processing unit is configured to input the attention result map into the logistic regression module and obtain the attention influence map output by the logistic regression module.

12. The device according to claim 11, wherein The channel attention mechanism network includes: a first convolution kernel module, a normalization module and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module and an adder connected in sequence; The input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network.

13. The apparatus according to claim 11 or 12, further comprising a channel dimension expansion unit, configured to expand the channel dimension of the standard scale feature map according to the number of processing channels of the channel attention mechanism network before inputting the standard scale feature map into the pre-constructed channel attention mechanism network; And, a channel dimension reduction unit, configured to input the attention result map into the logistic regression In the module, after obtaining the attention influence map output by the logistic regression module, the channel dimension of the attention influence map is reduced to the channel dimension state before expansion.

14. The device according to claim 10, wherein: The image adversarial sample generation module is set as: The attention influence map is pixel-superimposed with the original image sample to obtain an image adversarial sample corresponding to the original image sample.

15. A training device for an image processing model, comprising: An image adversarial sample generation module, configured to obtain a plurality of original image samples, and generate a plurality of image adversarial samples corresponding to the plurality of original image samples through the apparatus according to any one of claims 10 to 14; The image processing model training module is configured to use the original image samples and the image adversarial samples corresponding to the original image samples to train the preset deep learning model to obtain a trained image processing model.

16. The device according to claim 15, wherein: The image processing model training module comprises: a spliced ​​image sample generating unit, configured to splice the original image sample and the adversarial image sample to obtain a spliced ​​image sample; An output result acquisition unit, configured to input the spliced ​​image sample into the deep learning model to obtain an output result of the original image sample and an output result of the adversarial image sample; a loss determining unit, configured to determine the loss of the original image sample and the loss of the adversarial image sample according to the output result of the original image sample and the output result of the adversarial image sample; A parameter adjustment unit is configured to adjust the parameters of the deep learning model according to the loss of the original image sample and the loss of the adversarial image sample.

17. The apparatus according to claim 16, wherein the deep learning model comprises at least one statistical layer, each statistical layer comprising an original statistical sublayer and an auxiliary statistical sublayer; Output result acquisition unit, set to: For each statistical layer, in response to the feature of the spliced ​​image sample being input into the statistical layer, the feature of the spliced ​​image sample is split into the feature of the original image sample and the feature of the adversarial image sample; Inputting the features of the original image samples into the original statistical sublayer of the statistical layer to obtain feature distribution information of the original image samples; Inputting the features of the adversarial image samples into the auxiliary statistical sublayer of the statistical layer to obtain feature distribution information of the adversarial image samples; The feature distribution information of the original image sample and the feature distribution information of the adversarial image sample are concatenated to obtain the output result of the statistical layer; According to the output result of the statistical layer, the output result of the original image sample and the output result of the adversarial image sample are determined.

18. An image processing device, comprising: An image acquisition module, configured to acquire an image to be processed; The processing result generating module is configured to input the image to be processed into the image processing model trained by the device according to any one of claims 15 to 17 to obtain the processing result of the image to be processed.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating image adversarial samples described in any one of claims 1-5, execute the method for training an image processing model described in any one of claims 6-8, or execute the image processing method described in claim 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute a method for generating an image adversarial sample according to any one of claims 1-5, a method for training an image processing model according to any one of claims 6-8, or an image processing method according to claim 9.

21. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the method for generating an image adversarial sample as described in any one of claims 1 to 5, implements the method for training an image processing model as described in any one of claims 6 to 8, or implements the image processing method as described in claim 9.

Images