Sample generation, model training, image processing methods, devices, equipment and media

Abstract

The present disclosure provides a generation of image adversarial samples, training of image processing models, image processing methods, devices, training devices of image processing models, equipment and non-medium, and relates to the field of image processing technology, especially to the field of deep learning. The method for generating image adversarial samples includes: obtaining an original image sample, and obtaining a feature vector map corresponding to the original image sample; performing image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; using a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map; according to the attention influence map, adding perturbations to the original image sample to obtain an image adversarial sample corresponding to the original image sample. Such an operation not only avoids the introduction of too much calculation amount for obtaining training gradients of pixel points, but also makes the generated image adversarial samples more diverse and more flexible.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, in particular to the field of deep learning, and specifically to a method for generating image adversarial samples, a method for training image processing models, an image processing method, a device for generating image adversarial samples, a device for training image processing models, an image processing device, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] In many model training scenarios where there is a lack of training data, the accuracy, generalization, and adversarial nature of the model can be improved by constructing adversarial samples.

[0003] Adversarial samples refer to input samples formed by adding subtle interference to the data set, which are used to "mislead" the model to give an erroneous output with high confidence. In the context of regularization, the error rate of the original independent and identically distributed test set can be reduced by using adversarial training. Related technologies mainly use the training gradient of the original image to generate adversarial samples by generating corresponding interference on the original image.

[0004] The main defects of the relevant technology are that the adversarial sample is generally given a discrete value related to the gradient (a fixed hyperparameter of the model) based on the original image, and the options for this discrete value are only 0 and 1. The adversarial samples generated in this way lack flexibility and cannot cope with various image processing scenarios; in addition, since the training gradient of the original image needs to be introduced for calculation, the amount of calculation is generally large. Summary of the invention

[0005] The present disclosure provides a method for generating image adversarial samples, a method for training an image processing model, an image processing method, a device for generating image adversarial samples, a device for training an image processing model, an image processing device, an electronic device, and a non-transitory computer-readable storage medium.

[0006] According to one aspect of an embodiment of the present disclosure, a method for generating an image adversarial sample is provided, comprising:

[0007] Obtaining an original image sample and obtaining a feature vector map corresponding to the original image sample;

[0008] According to the image scale of the original image sample, the feature vector map is subjected to image scaling processing to obtain a standard scale feature map;

[0009] Using a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map;

[0010] According to the attention influence map, perturbations are added to the original image samples to obtain image adversarial samples corresponding to the original image samples.

[0011] According to another aspect of the embodiment of the present disclosure, a method for training an image processing model is also provided, including:

[0012] Acquire a plurality of original image samples, and generate image adversarial samples corresponding to the plurality of original image samples by using the method for generating an image adversarial sample as described in any embodiment of the present disclosure;

[0013] The original image samples and the image adversarial samples corresponding to the original image samples are used to train a preset deep learning model to obtain a trained image processing model.

[0014] According to another aspect of the present disclosure, there is also provided an image processing method, including:

[0015] Get the image to be processed;

[0016] The image to be processed is input into the image processing model trained by the image processing model training method as described in any embodiment of the present disclosure to obtain a processing result of the image to be processed.

[0017] According to another aspect of an embodiment of the present disclosure, a device for generating an image adversarial sample is provided, including:

[0018] A feature vector map acquisition module is used to acquire original image samples and acquire a feature vector map corresponding to the original image samples;

[0019] A standard scale feature map generation module, used to perform image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map;

[0020] An attention influence map generation module is used to process the standard scale feature map using a target type attention mechanism network to obtain an attention influence map;

[0021] The image adversarial sample generation module is used to add perturbations to the original image samples according to the attention influence map to obtain image adversarial samples corresponding to the original image samples.

[0022] According to another aspect of the embodiment of the present disclosure, there is also provided a training device for an image processing model, including:

[0023] An image adversarial sample generation module, used to obtain a plurality of original image samples, and generate image adversarial samples corresponding to the plurality of original image samples by generating image adversarial samples as described in any embodiment of the present disclosure;

[0024] The image processing model training module is used to use the original image samples and the image adversarial samples corresponding to the original image samples to train the preset deep learning model to obtain a trained image processing model.

[0025] According to another aspect of the present disclosure, there is also provided an image processing apparatus, including:

[0026] An image acquisition module, used for acquiring an image to be processed;

[0027] A processing result generating module is used to input the image to be processed into the image processing model trained by the training device of the image processing model described in any embodiment of the present disclosure to obtain a processing result of the image to be processed.

[0028] According to another aspect of the present disclosure, there is also provided an electronic device, including:

[0029] at least one processor; and

[0030] a memory communicatively connected to the at least one processor; wherein,

[0031] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating image adversarial samples as described in any embodiment of the present disclosure, execute the method for training the image processing model as described in any embodiment of the present disclosure, or execute the image processing method as described in any embodiment of the present disclosure.

[0032] According to another aspect of an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is further provided, wherein the computer instructions are used to enable a computer to execute the method for generating image adversarial samples according to any embodiment of the present disclosure, execute the method for training an image processing model according to any embodiment of the present disclosure, or execute the image processing method according to any embodiment of the present disclosure.

[0033] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0035] Figure 1 is a schematic diagram of a method for generating an image adversarial sample according to an embodiment of the present disclosure;

[0036] Figure 2 is a schematic diagram of another method for generating an image adversarial sample according to an embodiment of the present disclosure;

[0037] Figure 3 is a schematic diagram of the structure of a channel attention mechanism network applicable to the embodiments of the present disclosure;

[0038] Figure 4 is a schematic diagram of a complete process of generating an image adversarial sample applicable to an embodiment of the present disclosure;

[0039] Figure 5 is a schematic diagram of a training method for an image processing model provided according to an embodiment of the present disclosure;

[0040] Figure 6 is a schematic diagram of an image processing method provided according to an embodiment of the present disclosure;

[0041] Figure 7 is a schematic diagram of a device for generating an image adversarial sample according to an embodiment of the present disclosure;

[0042] Figure 8 is a schematic diagram of a training device for an image processing model provided according to an embodiment of the present disclosure;

[0043] Fig. 9 is a schematic diagram of an image processing device provided according to an embodiment of the present disclosure;

[0044] Fig.10 It is a block diagram of an electronic device used to implement the generation of image adversarial samples, training of image processing models or image processing methods of the embodiments of the present disclosure. DETAILED DESCRIPTION

[0045] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0046] Figure 1 1 is a schematic diagram of a method for generating an image adversarial sample according to an embodiment of the present disclosure. The embodiment of the present disclosure can be applied to the case of efficiently and flexibly generating an image adversarial sample corresponding to an original image sample. The method can be executed by a device for generating an image adversarial sample, which can be implemented in hardware and / or software and can generally be integrated in a terminal or server with data processing function.

[0047] Correspondingly, such as Figure 1 As shown, the method may specifically include:

[0048] S110, obtaining an original image sample, and obtaining a feature vector map corresponding to the original image sample.

[0049] The original image sample is a sample image used to train the set deep learning model. The type of the original image sample matches the training target of the deep learning model, for example, a face type or a multi-target type, etc., which is not limited in this embodiment.

[0050] The feature vector graph can be understood as the result obtained after extracting the multi-dimensional image features in the original image sample and arranging the multi-dimensional image features according to the image organization form.

[0051] Optionally, an image feature extraction model can be pre-trained, the input of which is a set image, and the output is an image feature map of the image. Alternatively, one or more image feature extraction layers in an existing image processing model, such as an image classification model or an image detection model, can be obtained, and the parameters of the image feature extraction layer can be fixed before use as the aforementioned image feature extraction model.

[0052] Generally speaking, the image scale of the feature vector map is much smaller than the image scale of the original image sample. For example, after feature extraction of a 1024*768 original image sample, a 32*32 feature vector map may be obtained.

[0053] S120 , performing image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard-scale feature map.

[0054] As mentioned above, since the feature vector map is generally inconsistent with the image scale of the original image sample, in order to facilitate subsequent processing, it is first necessary to convert the feature vector map into a standard scale feature map that is consistent with the image scale of the original image sample.

[0055] Continuing with the previous example, after obtaining a 32*32 feature vector map, it is necessary to first convert the 32*32 feature vector map into a standard scale feature map of 1024*768.

[0056] That is, the standard scale feature map has the same image scale as the original image sample, but each pixel position in the standard scale feature map carries not the pixel value of the pixel point, but the feature value of the image at the set dimension.

[0057] Optionally, the image scaling process can be performed on the feature vector image by directly calling the existing image scaling function resize(.). Alternatively, the image scaling process can be performed on the feature vector image by customizing the scaling mapping relationship, which is not limited in this embodiment.

[0058] S130, using a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map.

[0059] In this embodiment, the inventor comprehensively considers the various disadvantages of using the training gradient of each pixel in the original image sample to construct image perturbations. Instead of obtaining training gradients as in the related art, a new method of constructing image perturbations using an attention mechanism is proposed.

[0060] Among them, the attention mechanism is a mechanism that performs weighted processing on different parts of the input data in order to focus on important information more accurately. The types of attention mechanisms mainly include: channel attention mechanism, spatial attention mechanism, and hybrid attention mechanism. The inventors have found through multiple experimental studies that when an attention mechanism network is introduced to perform attention processing on an image, especially the image features of an image, the resulting processing graph can reflect the weight enhancement of important image features, and at the same time, the enhanced weight values ​​will not have obvious mutations. Based on this, the inventors creatively proposed to directly use the attention mechanism network to extract the tiny attention differences of the image features of the original image samples under a specific attention mechanism, and use the tiny attention differences (that is, the attention influence graph) as perturbations to finally generate image adversarial samples.

[0061] Optionally, the attention result map output by the target type's attention mechanism network can be directly used as the attention influence map, or the data at each pixel position in the attention result map can be normalized to limit the data at each pixel position to a set numerical range, for example, between [0, 1]. This embodiment does not impose any restrictions on this.

[0062] S140. Add disturbance to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample.

[0063] Among them, since the standard scale feature map is consistent with the image scale of the original image sample, the image scale of the attention influence map is also consistent with the image scale of the original image sample. Furthermore, the attention influence map can be directly used to add perturbations to the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0064] It can be understood that, through the above settings, the training gradients of each pixel in the original image sample are not used at all. Only by simply using an attention mechanism network, the perturbation information of the original image sample can be flexibly and variably obtained.

[0065] The technical solution of the disclosed embodiment obtains a feature vector map corresponding to the original image sample; performs image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; uses a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map; adds perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample, providing a new way to construct an image adversarial sample without using the training gradient of the original image sample. This operation not only avoids introducing too much computational effort in obtaining the training gradient of the pixel points, but also makes the generated image adversarial samples more diverse and more flexible; at the same time, by using the image features of the original image sample to generate the image adversarial sample, the generation effect of the image adversarial sample is more ideal. Furthermore, the training effect of the subsequent image processing model can be improved to a certain extent.

[0066] In an optional implementation of this embodiment, adding a perturbation to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample may include:

[0067] The attention influence map is pixel-superimposed with the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0068] As mentioned above, since the image scale of the attention influence map is consistent with that of the original image sample, and the value of each pixel position in the attention influence map is generally small, it is possible to directly add the attention influence map to the data of the same pixel position in the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0069] Through the above settings, image adversarial samples corresponding to the original image samples can be obtained simply and conveniently, which saves the computational complexity of the generation process and ensures the generation diversity and generation effect of the adversarial image samples.

[0070] Figure 2 Schematic diagram of another method for generating an image adversarial sample according to an embodiment of the present disclosure. This embodiment is refined based on the above embodiments. In this embodiment, the target type attention mechanism network is optimized to a channel attention mechanism network, and the specific structure of the channel attention mechanism network is limited.

[0071] Correspondingly, such as Figure 2 As shown, the method may specifically include:

[0072] S210: Obtain an original image sample, and obtain a feature vector map corresponding to the original image sample.

[0073] S220 , performing image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard-scale feature map.

[0074] S230: Input the standard scale feature map into a pre-built channel attention mechanism network to obtain an attention result map output by the channel attention mechanism network.

[0075] As mentioned above, in this embodiment, the target type attention mechanism network is concretized into a channel attention mechanism network. The core idea of ​​the channel attention mechanism network is to extract features in each convolution kernel channel, strengthen some interesting channel features, and weaken some uninteresting channel features.

[0076] Among them, Figure 3 A schematic diagram of the structure of a channel attention mechanism network applicable to the embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the channel attention mechanism network includes: a first convolution kernel module, a normalization module and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module and an adder connected in sequence.

[0077] The input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network.

[0078] Optionally, the first convolution kernel module, the second convolution kernel module and the third convolution kernel module can all be 1*1 convolution kernels, and only compress or expand channel features. The normalization module can be a module based on the Softmax activation function, which is used to convert the input channel features into probability values ​​between [0, 1]. The layer normalization module can be a module based on the ReLU activation function, which is a linear rectification function, referring to the mathematical ramp function.

[0079] The core idea of ​​the channel attention mechanism network provided by the embodiment of the present disclosure is to extract the channel relationship information between different channels in the input standard scale feature map by using multiple convolution kernels for operation, and finally, use the extracted channel relationship information to slightly modify the standard scale feature map to obtain the attention result map. By using the above structure to process the standard scale feature map, the difference between the attention result map and the standard scale feature map can be made as small as possible while ensuring that the difference between the attention result map and the standard scale feature map is slightly different. At the same time, the position where the difference occurs also has a certain randomness, thereby making the final attention result map more flexible.

[0080] S240: Input the attention result graph into a logistic regression module to obtain the attention influence graph output by the logistic regression module.

[0081] It can be understood that although the difference between the attention result map and the input standard scale feature map is not large, in order to further ensure that the above difference value is within a controllable numerical range, it is further considered to input the attention result map into the logistic regression module to ensure that the data of each pixel position in the attention result map is within a limited numerical range.

[0082] Among them, the logistic regression module is specifically a module based on the Sigmoid activation function. After the attention influence map is obtained through processing by the logistic regression module, it can be ensured that the value of each pixel position in the attention influence map falls within the range of (0,1).

[0083] S250, superimposing the attention influence map and the original image sample pixel by pixel to obtain an image adversarial sample corresponding to the original image sample.

[0084] The technical solution of the disclosed embodiment generates an attention influence map by using a channel attention mechanism network with a specific structure in combination with a logistic regression module, and uses the attention influence map as a perturbation to generate an image adversarial sample corresponding to the original image sample. This can effectively ensure the diversity and generation effect of the image adversarial samples and meet different application requirements of image adversarial samples.

[0085] On the basis of the above embodiments, before inputting the standard scale feature map into the pre-built channel attention mechanism network, the following may also be included:

[0086] Expanding the channel dimension of the standard scale feature map according to the number of processing channels of the channel attention mechanism network;

[0087] Correspondingly, after inputting the attention result graph into the logistic regression module and obtaining the attention influence graph output by the logistic regression module, the following steps may also be included:

[0088] The channel dimension of the attention influence map is reduced to the channel dimension state before expansion.

[0089] In this embodiment, considering that when the standard scale feature map is corrected using the channel attention mechanism network, the fewer the number of channels, the larger the correction value in a single channel may be, and the more the number of channels, the smaller the correction value in a single channel will be. Furthermore, in order to ensure the actual required slight jitter, so that each image feature in the standard scale feature map will not be corrected too much by the channel attention mechanism network, the channel dimension of the standard scale feature map can be expanded first according to the number of processing channels of the channel attention mechanism network (here it can be understood as the maximum number of channels that the channel attention mechanism network can correct), and after obtaining the attention influence map output by the logistic regression module, the channel dimension of the attention influence map is reduced to the channel dimension state before the expansion, so as to ensure that there is only a small and random difference value between the attention influence map and the standard scale feature map.

[0090] In order to clearly describe the technical solutions of various embodiments of the present disclosure, Figure 4 A schematic diagram of a complete image adversarial sample generation process applicable to the embodiments of the present disclosure is shown in FIG. Figure 4 As shown in the figure, after obtaining the feature vector map corresponding to the original image sample, the feature vector map is first converted into a standard scale feature map consistent with the image scale of the original image sample through the image scaling module. After that, a 1*1 convolution kernel can be used as a channel dimension expansion module to expand the maximum channel dimension of the standard scale feature map, and the standard scale feature map after the channel expansion is input into the channel attention mechanism network to obtain the attention result map, and then, a 1*1 convolution kernel is used as a channel dimension reduction module to restore the channel dimension of the attention result map, so that the channel dimension of the attention result map is consistent with the channel dimension of the standard scale feature map. Finally, the data of each pixel position in the attention result map is normalized to an output result of (0,1) through the logistic regression module to obtain the attention influence map.

[0091] Figure 5 1 is a schematic diagram of a training method for an image processing model provided according to an embodiment of the present disclosure. The embodiment of the present disclosure is applicable to the case where an image processing model is obtained by training an original image sample and an image adversarial sample corresponding to the original image sample. The method can be performed by a training device for an image processing model, which can be implemented in hardware and / or software and can generally be integrated in a terminal or server with data processing function.

[0092] Correspondingly, such as Figure 5 Said method may include:

[0093] S510: Acquire multiple original image samples, and generate image adversarial samples corresponding to the multiple original image samples by using the method for generating image adversarial samples as described in any embodiment of the present disclosure.

[0094] As mentioned above, in many model training scenarios where training data is lacking, the accuracy, generalization and adversarial performance of the model can be improved by constructing adversarial samples. The embodiments of the present disclosure are applied to the above application scenarios where original image samples are lacking.

[0095] Furthermore, through the technical solutions of the aforementioned embodiments, image adversarial samples corresponding to each original image sample can be generated.

[0096] S520: Use the original image samples and the image adversarial samples corresponding to the original image samples to train a preset deep learning model to obtain a trained image processing model.

[0097] Optionally, the method of using the original image sample and the image adversarial sample corresponding to the original image sample to train the preset deep learning model may be:

[0098] The original image sample and the adversarial image sample are spliced ​​to obtain a spliced ​​image sample; the spliced ​​image sample is input into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample; the loss of the original image sample and the loss of the adversarial image sample are determined according to the output result of the original image sample and the output result of the adversarial image sample; the parameters of the deep learning model are adjusted according to the loss of the original image sample and the loss of the adversarial image sample.

[0099] It can be understood that the original image samples and the image adversarial samples have different data distributions, and the loss function for hybrid training of the original image samples and the image adversarial samples is shown in the following formula.

[0100]

[0101] Among them, L(θ, x, y) represents the loss of the original image sample, represents the loss of the image adversarial sample, and E represents the comprehensive operation of the loss of the original image sample and the loss of the image adversarial sample, such as averaging or weighted summation. ∈ is a preset parameter, and θ is a parameter of the deep learning model.

[0102] This loss essentially includes the loss of the original image sample and the loss of the image adversarial sample. Hybrid training essentially involves combined training under two different distributions. In order to decompose this mixed distribution into two simpler distributions, the structure of the deep learning model can be improved.

[0103] The deep learning model includes at least one statistical layer, each statistical layer includes an original statistical sublayer and an auxiliary statistical sublayer;

[0104] Accordingly, inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample may include:

[0105] For each of the statistical layers, in response to the feature of the spliced ​​image sample being input into the statistical layer, the feature of the spliced ​​image sample is split into the feature of the original image sample and the feature of the adversarial image sample; the feature of the original image sample is input into the original statistical sublayer to obtain the feature distribution information of the original image sample; the feature of the adversarial image sample is input into the auxiliary statistical sublayer to obtain the feature distribution information of the adversarial image sample; the feature distribution information of the original image sample and the feature distribution information of the adversarial image sample are spliced ​​to obtain the output result of the statistical layer; and according to the output result of the statistical layer, the output result of the original image sample and the output result of the adversarial image sample are determined.

[0106] This embodiment inserts the auxiliary statistical sublayer in parallel with the original statistical sublayer, inputs the features of the original image samples into the original statistical sublayer, and inputs the features of the image adversarial samples into the auxiliary statistical sublayer. The original statistical sublayer counts the feature distribution information of the original image samples, and the auxiliary statistical sublayer counts the feature distribution information of the image adversarial samples. Thus, the deep learning model can count the feature distribution information of both real images and simulated images, thus expanding the knowledge domain of the deep learning model. For newly arrived real images, the original statistical sublayer can be selected for feature statistics to avoid counting the features into the wrong distribution curve, thereby ensuring the accuracy of the deep learning model in identifying the target object.

[0107] The technical solution of the disclosed embodiment uses image adversarial samples generated based on an attention mechanism network and combines them with original image samples to train a deep learning model. Since the generated image adversarial samples are more diverse and more flexible, the training effect of subsequent image processing models can be improved to a certain extent. In the model training scenario where original image samples are lacking, the accuracy, generalization and adversarial ability of the model can be effectively improved.

[0108] Figure 6is a schematic diagram of an image processing method provided according to an embodiment of the present disclosure, which can be applied to the case where an image processing model for generating image adversarial samples constructed using an attention mechanism network is used to generate image processing results for a specific scene. The method can be performed by an image processing device, which can be implemented in hardware and / or software and can generally be integrated in a terminal or server with data processing functions.

[0109] Correspondingly, such as Figure 6 Said method may include:

[0110] S610: Obtain an image to be processed.

[0111] The to-be-processed image may be understood as an image that needs to be processed in a set image processing scenario, such as an image detection scenario or a target object recognition scenario.

[0112] S620. Input the image to be processed into the image processing model trained by the image processing model training method described in any embodiment of the present disclosure to obtain a processing result of the image to be processed.

[0113] The technical solution of the disclosed embodiment uses an image processing model for generating image adversarial samples constructed by an attention mechanism network to generate an image processing result that is similar to the image to be processed. In scenarios where the number of original image samples of the image processing model is insufficient, the image processing model that meets the requirements of accuracy, generalization, and adversarial performance can be efficiently and accurately trained to meet the image processing needs in various scenarios.

[0114] As an implementation of the above-mentioned methods for generating the image adversarial samples, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned methods for generating the image adversarial samples.

[0115] Figure 7 is a schematic diagram of a device for generating an image adversarial sample according to an embodiment of the present disclosure. Figure 7 As shown, the image adversarial sample generation device includes: a feature vector map acquisition module 710, a standard scale feature map generation module 720, an attention influence map generation module 730 and an image adversarial sample generation module 740, wherein:

[0116] The feature vector map acquisition module 710 is used to acquire the original image sample and acquire the feature vector map corresponding to the original image sample;

[0117] A standard scale feature map generation module 720 is used to perform image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map;

[0118] An attention influence map generation module 730 is used to process the standard scale feature map using a target type attention mechanism network to obtain an attention influence map;

[0119] The image adversarial sample generation module 740 is used to add perturbations to the original image samples according to the attention influence map to obtain image adversarial samples corresponding to the original image samples.

[0120] The technical solution of the disclosed embodiment obtains a feature vector map corresponding to the original image sample; performs image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; uses a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map; adds perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample, providing a new way to construct an image adversarial sample without using the training gradient of the original image sample. This operation not only avoids introducing too much computational effort in obtaining the training gradient of the pixel points, but also makes the generated image adversarial samples more diverse and more flexible; at the same time, by using the image features of the original image sample to generate the image adversarial sample, the generation effect of the image adversarial sample is more ideal. Furthermore, the training effect of the subsequent image processing model can be improved to a certain extent.

[0121] On the basis of the above embodiments, the attention influence graph generation module may include:

[0122] An attention result map generating unit, used for inputting the standard scale feature map into a pre-built channel attention mechanism network to obtain an attention result map output by the channel attention mechanism network;

[0123] The logistic regression processing unit is used to input the attention result map into the logistic regression module to obtain the attention influence map output by the logistic regression module.

[0124] On the basis of the above embodiments, the channel attention mechanism network may include: a first convolution kernel module, a normalization module and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module and an adder connected in sequence;

[0125] The input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network.

[0126] On the basis of the above embodiments, a channel dimension expansion unit may be further included, which is used to expand the channel dimension of the standard scale feature map according to the number of processing channels of the channel attention mechanism network before inputting the standard scale feature map into the pre-constructed channel attention mechanism network;

[0127] And, a channel dimension reduction unit is used to reduce the channel dimension of the attention influence map to the channel dimension state before expansion after inputting the attention result map into the logistic regression module and obtaining the attention influence map output by the logistic regression module.

[0128] On the basis of the above embodiments, the image adversarial sample generation module can be specifically used for:

[0129] The attention influence map is pixel-superimposed with the original image sample to obtain an image adversarial sample corresponding to the original image sample.

[0130] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0131] As an implementation of the training methods for the above-mentioned image processing models, the present disclosure also provides an optional embodiment of an execution device for implementing the training methods for the above-mentioned image processing models.

[0132] Figure 8 Schematic diagram of a training device for an image processing model provided according to an embodiment of the present disclosure. Figure 8 As shown, the training device of the image processing model includes: an image adversarial sample generation module 810 and an image processing model training module 820, wherein:

[0133] An image adversarial sample generating module 810, configured to obtain a plurality of original image samples, and generate image adversarial samples corresponding to the plurality of original image samples by using the image adversarial sample generating device as described in any embodiment of the present disclosure;

[0134] The image processing model training module 820 is used to train a preset deep learning model using the original image samples and image adversarial samples corresponding to the original image samples to obtain a trained image processing model.

[0135] The technical solution of the disclosed embodiment uses image adversarial samples generated based on an attention mechanism network and combines them with original image samples to train a deep learning model. Since the generated image adversarial samples are more diverse and more flexible, the training effect of subsequent image processing models can be improved to a certain extent. In the model training scenario where original image samples are lacking, the accuracy, generalization and adversarial ability of the model can be effectively improved.

[0136] On the basis of the above embodiments, the image processing model training module may include:

[0137] A spliced ​​image sample generating unit, used for splicing the original image sample and the adversarial image sample to obtain a spliced ​​image sample;

[0138] An output result acquisition unit, used for inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the adversarial image sample;

[0139] A loss determination unit, configured to determine the loss of the original image sample and the loss of the adversarial image sample according to the output result of the original image sample and the output result of the adversarial image sample;

[0140] A parameter adjustment unit is used to adjust the parameters of the deep learning model according to the loss of the original image sample and the loss of the adversarial image sample.

[0141] Based on the above embodiments, the deep learning model may include at least one statistical layer, each statistical layer includes an original statistical sublayer and an auxiliary statistical sublayer;

[0142] The output result acquisition unit can be used for:

[0143] For each of the statistical layers, in response to the feature of the spliced ​​image sample being input into the statistical layer, the feature of the spliced ​​image sample is split into the feature of the original image sample and the feature of the adversarial image sample;

[0144] Inputting the features of the original image samples into the original statistical sublayer to obtain feature distribution information of the original image samples;

[0145] Inputting the features of the adversarial image sample into the auxiliary statistical sublayer to obtain feature distribution information of the adversarial image sample;

[0146] The feature distribution information of the original image sample and the feature distribution information of the adversarial image sample are concatenated to obtain the output result of the statistical layer;

[0147] According to the output result of the statistical layer, the output result of the original image sample and the output result of the adversarial image sample are determined.

[0148] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] As an implementation of the above-mentioned image processing methods, the present disclosure also provides an optional embodiment of an execution device for implementing the above-mentioned image processing methods.

[0150] Fig. 9 is a schematic diagram of an image processing device provided according to an embodiment of the present disclosure. Fig. 9 As shown, the image processing device includes: an image acquisition module 910 and a processing result generation module 920, wherein:

[0151] An image acquisition module 910 is used to acquire an image to be processed;

[0152] The processing result generating module 920 is used to input the image to be processed into the image processing model trained by the training device of the image processing model as described in any embodiment of the present disclosure, so as to obtain the processing result of the image to be processed.

[0153] The technical solution of the disclosed embodiment uses an image processing model for generating image adversarial samples constructed by an attention mechanism network to generate an image processing result that is similar to the image to be processed. In scenarios where the number of original image samples of the image processing model is insufficient, the image processing model that meets the requirements of accuracy, generalization, and adversarial performance can be efficiently and accurately trained to meet the image processing needs in various scenarios.

[0154] The above-mentioned product can execute the method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0155] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0156] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0157] Fig.10A schematic block diagram of an example electronic device 1000 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0158] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0159] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0160] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as performing the generation of image adversarial samples, the training of image processing models, or the image processing methods as described in any embodiment of the present disclosure. For example, in some embodiments, the generation of image adversarial samples, the training of image processing models, or the image processing methods as described in any embodiment of the present disclosure may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the generation of image adversarial samples, training of image processing models, or image processing methods described above as described in any embodiment of the present disclosure may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the generation of image adversarial samples, training of image processing models, or image processing methods as described in any embodiment of the present disclosure in any other appropriate manner (e.g., by means of firmware).

[0161] Specifically, the method for generating the image adversarial sample includes:

[0162] Obtaining an original image sample and obtaining a feature vector map corresponding to the original image sample;

[0163] According to the image scale of the original image sample, the feature vector map is subjected to image scaling processing to obtain a standard scale feature map;

[0164] Using a target type of attention mechanism network to process the standard scale feature map to obtain an attention influence map;

[0165] According to the attention influence map, perturbations are added to the original image samples to obtain image adversarial samples corresponding to the original image samples.

[0166] And, the training method of the image processing model includes:

[0167] Acquire a plurality of original image samples, and generate image adversarial samples corresponding to the plurality of original image samples by using the method for generating an image adversarial sample as described in any embodiment of the present disclosure;

[0168] The original image samples and the image adversarial samples corresponding to the original image samples are used to train a preset deep learning model to obtain a trained image processing model.

[0169] At the same time, the image processing method includes:

[0170] Get the image to be processed;

[0171] The image to be processed is input into the image processing model trained by the image processing model training method as described in any embodiment of the present disclosure to obtain a processing result of the image to be processed.

[0172] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0174] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0176] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0177] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or a cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services. The server may also be a server of a distributed system, or a server combined with a blockchain.

[0178] Artificial intelligence is a discipline that studies how to use computers to simulate certain human thought processes and intelligent behaviors (such as learning, reasoning, thinking, planning, etc.). It includes both hardware-level and software-level technologies. Artificial intelligence hardware technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, and big data processing; artificial intelligence software technologies mainly include computer vision technology, speech recognition technology, natural language processing technology, machine learning / deep learning technology, big data processing technology, knowledge graph technology, and other major directions.

[0179] Cloud computing refers to a technology system that uses network access to elastically scalable shared physical or virtual resource pools. Resources can include servers, operating systems, networks, software, applications, and storage devices, and can be deployed and managed on demand and in a self-service manner. Cloud computing technology can provide efficient and powerful data processing capabilities for technical applications such as artificial intelligence and blockchain, as well as model training.

[0180] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions provided by this disclosure can be achieved, and this document does not limit this.

[0181] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating an image adversarial sample, comprising: Obtaining an original image sample and obtaining a feature vector map corresponding to the original image sample; According to the image scale of the original image sample, the feature vector map is subjected to image scaling processing to obtain a standard scale feature map; Expanding the channel dimension of the standard scale feature map according to the number of processing channels of the channel attention mechanism network; Input the standard scale feature map into a pre-built channel attention mechanism network to obtain an attention result map output by the channel attention mechanism network; Input the attention result map into a logistic regression module to obtain an attention influence map output by the logistic regression module; Reducing the channel dimension of the attention influence map to the channel dimension state before expansion; According to the attention influence map, adding disturbance to the original image sample to obtain an image adversarial sample corresponding to the original image sample; The channel attention mechanism network includes: a first convolution kernel module, a normalization module and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module and an adder connected in sequence; The input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network; Among them, the attention influence map is consistent with the image scale of the original image sample, and the data of each pixel position in the attention influence map falls within the range of (0,1). By directly adding the attention influence map and the data of the same pixel position in the original image sample, an image adversarial sample corresponding to the original image sample is obtained.

2. A training method for an image processing model, comprising: Acquire a plurality of original image samples, and generate image adversarial samples corresponding to the plurality of original image samples by the method as claimed in claim 1; The original image samples and the image adversarial samples corresponding to the original image samples are used to train a preset deep learning model to obtain a trained image processing model.

3. The method according to claim 2, wherein: Using the original image sample and the image adversarial sample corresponding to the original image sample, training a preset deep learning model includes: Splicing the original image sample and the image adversarial sample to obtain a spliced ​​image sample; Inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the image adversarial sample; Determine the loss of the original image sample and the loss of the image adversarial sample according to the output result of the original image sample and the output result of the image adversarial sample; Adjusting parameters of the deep learning model according to the loss of the original image sample and the loss of the image adversarial sample.

4. The method according to claim 3, wherein: The deep learning model includes at least one statistical layer, each statistical layer includes an original statistical sublayer and an auxiliary statistical sublayer; The step of inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the image adversarial sample includes: For each of the statistical layers, in response to the feature of the spliced ​​image sample being input into the statistical layer, the feature of the spliced ​​image sample is split into the feature of the original image sample and the feature of the image adversarial sample; Inputting the features of the original image samples into the original statistical sublayer to obtain feature distribution information of the original image samples; Inputting the features of the image adversarial sample into the auxiliary statistical sublayer to obtain feature distribution information of the image adversarial sample; Concatenating the feature distribution information of the original image sample and the feature distribution information of the image adversarial sample to obtain an output result of the statistical layer; According to the output result of the statistical layer, the output result of the original image sample and the output result of the image adversarial sample are determined.

5. An image processing method, comprising: Get the image to be processed; The image to be processed is input into the image processing model trained by the method according to any one of claims 2 to 4 to obtain a processing result of the image to be processed.

6. A device for generating an image adversarial sample, comprising: A feature vector map acquisition module is used to acquire original image samples and acquire a feature vector map corresponding to the original image samples; A standard scale feature map generation module, used to perform image scaling processing on the feature vector map according to the image scale of the original image sample to obtain a standard scale feature map; A channel dimension expansion unit, used to expand the channel dimension of the standard scale feature map according to the number of processing channels of the channel attention mechanism network; An attention result map generating unit, used for inputting the standard scale feature map into a pre-built channel attention mechanism network to obtain an attention result map output by the channel attention mechanism network; A logistic regression processing unit, used to input the attention result map into a logistic regression module to obtain an attention influence map output by the logistic regression module; A channel dimension reduction unit, used to reduce the channel dimension of the attention influence map to the channel dimension state before expansion; An image adversarial sample generation module, used to add perturbations to the original image sample according to the attention influence map to obtain an image adversarial sample corresponding to the original image sample; The channel attention mechanism network includes: a first convolution kernel module, a normalization module and a multiplier connected in sequence, and a second convolution kernel module, a layer normalization module, a third convolution kernel module and an adder connected in sequence; The input end of the channel attention mechanism network is respectively connected to the first convolution kernel module, the first input end of the multiplier and the first input end of the adder, the output end of the normalization module is connected to the second input end of the multiplier, the output end of the multiplier is connected to the second convolution kernel, the output end of the third convolution kernel module is connected to the second input end of the adder, and the output end of the adder serves as the output end of the channel attention mechanism network; Among them, the attention influence map is consistent with the image scale of the original image sample, and the data of each pixel position in the attention influence map falls within the range of (0,1). By directly adding the attention influence map and the data of the same pixel position in the original image sample, an image adversarial sample corresponding to the original image sample is obtained.

7. A training device for an image processing model, comprising: An image adversarial sample generation module, configured to obtain a plurality of original image samples, and generate image adversarial samples corresponding to the plurality of original image samples through the apparatus as claimed in claim 6; The image processing model training module is used to use the original image samples and the image adversarial samples corresponding to the original image samples to train the preset deep learning model to obtain a trained image processing model.

8. The device according to claim 7, wherein: The image processing model training module comprises: A spliced ​​image sample generating unit, used for splicing the original image sample and the image adversarial sample to obtain a spliced ​​image sample; An output result acquisition unit, used for inputting the spliced ​​image sample into the deep learning model to obtain the output result of the original image sample and the output result of the image adversarial sample; A loss determination unit, configured to determine the loss of the original image sample and the loss of the image adversarial sample according to the output result of the original image sample and the output result of the image adversarial sample; A parameter adjustment unit is used to adjust the parameters of the deep learning model according to the loss of the original image sample and the loss of the image adversarial sample.

9. The apparatus according to claim 8, wherein the deep learning model comprises at least one statistical layer, each statistical layer comprising an original statistical sublayer and an auxiliary statistical sublayer; The output result acquisition unit is specifically used for: For each of the statistical layers, in response to the feature of the spliced ​​image sample being input into the statistical layer, the feature of the spliced ​​image sample is split into the feature of the original image sample and the feature of the image adversarial sample; Inputting the features of the original image samples into the original statistical sublayer to obtain feature distribution information of the original image samples; Inputting the features of the image adversarial sample into the auxiliary statistical sublayer to obtain feature distribution information of the image adversarial sample; Concatenating the feature distribution information of the original image sample and the feature distribution information of the image adversarial sample to obtain an output result of the statistical layer; According to the output result of the statistical layer, the output result of the original image sample and the output result of the image adversarial sample are determined.

10. An image processing device, comprising: An image acquisition module, used for acquiring an image to be processed; A processing result generating module is used to input the image to be processed into the image processing model trained by the device as described in any one of claims 7 to 9 to obtain a processing result of the image to be processed.

11. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method for generating image adversarial samples described in claim 1, execute the method for training the image processing model described in any one of claims 2-4, or execute the image processing method described in claim 5.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable a computer to execute the method for generating an image adversarial sample according to claim 1, to execute the method for training an image processing model according to any one of claims 2 to 4, or to execute the image processing method according to claim 5.

13. A computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the method for generating image adversarial samples described in claim 1, implements the method for training an image processing model described in any one of claims 2 to 4, or implements the image processing method described in claim 5.

Images