Image processing method, device, apparatus and storage medium

Abstract

This application discloses an image processing method, apparatus, device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: acquiring an original image; performing feature encoding processing on the original image to obtain a first feature map; acquiring a second feature map and a third feature map of the original image based on the first feature map; wherein the second feature map represents an image perturbation to be superimposed on the original image, and each position on the third feature map has different feature values, each feature value being used to characterize the importance of the image feature at the corresponding position; generating a noisy image based on the second and third feature maps; and superimposing the original image and the noisy image to obtain a first adversarial example. This application can generate high-quality adversarial examples, thereby achieving good attack effects.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, device, and storage medium. Background Technology

[0002] Methods that exploit the weaknesses of deep learning to impair the image recognition capabilities of image recognition models are collectively known as adversarial attacks. These attacks involve adding noise, imperceptible to the human eye, to an image, causing the deep learning-based image recognition model to fail in its task. In other words, the goal of an adversarial attack is to add perturbations to the original image that are difficult for the human eye to detect, thereby making the model's output completely inconsistent with the actual classification of the original image. Images with added noise that appear identical to the original image to the human eye are called adversarial examples.

[0003] Related technologies employ search- or optimization-based methods for adversarial attacks. These methods involve multiple forward operations and gradient calculations when generating adversarial examples, searching within a certain search space for perturbations that cause image recognition models to fail. This results in a significant time commitment for generating even a single adversarial example, which is unacceptable for scenarios with a large number of images, leading to poor timeliness. To address this issue, adversarial generative networks (GGNs) have been proposed. However, training a GGN involves a game between the generator and discriminator, which can cause unstable generated perturbations, resulting in inconsistent attack effectiveness.

[0004] As can be seen from the above description, it is currently impossible to achieve an effective attack. Therefore, how to perform image processing to generate high-quality adversarial examples has become a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] This application provides an image processing method, apparatus, device, and storage medium capable of generating high-quality adversarial examples, thereby achieving good attack results. The technical solution is as follows:

[0006] On the one hand, an image processing method is provided, the method comprising:

[0007] The original image is acquired, and feature encoding processing is performed on the original image to obtain a first feature map;

[0008] Based on the first feature map, a second feature map and a third feature map of the original image are obtained; wherein, the second feature map refers to the image perturbation to be superimposed on the original image, and each position on the third feature map has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position.

[0009] A noisy image is generated based on the second feature map and the third feature map;

[0010] The original image is superimposed on the noisy image to obtain the first adversarial sample.

[0011] On the other hand, an image processing apparatus is provided, the apparatus comprising:

[0012] The encoding module is configured to acquire the original image, perform feature encoding processing on the original image, and obtain a first feature map;

[0013] The decoding module is configured to obtain a second feature map and a third feature map of the original image based on the first feature map; wherein the second feature map refers to the image perturbation to be superimposed on the original image, and each position on the third feature map has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position;

[0014] The first processing module is configured to generate a noisy image based on the second feature map and the third feature map;

[0015] The second processing module is configured to overlay the original image with the noisy image to obtain a first adversarial sample.

[0016] In some embodiments, the encoding module is configured to:

[0017] The original image is input into the feature encoder of the adversarial attack network for feature encoding processing to obtain the first feature map, the size of which is smaller than the original image.

[0018] The feature encoder includes convolutional layers and residual blocks, with the residual blocks following the convolutional layers in the connection order. Each residual block includes an identity mapping and at least two convolutional layers, and the identity mapping of each residual block points from the input of the residual block to the output of the residual block.

[0019] In some embodiments, the decoding module includes a first decoding unit, which is configured to:

[0020] The first feature map is input into the first feature decoder of the adversarial attack network for first feature decoding processing to obtain the original noise feature map.

[0021] The noise feature values ​​at each position on the original noise feature map are suppressed to obtain the second feature map, and the size of the second feature map is the same as the size of the original image.

[0022] The first feature decoder includes a deconvolutional layer and a convolutional layer, wherein the convolutional layer is located after the deconvolutional layer in the connection order.

[0023] In some embodiments, the decoding module includes a first decoding unit, which is configured to:

[0024] The noise feature values ​​at each location on the original noise feature map are compared with the target threshold.

[0025] For any position on the original noise feature map, in response to the noise feature value at any position being greater than the target threshold, the noise feature value at any position is replaced with the target threshold.

[0026] In some embodiments, the decoding module further includes a second decoding unit, which is configured to:

[0027] The first feature map is input into the second feature decoder of the adversarial attack network for second feature decoding processing to obtain the third feature map of the original image;

[0028] The image feature values ​​at each position on the third feature map are normalized, and the size of the third feature map is the same as the size of the original image;

[0029] The second feature decoder includes a deconvolutional layer and a convolutional layer, wherein the convolutional layer is located after the deconvolutional layer in the connection order.

[0030] In some embodiments, the adversarial attack network further includes an image recognition model; the apparatus further includes a classification module; the classification module is configured to input the first adversarial sample into the image recognition model to obtain an image recognition result output by the image recognition model.

[0031] In some embodiments, the training process of the adversarial attack network includes:

[0032] Obtain second adversarial examples of sample images included in the training dataset;

[0033] The sample image and the second adversarial sample are input together into the image recognition model for feature encoding processing to obtain the feature data of the sample image and the feature data of the second adversarial sample;

[0034] Based on the feature data of the sample image and the feature data of the second adversarial example, a first loss function and a second loss function are constructed respectively.

[0035] Obtain the third feature map of the sample image. Each position on the third feature map of the sample image has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position.

[0036] Based on the third feature map of the sample image, a third loss function is constructed;

[0037] The adversarial attack network is obtained by performing end-to-end training based on the first loss function, the second loss function, and the third loss function.

[0038] In some embodiments, the training process of the adversarial attack network includes:

[0039] The feature angles of the sample image are extracted from the feature data of the sample image.

[0040] In the feature data of the second adversarial sample, the feature angle of the second adversarial sample is separated;

[0041] Based on the feature angles of the sample image and the feature angles of the second adversarial example, a first loss function is constructed. The optimization objective of the first loss function is to increase the feature angle between the sample image and the second adversarial example.

[0042] In some embodiments, the training process of the adversarial attack network includes:

[0043] In the feature data of the sample image, the feature magnitude of the sample image is extracted;

[0044] In the feature data of the second adversarial example, the feature magnitude value of the second adversarial example is separated;

[0045] Based on the feature magnitude of the sample image and the feature magnitude of the second adversarial example, a second loss function is constructed. The optimization objective of the second loss function is to reduce the difference in feature magnitude between the sample image and the second adversarial example.

[0046] In some embodiments, the training process of the adversarial attack network includes:

[0047] Obtain the first sum of the second loss function and the third loss function; and obtain the product of the target constant and the first sum.

[0048] The second sum of the product of the first sum and the first sum is used as the final loss function for end-to-end training to obtain the adversarial attack network.

[0049] In some embodiments, the first feature decoder and the second feature decoder of the adversarial attack network have the same structure.

[0050] On the other hand, a computer device is provided, the device including a processor and a memory, the memory storing at least one piece of program code, the at least one piece of program code being loaded and executed by the processor to implement the image processing method described above.

[0051] On the other hand, a computer-readable storage medium is provided, wherein at least one piece of program code is stored therein, the at least one piece of program code being loaded and executed by a processor to implement the image processing method described above.

[0052] On the other hand, a computer program product or computer program is provided, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the above-described image processing method.

[0053] The image processing scheme provided in this application requires only one forward operation to generate adversarial examples. Specifically, after extracting features from the original image to obtain a first feature map, a second feature map and a third feature map of the original image are obtained based on the first feature map. The second feature map represents an image perturbation that is difficult for the human eye to perceive and is to be superimposed onto the original image. The third feature map has different feature values ​​at each position, each feature value representing the importance of the image feature at that position. Then, a noisy image is generated based on the second and third feature maps. The adversarial example is obtained by superimposing the original image and the noisy image. Because this image processing method can quickly generate adversarial examples, it has good timeliness. Furthermore, the generated perturbation is stable, and the presence of the third feature map allows the noise to be concentrated in important areas, resulting in a higher quality adversarial example and thus effectively improving the attack effect.

[0054] In summary, the embodiments of this application can achieve good attack results against adversarial attacks. In terms of application, the embodiments of this application can effectively improve the resistance of image recognition models to adversarial attacks; that is, this image processing scheme can be used as a data augmentation method to optimize existing image recognition models, thereby improving the classification accuracy of existing image recognition models. Attached Figure Description

[0055] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0056] Figure 1 This is a schematic diagram of the implementation environment of an image processing method provided in an embodiment of this application;

[0057] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application;

[0058] Figure 3 This is a schematic diagram of the structure of an adversarial attack network provided in an embodiment of this application;

[0059] Figure 4 This is a schematic diagram of another adversarial attack network structure provided in an embodiment of this application;

[0060] Figure 5 This is a schematic diagram of the structure of a residual block provided in an embodiment of this application;

[0061] Figure 6 This is a flowchart of another image processing method provided in the embodiments of this application;

[0062] Figure 7 This is a flowchart of another image processing method provided in the embodiments of this application;

[0063] Figure 8 This is a schematic diagram of the training process of an adversarial attack network provided in an embodiment of this application;

[0064] Figure 9 This is a schematic diagram of an angle mode separation optimization loss function provided in an embodiment of this application;

[0065] Figure 10 This is a schematic diagram illustrating the result of an adversarial attack provided in an embodiment of this application;

[0066] Figure 11 This is a schematic diagram illustrating another counter-attack result provided in an embodiment of this application;

[0067] Figure 12 This is a schematic diagram illustrating another counter-attack result provided in an embodiment of this application;

[0068] Figure 13 This is a schematic diagram illustrating another counter-attack result provided in an embodiment of this application;

[0069] Figure 14 This is a schematic diagram illustrating another counter-attack result provided in an embodiment of this application;

[0070] Figure 15 This is a schematic diagram of the structure of an image processing device provided in an embodiment of this application;

[0071] Figure 16This is a schematic diagram of the structure of a computer device provided in an embodiment of this application;

[0072] Figure 17 This is a schematic diagram of the structure of another computer device provided in an embodiment of this application. Detailed Implementation

[0073] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0074] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms.

[0075] These terms are simply used to distinguish one element from another. For example, without departing from the various examples, the first element can be referred to as the second element, and similarly, the second element can be referred to as the first element. Both the first and second elements can be elements, and in some cases, they can be separate and distinct elements.

[0076] "At least one" means one or more elements. For example, at least one element can be one element, two elements, three elements, or any integer number of elements greater than or equal to one. "At least two" means two or more elements. For example, at least two elements can be two elements, three elements, or any integer number of elements greater than or equal to two.

[0077] The image processing solutions provided in this application relate to Artificial Intelligence (AI) technology. AI is the theory, methods, techniques, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0078] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0079] Machine learning (ML) is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0080] Optionally, embodiments of this application relate to deep residual networks (ResNet) in machine learning.

[0081] The depth of a neural network is crucial to its performance; ideally, as long as the network doesn't overfit, greater depth is better. However, a challenge arises in actual neural network training: as the network depth increases, gradients tend to vanish (gradient diffusion), making model optimization difficult and leading to a decrease in accuracy. In other words, increasing the depth of a neural network results in a degradation problem: accuracy initially increases, then saturates, and further increases in depth lead to a decrease in accuracy.

[0082] As described above, when the number of network layers reaches a certain level, the network performance saturates, and further increasing the network depth will cause performance degradation. However, this degradation is not caused by overfitting, because both training accuracy and testing accuracy decrease. This indicates that neural networks become difficult to train after reaching a certain depth. ResNet was developed to address this performance degradation problem as network depth increases. ResNet proposes a Deep Residual Learning (DRL) framework to solve this performance degradation problem caused by increasing network depth.

[0083] Suppose a relatively shallow network reaches saturation accuracy. Adding a few more identity mapping layers after this network will at least not increase the error; that is, deeper networks should not lead to an increase in error on the training set. The idea of ​​using identity mapping to directly pass the output of the previous layer to the next layer is the inspiration for ResNet.

[0084] For further explanation of ResNet, please refer to the following section.

[0085] The following describes some key terms or abbreviations that may be involved in the embodiments of this application.

[0086] Adversarial attacks occur when noise, imperceptible to the human eye, is added to an image (also called the original image), causing deep learning-based image recognition models to fail in their tasks. In other words, the goal of an adversarial attack is to add perturbations to the original image that are difficult for the human eye to detect, thus making the image recognition model's classification completely inconsistent with the actual classification of the original image. Images with added noise that appear identical to the original image to the human eye are called adversarial examples or attack images.

[0087] In other words, the original image and the adversarial example are visually identical; they possess visual consistency, which makes it impossible for the human eye to distinguish subtle differences between them. That is, visual consistency means that after adding imperceptible perturbations to the original image to obtain the adversarial example, the original image and the adversarial example appear identical to the human eye, and the human eye cannot distinguish subtle differences between them.

[0088] The implementation environment of the image processing method provided in the embodiments of this application will be described below.

[0089] See Figure 1 The implementation environment includes: training equipment 110 and application equipment 120.

[0090] During the training phase, training device 110 performs end-to-end training based on a defined loss function to obtain an adversarial network (also called an autoencoder) for adversarial attacks. During the application phase, application device 120 can use the autoencoder to generate adversarial examples of the input original image. In other words, during the training phase, an autoencoder for generating adversarial examples is obtained through end-to-end training; correspondingly, during the application phase, for an input original image, the autoencoder can generate an adversarial example that appears identical to the original image to the human eye, which can then be used to attack the image recognition model.

[0091] In summary, the image processing scheme provided in this application uses a trained autoencoder to generate an image perturbation (to obtain a noisy image), and then superimposes the generated image perturbation (i.e., the noisy image) onto the original image to generate an adversarial sample, thereby causing the image recognition model to misidentify the adversarial sample.

[0092] Optionally, the training device 110 and application device 120 described above are computer devices, such as terminals or servers. In some embodiments, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.

[0093] In another embodiment, the training device 110 and the application device 120 may be the same device, or they may be different devices. Furthermore, when the training device 110 and the application device 120 are different devices, they may be of the same type, such as both being terminals; or they may be of different types, such as the training device 110 being a server and the application device 120 being a terminal. This application does not impose any limitations on these aspects.

[0094] The image processing scheme provided in the embodiments of this application will be described below through the following implementation methods.

[0095] Figure 2 This is a flowchart of an image processing method provided in an embodiment of this application. See also... Figure 2 In the application phase, the method flow provided in this application embodiment includes:

[0096] 201. Obtain the original image, perform feature encoding processing on the original image, and obtain the first feature map.

[0097] Optionally, the original image is an RGB (Red Green Blue) image; alternatively, the original image can be an image including people or objects (such as animals or plants), and this application does not impose any restrictions on this. The original image is referred to herein by the symbol I.

[0098] In some embodiments, feature encoding processing is performed on the original image to obtain a first feature map, including but not limited to the following methods: inputting the original image... Figure 3 The feature encoder 301 of the adversarial attack network shown performs feature encoding processing to obtain a first feature map. This feature encoding processing is also known as feature extraction processing, and the size of the first feature map is smaller than the original image.

[0099] Optionally, see Figure 4 The feature encoder 301 employs a convolutional neural network, including convolutional layers and residual blocks (ResBlocks), wherein the residual blocks are located after the convolutional layers in the connection order. For example, as... Figure 4 As shown, the feature encoder 301 includes multiple convolutional layers and multiple ResBlocks connected in sequence, such as three convolutional layers and six ResBlocks, which is not limited in this application. Furthermore, the kernel sizes of the multiple convolutional layers can be the same or different, which is also not limited in this application.

[0100] by Figure 4 Taking the feature encoder structure shown as an example, assuming the input size of the original image is w*h and the number of channels is 3, after the first convolutional layer, the width (w) and height (h) of the original image become half of their original values, and the number of channels increases from 3 to 32, forming a feature map of w / 2 * h / 2 * 128. After the second convolutional layer, the width (w) and height (h) of the original image become one-quarter of their original values, and the number of channels increases from 32 to 64, forming a feature map of w / 4 * h / 4 * 64. After the third convolutional layer, the width (w) and height (h) of the original image become one-quarter of their original values, and the number of channels increases from 64 to 128, forming a feature map of w / 2 * h / 2 * 128. This feature map then passes through a sub-network consisting of six ResBlocks to generate a new feature map. In other words, after six ResBlocks, we obtain a feature map of w / 4 * h / 4 * 128. The first feature map of 128 is the feature map obtained after the original image has undergone feature encoding processing by the feature encoder 301.

[0101] Optionally, each residual block includes an identity mapping layer and at least two convolutional layers. The identity mapping of any residual block points from the input to the output of any residual block. Specifically, for any set A, if the mapping f: A → A is defined as f(a) = a, meaning that each element a in A corresponds to itself, then f is called the identity mapping on A.

[0102] The following section provides a detailed explanation of deep residual networks.

[0103] Suppose the input to a neural network is x, and the desired network layer relationship mapping is H(x). Let stacked nonlinear layers fit another mapping F(x) = H(x) - x. Then the original mapping H(x) becomes F(x) + x. Assuming that optimizing the residual mapping F(x) is easier than optimizing the original mapping H(x), we first calculate the residual mapping F(x). Then the original mapping is F(x) + x, and F(x) + x can be implemented using shortcut connections.

[0104] Figure 5 A schematic diagram of a residual block is shown. Figure 5 As shown, each residual block in a deep residual network includes an identity mapping and at least two convolutional layers. The identity mapping of a residual block points from its input to its output.

[0105] That is, by adding an identity mapping, the original function H(x) to be learned is transformed into F(x)+x. Although these two expressions achieve the same effect, the optimization difficulty is different. By using reformulation to decompose a problem into a residual problem between multiple scales, the optimization training effect can be effectively achieved. Figure 5 As shown, this residual block is implemented through Shortcut connections. By superimposing the input and output of this residual block through Shortcut connections, the training speed of the model is greatly increased and the training effect is improved without adding extra parameters and computation to the network. Furthermore, this simple structure can effectively solve the degradation problem when the number of layers in the model increases.

[0106] In other words, H(x) is a complex latent mapping of expectation, which is difficult to learn. If we directly... Figure 5 The Shortcut connection in ResNet passes the input x to the output as the initial result. Therefore, the target to be learned is F(x) = H(x) - x. Thus, the ResNet network essentially changes its learning objective; it no longer learns a complete output, but rather the difference between the optimal solution H(x) and the identity mapping x, i.e., the residual mapping F(x). It's important to note that "Shortcut" originally refers to a shortcut, but in this paper it refers to a connection that bypasses layers. In ResNet, the Shortcut connection has no weights; after passing x, each residual block only learns the residual mapping F(x). Furthermore, because the network is stable and easy to learn, its performance gradually improves with increasing network depth. Therefore, when the network has sufficient layers, optimizing the residual mapping F(x) = H(x) - x will easily optimize a complex nonlinear mapping H(x).

[0107] Based on the above description, it can be seen that, compared to ordinary directly connected convolutional neural networks, ResNet networks have many bypass branches that directly connect the input to subsequent layers, allowing those layers to directly learn the residuals. This structure is called Shortcut Connection. Traditional convolutional or fully connected layers often suffer from information loss or degradation during information transmission. ResNet networks solve this problem to some extent by directly routing the input to the output, preserving the integrity of the information. The entire network only needs to learn the difference between the input and output, simplifying the learning objective and reducing the difficulty.

[0108] It should be noted that the first feature map obtained by the feature encoder 301 will be input into the first feature decoder (also known as the noise decoder) 302 and the second feature decoder (also known as the salience region decoder) 303 of the adversarial attack network. Figure 3 Since the first feature decoder 302 and the second feature decoder 303 have a symmetrical structure, and this paper proposes the concept of salient regions, this adversarial attack network is also called a symmetric autoencoder based on salient regions. For details, please refer to step 202 below.

[0109] 202. Based on the first feature map, obtain the second feature map and the third feature map of the original image; wherein, the second feature map refers to the image perturbation to be superimposed on the original image, and each position on the third feature map has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position.

[0110] Optionally, this step is performed by Figure 3 The first feature decoder 302 and the second feature decoder 303 in the illustrated adversarial attack network are executed accordingly. Figure 2 Step 202 in the text can be replaced with Figure 6 Steps 2021 to 2024 in the process.

[0111] 2021. Input the first feature map into the first feature decoder of the adversarial attack network for first feature decoding processing to obtain the original noise feature map.

[0112] In some embodiments, see Figure 4 The first feature decoder 302 includes deconvolutional layers and convolutional layers, wherein the convolutional layers are connected after the deconvolutional layers. For example, as... Figure 4 As shown, the first feature decoder 302 includes two 3x3 deconvolutional layers and one 7x7 convolutional layer. The deconvolutional layers transform the smaller input feature map into a larger one.

[0113] like Figure 4As shown, the feature map input to the first feature decoder 302 is a w / 4 * h / 4 * 128 first feature map obtained after encoding by the feature encoder 301. This first feature map becomes a w / 2 * h / 2 * 64 feature map after passing through the first 3x3 deconvolution layer; it becomes a w * h * 32 feature map after passing through the second 3x3 deconvolution layer; and it becomes a w * h * 3 feature map after passing through a 7x7 convolution layer, which is the original noise feature map. The original noise feature map is referred to as N0 in this paper.

[0114] 2022. The noise feature values ​​at each position on the original noise feature map are suppressed to obtain the second feature map of the original image.

[0115] Optionally, to avoid excessive noise, embodiments of this application impose a restriction on the noise feature values ​​of the original noise feature map, thereby obtaining a second feature map. This suppression processing of the noise feature values ​​at each location on the original noise feature map includes, but is not limited to: comparing the noise feature values ​​at each location on the original noise feature map with a target threshold; for any location on the original noise feature map, in response to the noise feature value at any location being greater than the target threshold, replacing the noise feature value at that location with the target threshold. This can be expressed as the following formula:

[0116]

[0117] in min(a,b) means taking the smallest of a and b; is a hyperparameter that refers to the target threshold mentioned above, which is used to limit the maximum value of the noise feature; The smaller the value, the less noise is generated, and the less likely it is to be detected by the human eye when superimposed on the original image, resulting in a better quality of the final attack image.

[0118] The second feature map is referred to by the symbol N in this paper, and the second feature map of the original image I can be represented as N(I). Furthermore, the size of the second feature map is the same as the size of the original image. Additionally, this second feature map represents the noise to be superimposed on the original image, i.e., image perturbation.

[0119] 2023. The first feature map is input into the second feature decoder of the adversarial attack network for second feature decoding processing to obtain the third feature map of the original image.

[0120] In some embodiments, the second feature decoder 303 includes a deconvolution layer and a convolutional layer, wherein the convolutional layer is located after the deconvolutional layer in the connection order.

[0121] Optionally, such as Figure 4As shown, the second feature decoder 303 has the same structure as the first feature decoder 302. That is, the salient region decoder and the noise decoder have the same structure, consisting of two 3x3 deconvolutional layers and one 7x7 convolutional layer. The input to the salient region decoder is also the output of the first feature encoder 301, and the output of the salient region decoder is the salient region feature map of the original image. In detail, as... Figure 4 As shown, the feature map input to the first feature decoder 302 is a w / 4 * h / 4 * 128 first feature map obtained by the feature encoder 301. This first feature map becomes a w / 2 * h / 2 * 64 feature map after passing through the first 3x3 deconvolution layer of the second feature decoder 303; after passing through the second 3x3 deconvolution layer, it becomes a w * h * 32 feature map; and after passing through a 7x7 convolution layer, it becomes a w * h * 1 feature map, which is the salient region feature map.

[0122] 2024. Normalize the image feature values ​​at each position on the third feature map.

[0123] The third feature map has the same size as the original image, and is referred to as M in this paper.

[0124] It should be noted that the motivation for designing the saliency region decoder is that, for neural networks, certain regions in the input image are very important, while other regions are relatively unimportant. Therefore, this paper uses a second feature decoder to decode the input features (the first feature map) to obtain a feature map M, which is called the saliency region feature map. Then, the image feature values ​​at each location on this feature map are normalized to the range [0,1].

[0125] 203. Generate a noisy image based on the second and third feature maps.

[0126] In some embodiments, a noisy image is generated based on the second feature map and the third feature map, including but not limited to: performing positional multiplication on the second feature map obtained after step 2022 and the third feature map obtained after step 2024 to obtain the noisy image.

[0127] It should be noted that the larger the image feature value at any position on the salient region feature map, the more important the image feature at that position is, and the greater the probability that the noise feature value at the corresponding position will be preserved. This allows the noise to be more concentrated in the important areas of the image, which can improve the success rate of the attack.

[0128] 204. Overlay the original image with the noisy image to obtain the first adversarial sample.

[0129] In some embodiments, see Figure 3 and Figure 4 By overlaying the original image I with the noisy image P positionally, an adversarial sample of the original image I is obtained. This adversarial sample is referred to as the first adversarial sample in this paper, denoted by the symbol... Refers to.

[0130] It should be noted that the original image and the first adversarial example are visually identical. That is, after adding perturbations that are imperceptible to the human eye to the original image to obtain the first adversarial example, the original image and the first adversarial example appear identical to the human eye, and the subtle differences between them cannot be distinguished. However, the original image and the first adversarial example are physically inconsistent. That is, compared to the original image, the first adversarial example includes not only all the image information of the original image but also noise that is difficult for the human eye to perceive; in other words, the first adversarial example includes all the image information of the original image plus noise information that is difficult for the human eye to perceive.

[0131] Further, see Figure 3 and Figure 4 The adversarial attack network also includes an image recognition model 304. After obtaining the first adversarial example, see [link to relevant documentation]. Figure 7 The method provided in this application embodiment also includes the following step 205.

[0132] 205. Input the first adversarial sample into the image recognition model to obtain the image recognition result output by the image recognition model.

[0133] Optionally, after obtaining the first adversarial sample Then, the first adversarial sample can be... The input is used to attack the image recognition model.

[0134] The image processing scheme provided in this application requires only one forward operation to generate adversarial examples. Specifically, after extracting features from the original image to obtain a first feature map, a second feature map and a third feature map of the original image are obtained based on the first feature map. The second feature map represents an image perturbation that is difficult for the human eye to perceive and is to be superimposed onto the original image. The third feature map has different feature values ​​at each position, each feature value representing the importance of the image feature at that position. Then, a noisy image is generated based on the second and third feature maps. The adversarial example is obtained by superimposing the original image and the noisy image. Because this image processing method can quickly generate adversarial examples, it has good timeliness. Furthermore, the generated perturbation is stable, and the presence of the third feature map allows the noise to be concentrated in important areas, resulting in a higher quality adversarial example and thus effectively improving the attack effect.

[0135] In summary, the embodiments of this application can achieve good attack results against adversarial attacks. In terms of application, the embodiments of this application can effectively improve the resistance of image recognition models to adversarial attacks; that is, this image processing scheme can be used as a data augmentation method to optimize existing image recognition models, thereby improving the classification accuracy of existing image recognition models.

[0136] In other embodiments, during the training phase, see [link to documentation]. Figure 8 The training process of the aforementioned adversarial attack network includes, but is not limited to, the following steps.

[0137] 801. Obtain the second adversarial examples of the sample images included in the training dataset.

[0138] In this paper, adversarial examples of sample images are collectively referred to as second adversarial examples. In addition, the training dataset includes multiple sample images, and each sample image corresponds to one adversarial example, that is, the number of second adversarial examples is also multiple.

[0139] Optionally, similar to the image processing steps 201 to 204 above, for any sample image, a second adversarial sample of that sample image is obtained, including but not limited to the following steps:

[0140] 8011. The sample image is processed by feature encoder 301 of the adversarial attack network to obtain the first feature map of the sample image. For detailed implementation, please refer to step 201 above.

[0141] 8012. Input the first feature map of the sample image into the first feature decoder 302 and the second feature decoder 303 of the adversarial attack network, respectively.

[0142] 8013. The first feature decoder 302 performs first feature decoding processing on the first feature map of the sample image to obtain the original noise feature map of the sample image; the noise feature values ​​at each position on the original noise feature map of the sample image are suppressed to obtain the second feature map of the sample image.

[0143] 8014. The second feature decoder 303 performs second feature decoding processing on the first feature map of the sample image to obtain the third feature map of the sample image, and normalizes the image feature values ​​at each position on the third feature map of the sample image.

[0144] For detailed implementation of steps 8012 to 8014, please refer to step 202 above.

[0145] 8015. Based on the second and third feature maps of the sample image, generate a noisy image of the sample image; superimpose the sample image and the noisy image of the sample image to obtain the second adversarial sample of the sample image.

[0146] For a detailed explanation of the implementation of step 8015, please refer to steps 203 and 204 above.

[0147] 802. Input the sample image and the second adversarial sample into the image recognition model for feature encoding processing to obtain the feature data of the sample image and the feature data of the second adversarial sample.

[0148] See Figure 9 During the training phase, this step involves inputting the initial image and the corresponding adversarial prototype into the image recognition model to be attacked for feature extraction, thereby obtaining feature data.

[0149] 803. Based on the feature data of the sample image and the feature data of the second adversarial example, construct a first loss function and a second loss function respectively; and based on the third feature map of the sample image, construct a third loss function.

[0150] For neural networks, the feature angle is the main factor affecting image classification results, while the feature magnitude is the main factor affecting the degree of image variation. For this, see... Figure 9 This paper optimizes the loss function based on the angle magnitude. That is, in this embodiment, the feature angle and feature magnitude are considered separately, and two loss functions are designed, namely... and .like Figure 9 As shown, this is for the modulus space (high-dimensional space is simulated as a sphere). The aim is to bring the feature magnitudes of the initial image and the corresponding adversarial examples closer together. For example, this loss function is used to try to make the feature magnitudes of the adversarial examples as close as possible to the feature magnitudes of the initial image. This applies to the angular space (a high-dimensional space simulated as a sphere). Attempting to find the angle between the features of the initial image and the features of the corresponding adversarial examples Increase its size. This allows you to change the image classification result as much as possible without altering the appearance of the initial input image.

[0151] Accordingly, based on the feature data of the sample images and the feature data of the second adversarial examples, a first loss function and a second loss function are constructed respectively, including but not limited to the following steps:

[0152] 8031. In the feature data of the sample image, the feature angle of the sample image is separated; and in the feature data of the second adversarial example, the feature angle of the second adversarial example is separated.

[0153] 8032. Based on the feature angles of the sample image and the feature angles of the second adversarial sample, a first loss function is constructed, wherein the optimization objective of the first loss function is to increase the feature angle between the sample image and the second adversarial sample.

[0154] 8033. Based on the feature magnitude of the sample image and the feature magnitude of the second adversarial sample, construct a second loss function, wherein the optimization objective of the second loss function is to reduce the difference in feature magnitude between the sample image and the second adversarial sample.

[0155] Optionally, the first loss function and the second loss function are defined as follows:

[0156]

[0157]

[0158] Where i and j are both positive integers, and j refers to the number of sample images included in the training dataset; Refers to the network parameters of an image recognition model; Refers to the i-th sample image in the training dataset. Reference Noisy images; Reference Adversarial examples; This is a hyperparameter.

[0159] Optionally, the third loss function is defined as follows:

[0160]

[0161] in, Reference The salient region feature map; tr refers to the trace of the matrix; The purpose of this is to make the salient regions more concentrated; T refers to the rank of the matrix.

[0162] It should be noted that the trace of a matrix is ​​defined as follows: the sum of all elements on the main diagonal (from the upper left to the lower right) of an n×n matrix A is called the trace of matrix A, denoted as tr(A).

[0163] 804. End-to-end training is performed based on the first loss function, the second loss function, and the third loss function to obtain the adversarial attack network.

[0164] Optionally, end-to-end training is performed based on the first loss function, the second loss function, and the third loss function to obtain the adversarial attack network, including but not limited to: obtaining the first sum of the second loss function and the third loss function; and obtaining the product of the target constant and the first sum; using the second sum of the first sum and the product as the final loss function for end-to-end training to obtain the adversarial attack network.

[0165] The above text, expressed using a formula, yields the following formula:

[0166] , Refers to the target constant.

[0167] It should be noted that by performing end-to-end training according to the defined loss function, an autoencoder for adversarial attacks can be obtained. This autoencoder can then be used to generate adversarial examples of the original input image, which can then be used to attack the image recognition model.

[0168] During the training process of the adversarial attack network, the embodiments of this application optimize the loss function based on angle mode separation, which can change the image classification result as much as possible without changing the appearance of the original image or the initial image. That is, the generated adversarial samples are of higher quality, not only more consistent with the appearance of the original image or the initial image, but also achieve good attack effect, and the image recognition model that is not easily attacked can correctly classify the image.

[0169] The following describes the application scenarios of the image processing scheme provided in the embodiments of this application.

[0170] Adversarial examples generated based on autoencoders can enhance the resistance of image recognition models to adversarial attacks. Therefore, the image processing scheme provided in this application can be used as a data augmentation method to optimize existing image recognition models, thereby improving their classification accuracy. For example, this image processing scheme has achieved effective attack results in various recognition tasks, and even in black-box attacks.

[0171] Example 1: In the field of facial recognition, the image processing scheme provided in this application can serve as a data augmentation method to optimize existing facial recognition models, thereby improving the classification accuracy of existing facial recognition models. This is of great significance in scenarios such as security checks, identity verification, or mobile payments.

[0172] Example 2: In the field of item recognition, the image processing scheme provided in this application can serve as a data augmentation method to optimize existing item recognition models, thereby improving the classification accuracy of existing item recognition models. Optionally, this is of great significance in the process of item circulation, especially in unmanned retail fields such as unmanned shelves and smart retail cabinets.

[0173] In addition, the image processing scheme provided in this application embodiment can also attack some existing online image recognition tasks, thereby verifying the anti-attack capability of existing online image recognition tasks.

[0174] It should be noted that the application scenarios described above are for illustrative purposes only and are not intended to limit the scope of this application. In actual implementation, the technical solutions provided in the embodiments of this application can be flexibly applied according to actual needs.

[0175] The following is through Figures 10 to 14 The attack effects of the image processing scheme provided in the embodiments of this application are explained.

[0176] See Figure 10 ,in Figure 10 The left image in the image is an example image. Figure 10 The right image in the image above shows the image recognition result obtained by attacking an online image recognition service. For example... Figure 10 As shown, for the original image, the probability of it being identified as "food" by the online image recognition service is as high as 85%; after generating an adversarial sample of the original image based on the image processing method provided in this application embodiment, the probability of the adversarial sample being identified as "food" by the online image recognition service drops sharply to 25%.

[0177] See Figure 11 ,in Figure 11 The left image in the image is an example image. Figure 11 The right image in the image above shows the image recognition result obtained by attacking an online image recognition service. For example... Figure 11 As shown, for the original image, the probability of it being identified as a "Venetian gondola" by the online image recognition service is as high as 98%. After generating an adversarial sample of the original image based on the image processing method provided in this application embodiment, the probability of the adversarial sample being identified as a "Venetian gondola" by the online image recognition service drops sharply to 14%. Conversely, the probability of it being identified as a "puzzle" increases from 0% to 84%.

[0178] See Figure 12 ,in Figure 12 The left image in the image is an example image. Figure 12 The right image in the image above shows the image recognition result obtained by attacking an online image recognition service. For example... Figure 12 As shown, for the original image, the probability of it being identified as a "child" by the online image recognition service is as high as 90%. After generating an adversarial sample of the original image based on the image processing method provided in this application embodiment, the probability of the adversarial sample being identified as a "child" by the online image recognition service drops sharply to 14%. Conversely, the probability of it being identified as a "picture frame" increases from 13% to 52%.

[0179] See Figure 13 ,in Figure 13 The left column shows example images. Figure 13 The right column shows the image recognition results obtained by attacking a certain online image recognition service. For example... Figure 13 As shown, before the adversarial attack processing, all three images in the left column were identified as "masks," but after the adversarial attack processing, none of the three images in the left column were identified as "masks."

[0180] See Figure 14 ,in Figure 14 The left column shows example images. Figure 14 The right column shows the image recognition results obtained by attacking a certain online image recognition service. For example... Figure 14 As shown, before the adversarial attack processing, all three images in the left column were identified as "knapsacks," but after the adversarial attack processing, none of the three images in the left column were identified as "knapsacks."

[0181] In summary, combining Figures 10 to 14 As can be seen from the image recognition results, after the image processing scheme provided in this embodiment generates adversarial examples and attacks the online image recognition service, the accuracy of the online image recognition service in recognizing the generated adversarial examples decreases significantly, resulting in image classification errors, such as the inability to classify images correctly. Figure 13 The image shown is identified as a "face mask," and for example, it cannot be identified as a "face mask." Figure 14 The image recognition shown is a "knapsack" analogy, which intuitively illustrates the good attack effect of the image processing scheme provided in this application embodiment when subjected to adversarial attacks. Furthermore, in terms of application, the image processing scheme provided in this application embodiment can be used as a data augmentation method to optimize image recognition models or image recognition services, thereby improving the classification accuracy of existing image recognition models or image recognition services.

[0182] Figure 15 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. See also... Figure 15 The device includes:

[0183] Encoding module 1501 is configured to acquire an original image, perform feature encoding processing on the original image, and obtain a first feature map;

[0184] The decoding module 1502 is configured to obtain a second feature map and a third feature map of the original image based on the first feature map; wherein, the second feature map refers to the image perturbation to be superimposed on the original image, and each position on the third feature map has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position;

[0185] The first processing module 1503 is configured to generate a noisy image based on the second feature map and the third feature map;

[0186] The second processing module 1504 is configured to superimpose the original image with the noisy image to obtain a first adversarial sample.

[0187] The image processing scheme provided in this application requires only one forward operation to generate adversarial examples. Specifically, after extracting features from the original image to obtain a first feature map, a second feature map and a third feature map of the original image are obtained based on the first feature map. The second feature map represents an image perturbation that is difficult for the human eye to perceive and is to be superimposed onto the original image. The third feature map has different feature values ​​at each position, each feature value representing the importance of the image feature at that position. Then, a noisy image is generated based on the second and third feature maps. The adversarial example is obtained by superimposing the original image and the noisy image. Because this image processing method can quickly generate adversarial examples, it has good timeliness. Furthermore, the generated perturbation is stable, and the presence of the third feature map allows the noise to be concentrated in important areas, resulting in a higher quality adversarial example and thus effectively improving the attack effect.

[0188] In summary, the embodiments of this application can achieve good attack results against adversarial attacks. In terms of application, the embodiments of this application can effectively improve the resistance of image recognition models to adversarial attacks; that is, this image processing scheme can be used as a data augmentation method to optimize existing image recognition models, thereby improving the classification accuracy of existing image recognition models.

[0189] In some embodiments, the encoding module is configured to:

[0190] The original image is input into the feature encoder of the adversarial attack network for feature encoding processing to obtain the first feature map, the size of which is smaller than the original image.

[0191] The feature encoder includes convolutional layers and residual blocks, with the residual blocks following the convolutional layers in the connection order. Each residual block includes an identity mapping and at least two convolutional layers, and the identity mapping of each residual block points from the input of the residual block to the output of the residual block.

[0192] In some embodiments, the decoding module includes a first decoding unit, which is configured to:

[0193] The first feature map is input into the first feature decoder of the adversarial attack network for first feature decoding processing to obtain the original noise feature map.

[0194] The noise feature values ​​at each position on the original noise feature map are suppressed to obtain the second feature map, and the size of the second feature map is the same as the size of the original image.

[0195] The first feature decoder includes a deconvolutional layer and a convolutional layer, wherein the convolutional layer is located after the deconvolutional layer in the connection order.

[0196] In some embodiments, the decoding module includes a first decoding unit, which is configured to:

[0197] The noise feature values ​​at each location on the original noise feature map are compared with the target threshold.

[0198] For any position on the original noise feature map, in response to the noise feature value at any position being greater than the target threshold, the noise feature value at any position is replaced with the target threshold.

[0199] In some embodiments, the decoding module further includes a second decoding unit, which is configured to:

[0200] The first feature map is input into the second feature decoder of the adversarial attack network for second feature decoding processing to obtain the third feature map of the original image;

[0201] The image feature values ​​at each position on the third feature map are normalized, and the size of the third feature map is the same as the size of the original image;

[0202] The second feature decoder includes a deconvolutional layer and a convolutional layer, wherein the convolutional layer is located after the deconvolutional layer in the connection order.

[0203] In some embodiments, the first processing module is configured to perform position-wise multiplication of the second feature map and the third feature map to obtain the noise image.

[0204] In some embodiments, the adversarial attack network further includes an image recognition model; the apparatus further includes a classification module; the classification module is configured to input the first adversarial sample into the image recognition model to obtain an image recognition result output by the image recognition model.

[0205] In some embodiments, the training process of the adversarial attack network includes:

[0206] Obtain second adversarial examples of sample images included in the training dataset;

[0207] The sample image and the second adversarial sample are input together into the image recognition model for feature encoding processing to obtain the feature data of the sample image and the feature data of the second adversarial sample;

[0208] Based on the feature data of the sample image and the feature data of the second adversarial example, a first loss function and a second loss function are constructed respectively.

[0209] Obtain the third feature map of the sample image. Each position on the third feature map of the sample image has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position.

[0210] Based on the third feature map of the sample image, a third loss function is constructed;

[0211] The adversarial attack network is obtained by performing end-to-end training based on the first loss function, the second loss function, and the third loss function.

[0212] In some embodiments, the training process of the adversarial attack network includes:

[0213] The feature angles of the sample image are extracted from the feature data of the sample image.

[0214] In the feature data of the second adversarial sample, the feature angle of the second adversarial sample is separated;

[0215] Based on the feature angles of the sample image and the feature angles of the second adversarial example, a first loss function is constructed. The optimization objective of the first loss function is to increase the feature angle between the sample image and the second adversarial example.

[0216] In some embodiments, the training process of the adversarial attack network includes:

[0217] In the feature data of the sample image, the feature magnitude of the sample image is extracted;

[0218] In the feature data of the second adversarial example, the feature magnitude value of the second adversarial example is separated;

[0219] Based on the feature magnitude of the sample image and the feature magnitude of the second adversarial example, a second loss function is constructed. The optimization objective of the second loss function is to reduce the difference in feature magnitude between the sample image and the second adversarial example.

[0220] In some embodiments, the training process of the adversarial attack network includes:

[0221] Obtain the first sum of the second loss function and the third loss function; and obtain the product of the target constant and the first sum.

[0222] The second sum of the product of the first sum and the first sum is used as the final loss function for end-to-end training to obtain the adversarial attack network.

[0223] In some embodiments, the first feature decoder and the second feature decoder of the adversarial attack network have the same structure.

[0224] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0225] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0226] Figure 16 This illustration shows a structural block diagram of a computer device 1600 provided in an exemplary embodiment of this application. Taking a computer device as an example, the computer device 1600 typically includes a processor 1601 and a memory 1602.

[0227] Processor 1601 includes one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 1601 can be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1601 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1601 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1601 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0228] The memory 1602 may include one or more computer-readable storage media, which may be non-transitory. The memory 1602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1602 are used to store at least one program code, which is executed by the processor 1601 to implement the image processing method provided in the method embodiments of this application.

[0229] In some embodiments, the computer device 1600 may optionally include a peripheral device interface 1603 and at least one peripheral device. The processor 1601, memory 1602, and peripheral device interface 1603 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1603 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1604, a display screen 1605, a camera assembly 1606, an audio circuit 1607, and a power supply 1609.

[0230] Peripheral interface 1603 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1601 and memory 1602. In some embodiments, processor 1601, memory 1602 and peripheral interface 1603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1601, memory 1602 and peripheral interface 1603 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0231] The radio frequency (RF) circuit 1604 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1604 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1604 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1604 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1604 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0232] Display screen 1605 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1605 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1601 for processing. In this case, display screen 1605 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1605, disposed on the front panel of computer device 1600; in other embodiments, there may be at least two display screens, disposed on different surfaces of computer device 1600 or in a folded design; in still other embodiments, display screen 1605 may be a flexible display screen, disposed on a curved or folded surface of computer device 1600. Furthermore, display screen 1605 may be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1605 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0233] The camera assembly 1606 is used to acquire images or videos. Optionally, the camera assembly 1606 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1606 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0234] The audio circuit 1607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting them into electrical signals that are input to the processor 1601 for processing, or to the radio frequency circuit 1604 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each located at a different part of the computer device 1600. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1601 or the radio frequency circuit 1604 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1607 may also include a headphone jack.

[0235] Power supply 1609 is used to supply power to the various components in computer device 1600. Power supply 1609 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1609 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0236] In some embodiments, the computer device 1600 further includes one or more sensors 1610. The one or more sensors 1610 include, but are not limited to, an accelerometer 1611, a gyroscope 1612, a pressure sensor 1613, an optical sensor 1615, and a proximity sensor 1616.

[0237] Accelerometer 1611 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by computer device 1600. For example, accelerometer 1611 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1601 can control display screen 1605 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1611. Accelerometer 1611 can also be used for games or for acquiring user motion data.

[0238] The gyroscope sensor 1612 can detect the orientation and rotation angle of the computer device 1600. The gyroscope sensor 1612 can work in conjunction with the accelerometer sensor 1611 to acquire 3D motion data from the user on the computer device 1600. Based on the data acquired by the gyroscope sensor 1612, the processor 1601 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0239] Pressure sensor 1613 can be disposed on the side bezel of computer device 1600 and / or on the lower layer of display screen 1605. When pressure sensor 1613 is disposed on the side bezel of computer device 1600, it can detect the user's grip signal on computer device 1600, and processor 1601 can perform left / right hand recognition or quick operation based on the grip signal collected by pressure sensor 1613. When pressure sensor 1613 is disposed on the lower layer of display screen 1605, processor 1601 can control operable controls on the UI interface based on the user's pressure operation on display screen 1605. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0240] An optical sensor 1615 is used to collect ambient light intensity. In one embodiment, the processor 1601 can control the display brightness of the display screen 1605 based on the ambient light intensity collected by the optical sensor 1615. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1605 is increased; when the ambient light intensity is low, the display brightness of the display screen 1605 is decreased. In another embodiment, the processor 1601 can also dynamically adjust the shooting parameters of the camera assembly 1606 based on the ambient light intensity collected by the optical sensor 1615.

[0241] The proximity sensor 1616, also known as a distance sensor, is typically located on the front panel of the computer device 1600. The proximity sensor 1616 is used to detect the distance between the user and the front of the computer device 1600. In one embodiment, when the proximity sensor 1616 detects that the distance between the user and the front of the computer device 1600 is gradually decreasing, the processor 1601 controls the display screen 1605 to switch from a screen-on state to a screen-off state; when the proximity sensor 1616 detects that the distance between the user and the front of the computer device 1600 is gradually increasing, the processor 1601 controls the display screen 1605 to switch from a screen-off state to a screen-on state.

[0242] Those skilled in the art will understand that Figure 16 The structure shown does not constitute a limitation on the computer device 1600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0243] Figure 17This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Taking a server as an example, the server 1700 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1701 and one or more memories 1702. The memory 1702 stores at least one line of program code, which is loaded and executed by the processor 1701 to implement the image processing methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0244] In an exemplary embodiment, a computer-readable storage medium is also provided, such as a memory including program code that can be executed by a processor in a computer device to perform the image processing method described above. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0245] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the image processing method described above.

[0246] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0247] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The method includes: The original image is acquired, and feature encoding processing is performed on the original image to obtain a first feature map; The first feature map is input into the first feature decoder of the adversarial attack network for first feature decoding processing to obtain the original noise feature map; based on the original noise feature map, the second feature map of the original image is obtained; wherein, the second feature map refers to the image perturbation to be superimposed on the original image; the first feature decoder includes a deconvolution layer and a convolution layer; for the first feature decoder, the convolution layer is located after the deconvolution layer in the connection order; The first feature map is input into the second feature decoder of the adversarial attack network for second feature decoding processing to obtain the third feature map of the original image; the image feature values ​​at each position on the third feature map are normalized, the size of the third feature map is the same as the size of the original image, and each position on the third feature map has a different feature value, each feature value is used to characterize the importance of the image feature at the corresponding position; wherein, the second feature decoder includes a deconvolution layer and a convolution layer; for the second feature decoder, the convolution layer is located after the deconvolution layer in the connection order; A noisy image is generated based on the second feature map and the third feature map; The original image is superimposed on the noisy image to obtain the first adversarial sample.

2. The method according to claim 1, characterized in that, The step of performing feature encoding processing on the original image to obtain a first feature map includes: The original image is input into the feature encoder of the adversarial attack network for feature encoding processing to obtain the first feature map, the size of which is smaller than the original image. The feature encoder includes convolutional layers and residual blocks, with the residual blocks following the convolutional layers in the connection order. Each residual block includes an identity mapping and at least two convolutional layers, and the identity mapping of each residual block points from the input of the residual block to the output of the residual block.

3. The method according to claim 1, characterized in that, The step of obtaining the second feature map of the original image based on the original noise feature map includes: The noise feature values ​​at each position on the original noise feature map are suppressed to obtain the second feature map, and the size of the second feature map is the same as the size of the original image.

4. The method according to claim 3, characterized in that, The noise feature value suppression process at each position on the original noise feature map includes: The noise feature values ​​at each location on the original noise feature map are compared with the target threshold. For any position on the original noise feature map, in response to the noise feature value at any position being greater than the target threshold, the noise feature value at any position is replaced with the target threshold.

5. The method according to claim 1, characterized in that, The step of generating a noisy image based on the second feature map and the third feature map includes: The second feature map and the third feature map are multiplied by position to obtain the noise image.

6. The method according to any one of claims 1 to 5, characterized in that, The adversarial attack network also includes an image recognition model; the method further includes: The first adversarial sample is input into the image recognition model to obtain the image recognition result output by the image recognition model.

7. The method according to claim 6, characterized in that, The training process of the adversarial attack network includes: Obtain second adversarial examples of sample images included in the training dataset; The sample image and the second adversarial sample are input together into the image recognition model for feature encoding processing to obtain the feature data of the sample image and the feature data of the second adversarial sample; Based on the feature data of the sample image and the feature data of the second adversarial example, a first loss function and a second loss function are constructed respectively. Obtain the third feature map of the sample image. Each position on the third feature map of the sample image has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position. Based on the third feature map of the sample image, a third loss function is constructed; The adversarial attack network is obtained by performing end-to-end training based on the first loss function, the second loss function, and the third loss function.

8. The method according to claim 7, characterized in that, The first loss function and the second loss function are constructed based on the feature data of the sample image and the feature data of the second adversarial example, respectively, including: The feature angles of the sample image are extracted from the feature data of the sample image. In the feature data of the second adversarial sample, the feature angle of the second adversarial sample is separated; Based on the feature angles of the sample image and the feature angles of the second adversarial example, a first loss function is constructed. The optimization objective of the first loss function is to increase the feature angle between the sample image and the second adversarial example.

9. The method according to claim 7, characterized in that, The first loss function and the second loss function are constructed based on the feature data of the sample image and the feature data of the second adversarial example, respectively, including: In the feature data of the sample image, the feature magnitude of the sample image is extracted; In the feature data of the second adversarial example, the feature magnitude value of the second adversarial example is separated; Based on the feature magnitude of the sample image and the feature magnitude of the second adversarial example, a second loss function is constructed. The optimization objective of the second loss function is to reduce the difference in feature magnitude between the sample image and the second adversarial example.

10. The method according to claim 7, characterized in that, The process of performing end-to-end training based on the first loss function, the second loss function, and the third loss function to obtain the adversarial attack network includes: Obtain the first sum of the second loss function and the third loss function; and obtain the product of the target constant and the first sum. The second sum of the product of the first sum and the first sum is used as the final loss function for end-to-end training to obtain the adversarial attack network.

11. The method according to claim 6, characterized in that, The first and second feature decoders of the adversarial attack network have the same structure.

12. An image processing apparatus, characterized in that, The device includes: The encoding module is configured to acquire the original image, perform feature encoding processing on the original image, and obtain a first feature map; The decoding module is configured to obtain a second feature map and a third feature map of the original image based on the first feature map; wherein the second feature map refers to the image perturbation to be superimposed on the original image, and each position on the third feature map has a different feature value, and each feature value is used to characterize the importance of the image feature at the corresponding position; The first processing module is configured to generate a noisy image based on the second feature map and the third feature map; The second processing module is configured to overlay the original image with the noisy image to obtain a first adversarial sample. The decoding module includes a first decoding unit and a second decoding unit; The first decoding unit is configured to input the first feature map into the first feature decoder of the adversarial attack network for first feature decoding processing to obtain an original noise feature map; and to obtain a second feature map of the original image based on the original noise feature map; wherein the first feature decoder includes a deconvolution layer and a convolution layer; and for the first feature decoder, the convolution layer is located after the deconvolution layer in the connection order. The second decoding unit is configured to input the first feature map into the second feature decoder of the adversarial attack network for second feature decoding processing to obtain a third feature map of the original image; normalize the image feature values ​​at each position on the third feature map, and the size of the third feature map is consistent with the size of the original image; wherein the second feature decoder includes a deconvolution layer and a convolution layer; for the second feature decoder, the convolution layer is located after the deconvolution layer in the connection order.

13. A computer device, characterized in that, The device includes a processor and a memory, the memory storing at least one line of program code, the at least one line of program code being loaded and executed by the processor to implement the image processing method as claimed in any one of claims 1 to 11.

14. A computer-readable storage medium, characterized in that, The storage medium stores at least one piece of program code, which is loaded and executed by a processor to implement the image processing method as described in any one of claims 1 to 11.

Images