A method for training image reconstruction model based on adversarial generative network

Abstract

The embodiment of this specification discloses a method for training an image reconstruction model based on a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The method includes: obtaining a plurality of first data pairs, each of which includes an original image and desensitized data corresponding thereto, the desensitized data being the result of desensitizing the original image; alternately training the generator and the discriminator based on the plurality of first data pairs, and using the trained generator as an image reconstruction model.

Description

Technical Field

[0001] The present invention relates to the field of information technology, and in particular to a method for training an image reconstruction model based on a generative adversarial network. Background Art

[0002] Data desensitization can prevent the leakage of sensitive information in data (e.g., images) to a certain extent. However, attackers try to restore the original image containing sensitive information from desensitized data by various means.

[0003] In view of this, it is desirable to provide a method that can be used to test the defense capabilities of image desensitization algorithms. Summary of the invention

[0004] One of the embodiments of this specification provides a method for training an image reconstruction model based on a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The method may include: obtaining a plurality of first data pairs, each of which includes an original image and desensitized data corresponding thereto, the desensitized data being the result of desensitizing the original image; alternately training the generator and the discriminator based on the plurality of first data pairs, and using the trained generator as an image reconstruction model.

[0005] One of the embodiments of this specification provides a system for training an image reconstruction model based on a generative adversarial network. The generative adversarial network includes a generator and a discriminator. The system may include: a first acquisition module for obtaining a plurality of first data pairs, each first data pair including an original image and desensitized data corresponding thereto, the desensitized data being the result of image desensitization of the original image; a training module for alternately training the generator and the discriminator based on the plurality of first data pairs, and using the trained generator as an image reconstruction model.

[0006] In some embodiments, the training of the generator and / or the discriminator may include: obtaining generator input data based on the desensitized data in the first data pair; processing the generator input data by the generator to obtain a generated image output by the generator; processing the first data pair or the second data pair by the discriminator to obtain a discrimination result output by the discriminator, the discrimination result reflecting the probability that the image in the corresponding data pair is the original image corresponding to the desensitized data, wherein the second data pair includes the generated image and the corresponding desensitized data; adjusting the parameters of the generator and / or the discriminator based on the objective function.

[0007] One of the embodiments of the present specification provides an apparatus for training an image reconstruction model based on a generative adversarial network, comprising a processor and a storage device, wherein the storage device is used to store instructions, wherein when the processor executes the instructions, a method for training an image reconstruction model based on a generative adversarial network as described in any embodiment of the present specification is implemented.

[0008] One of the embodiments of the present specification provides an image reconstruction method, comprising: obtaining desensitized data, wherein the desensitized data is a result of desensitizing an original image; obtaining generator input data based on the desensitized data; processing the generator input data by a generator to obtain a generated image output by the generator, and using the generated image as a reconstruction result of the original image.

[0009] One of the embodiments of the present specification provides an image reconstruction system, including: a second acquisition module, used to obtain desensitized data, wherein the desensitized data is a result of desensitizing an original image; a third acquisition module, used to obtain generator input data based on the desensitized data; an image reconstruction module, used to process the generator input data through a generator, obtain a generated image output by the generator, and use it as a reconstruction result of the original image.

[0010] One of the embodiments of the present specification provides an image reconstruction apparatus, which includes a processor and a storage device, wherein the storage device is used to store instructions, and when the processor executes the instructions, the image reconstruction method described in any embodiment of the present specification is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] This specification will be further described in the form of exemplary embodiments, which will be described in detail by the accompanying drawings. These embodiments are not restrictive, and in these embodiments, the same number represents the same structure, wherein:

[0012] Figure 1 It is a schematic diagram of a scene of training an image reconstruction model based on a generative adversarial network according to some embodiments of this specification;

[0013] Figure 2 is an exemplary flow chart of a method for training an image reconstruction model based on a generative adversarial network according to some embodiments of this specification;

[0014] Figure 3 is an exemplary data processing flow shown in some embodiments of this specification;

[0015] Figure 4 is a schematic diagram of encoding the desensitized data in the first data pair according to some embodiments of this specification;

[0016] Figure 5 is an exemplary flow chart of an image reconstruction method according to some embodiments of this specification;

[0017] Figure 6 is an exemplary module diagram of a system for training an image reconstruction model based on a generative adversarial network according to some embodiments of this specification;

[0018] Figure 7 is an exemplary module diagram of an image reconstruction system according to some embodiments of the present specification. DETAILED DESCRIPTION

[0019] In order to more clearly illustrate the technical solutions of the embodiments of this specification, the following is a brief introduction to the drawings required for the description of the embodiments. Obviously, the drawings described below are only some examples or embodiments of this specification. For ordinary technicians in this field, this specification can also be applied to other similar scenarios based on these drawings without creative work. Unless it is obvious from the language environment or otherwise explained, the same reference numerals in the figures represent the same structure or operation.

[0020] It should be understood that the "system", "device", "unit" and / or "module" used herein are a method for distinguishing different components, elements, parts, portions or assemblies at different levels. However, if other words can achieve the same purpose, the words can be replaced by other expressions.

[0021] As shown in this specification, unless the context clearly indicates an exception, the words "a", "an", "a kind" and / or "the" do not refer to the singular, but also include the plural. Generally speaking, the terms "include" and "comprise" only indicate the inclusion of the steps and elements that have been clearly identified, and these steps and elements do not constitute an exclusive list. The method or device may also include other steps or elements.

[0022] Flowcharts are used in this specification to illustrate the operations performed by the system according to the embodiments of this specification. It should be understood that the preceding or following operations are not necessarily performed precisely in order. Instead, the steps may be processed in reverse order or simultaneously. At the same time, other operations may also be added to these processes, or one or more operations may be removed from these processes.

[0023] With the development of computer technology, image technology has gradually penetrated into various fields of real life. For example, face recognition technology has begun to be widely used in many scenarios, such as device unlocking, terminal application login, payment, authorization, etc. based on face recognition technology.

[0024] Some images may contain sensitive information, and data desensitization can prevent the leakage of sensitive information in the image to a certain extent. As an example only, by performing image transformation or feature extraction on the original face image or adding noise to remove or conceal sensitive information in the image, desensitized data is obtained, thereby reducing the possibility of the original face image or the vector representation obtained by feature extraction of the original face image being maliciously stolen (for example, using stolen face data for payment), thereby protecting the privacy of the face. It can be understood that sensitive information may refer to all information that is detrimental to the interests of the entity or to the personal privacy rights enjoyed by an individual according to law if improperly used or unauthorized access or modification is made. For example, sensitive information may include one or more of personal privacy information, internal confidential information, harmful information, etc.

[0025] However, attackers try to restore the original image containing sensitive information from the desensitized data through various means. Taking black box attacks as an example, the algorithm model of data desensitization (i.e., the target model) is unknown to the attacker, but the attacker can obtain multiple data pairs, each of which includes the original image and the desensitized data corresponding to it. Based on these data pairs, an image reconstruction model can be trained, and the image reconstruction model can be used to reconstruct (i.e., restore / recover) the original image based on the desensitized data. Taking facial images as an example, criminals can use the trained image reconstruction model to restore the original facial image from the desensitized data, which will cause the leakage of personal privacy or even more serious consequences. For example, criminals can use the restored facial image for facial recognition to obtain some sensitive permissions, such as face payment, device unlocking, etc.

[0026] In view of this, the embodiments of this specification provide a method and system for training an image reconstruction model based on a generative adversarial network (GAN). The trained image reconstruction model can be used to test the ability of an image desensitization algorithm to resist black box attacks or can be used to optimize the image desensitization algorithm so that the desensitized data generated by it is difficult to attack.

[0027] Figure 1 It is a schematic diagram of a scenario of training an image reconstruction model based on a generative adversarial network according to some embodiments of this specification.

[0028] like Figure 1 As shown, the scene 100 may include a generative adversarial network 110 and a target model 120. The generative adversarial network 110 may include a generator 112 and a discriminator 114, and the trained generator 112 may be used as an image reconstruction model.

[0029] The target model 120 can provide data services, such as data desensitization services. The service provider can hide the details of the target model and its training algorithm from the user, and only provide the user with an API (Application Programming Interface) to access the target model, that is, to treat the target model as a black box. The attacker can transmit the original image to the target model through the API as a user, and obtain the desensitized data obtained by the target model processing the original image. In this way, the attacker can obtain a data pair (such as a first data pair) for training the generator 112 and the discriminator 114, which includes the original image and the desensitized data corresponding thereto. It can be understood that the attacker can obtain the original image or even directly obtain the data pair through various means to obtain enough data pairs for training the generator 112 and the discriminator 114. For example, the attacker can obtain the original image from public data. For example, the attacker can directly purchase data pairs from some users.

[0030] It is worth noting that the method for training an image reconstruction model based on a generative adversarial network provided in the embodiments of this specification is intended to test the ability of a data desensitization algorithm to resist black box attacks by simulating potential attack methods. Therefore, the method for obtaining data pairs for training the generator 112 and the discriminator 114 is flexible. For example, data pairs can be directly obtained from the training data set of the target model 120.

[0031] The generator 112 may be used to output generated data (such as a generated image) based on the obtained input (such as desensitized data). The discriminator 114 may be used to determine whether the obtained input, such as the image in the first data pair or the second data pair, is the original image corresponding to the desensitized data therein, wherein the first data pair may include the desensitized data and the original image corresponding thereto, and the second data pair may include the desensitized data and the generated image corresponding thereto. It can be understood that the generated data refers to the output of the generator 112, and the opposite of the generated data is the real data (such as the original image). When the discriminator 114 determines that the obtained data pair does not contain the real data (such as the original image), it can be considered that the obtained data pair contains the generated data.

[0032] When the generator 112 and the discriminator 114 are trained to a certain degree, the discriminator 114 has a certain ability to "distinguish true from false", that is, the ability to distinguish whether the image corresponding to the desensitized data in the data pair is a real image (i.e., the original image, corresponding to "true") or a generated image (corresponding to "false"). At the same time, the generator 112 has a certain ability to "make the fake look real", that is, it makes it difficult for the discriminator 114 to distinguish whether the output of the generator 112 is a real image (i.e., the original image, corresponding to "true") or a generated image (corresponding to "false"). Therefore, when the generator 112 and the discriminator 114 are trained to a certain degree, the trained generator 112 can be used as an image reconstruction model.

[0033] In some embodiments, for facial images, the generator 112 may come from StyleGAN or StyleGAN2, that is, the generator 112 may use the generator of StyleGAN or the generator of StyleGAN2. The hidden layer space of the generator of StyleGAN or StyleGAN2 has the characteristic of separable control over the features of the generated image, that is, each element of the hidden layer vector can control different facial features (for example, element s1 controls eyebrows, element s2 controls mouth, element s3 controls nose, etc.), thereby generating images with rich texture details.

[0034] In some embodiments, the discriminator 114 may be from Patch-GAN, that is, the discriminator 114 may adopt the discriminator of Patch-GAN. In some embodiments, the discriminator 114 may be implemented based on a multi-layer perceptron.

[0035] For more details on the training of the generator 112 and the discriminator 114, please refer to Figure 2 and its related description.

[0036] Figure 2 It is an exemplary flow chart of a method for training an image reconstruction model based on a generative adversarial network according to some embodiments of this specification. It can be understood that process 200 corresponds to the model training stage.

[0037] In some embodiments, process 200 may be performed by Figure 6 Specifically, the process 200 may be stored in a storage device (such as an internal storage device or an external storage device of a processing device implementing the system 600) in the form of instructions, and when the instructions are executed, the process 200 may be implemented. The process 200 may include the following steps.

[0038] Step 210 , obtaining a plurality of first data pairs, each of which includes an original image and desensitized data corresponding thereto. In some embodiments, step 210 may be performed by a first obtaining module 610 .

[0039] Among them, the desensitized data refers to the result of desensitizing the original image. In some application scenarios, the desensitized data can be represented as a matrix or a vector.

[0040] In some embodiments, the desensitized data corresponding to the multiple original images can be obtained by accessing the target model. The input of the target model can be the original image, and the output of the target model can be the desensitized data. In some embodiments, the first data pair can come from a training data set of the target model.

[0041] Step 220 , alternately train the generator and the discriminator based on the plurality of first data pairs, and use the trained generator as at least a part of the image reconstruction model. In some embodiments, step 220 may be performed by the training module 620 .

[0042] refer to Figure 1 , through the generator 112, a generated image can be obtained based on the desensitized data in the first data pair. Further, a second data pair can be obtained based on the generated image and the desensitized data corresponding thereto. Both the first data pair and the second data pair can be used to train the discriminator 114.

[0043] It can be understood that alternating training means: when training the generator, stop training the discriminator; when training the discriminator, stop training the generator. As an example only, the training can be alternating in the order of discriminator → generator → discriminator → generator... until a generator and discriminator that meet the requirements are trained.

[0044] For details about the input-output process during training, please refer to Figure 3 and its related description.

[0045] like Figure 3 As shown, for any first data pair among the plurality of first data pairs, the processing flow 300 may include the following steps.

[0046] Step 310, obtaining generator input data based on the desensitized data in the first data pair.

[0047] In some embodiments, the training module 620 may use the masked data in the first data pair as generator input data.

[0048] In some embodiments, the training module 620 can obtain the generator input data based on the desensitized data and the noise. In some embodiments, the noise can be represented as a vector, the elements of which obey the Gaussian distribution. By adding noise, the generator output data (i.e., the generated image) can be made more diverse, for example, more generated data corresponding to the desensitized data is generated, which helps to improve the model effect. In some embodiments, the generator can have multiple inputs, and accordingly, the desensitized data and the noise can be used as different inputs. For example, the input of the generator of the conditional generative adversarial network includes conditional information and noise, wherein the conditional information can be obtained based on the desensitized data, the conditional information can guide the generation of the image, and the generation performance of the generator can be improved, making it easier to generate a generated image close to the original image. In some embodiments, the desensitized data and the noise can also be processed (integrated) into a single input and fed into the generator.

[0049] In some embodiments, Figure 4As shown, the training module 620 can encode the desensitized data in the first data pair through an encoder to obtain generator input data. For example, the training module 620 can use the encoded desensitized data as generator input data. In some embodiments, the desensitized data can be encoded by an encoder of U-Net.

[0050] In some embodiments, the training module 620 may obtain generator input data based on the encoded desensitized data and noise.

[0051] In one example, the training module 620 may concatenate the encoded desensitized data and the noise together to obtain generator input data.

[0052] In another example, the training module 620 can process the encoded desensitized data and noise respectively through a feedforward network to obtain a desensitized data vector and a noise vector. For the desensitized data vector, the training module 620 can calculate the mean and variance of each element in the desensitized data vector. Furthermore, the training module 620 adjusts the value of each element in the noise data vector based on the mean and variance of each element in the desensitized data vector to obtain the generator input data. Specifically, for each element y of the noise vector before adjustment, each element y′ of the adjusted noise vector can be calculated according to y′=(y-μ) / σ, where μ represents the mean and σ represents the variance. In some embodiments, the feedforward network may include one or more networks such as a fully connected layer and a multilayer perceptron.

[0053] In yet another example, the training module 620 may feed noise and encoded desensitized data into the generator as different inputs.

[0054] In some embodiments, the encoder and the feed-forward network are included with the generator as different parts of an image reconstruction network.

[0055] Step 320, the generator processes the generator input data through the generator to obtain a generated image output by the generator.

[0056] It can be understood that the generated image is the generator output data. Referring to the foregoing, in some embodiments, the generator can come from StyleGAN or StyleGAN2.

[0057] Step 330: Process the first data pair or the second data pair through a discriminator to obtain a discrimination result output by the discriminator.

[0058] Among them, the discrimination result output by the discriminator can reflect the probability that the image in the corresponding data pair (ie, the discriminator input data) is the original image corresponding to the desensitized data.

[0059] Referring to the foregoing, in some embodiments, the discriminator may come from Patch-GAN. The discriminator of Patch-GAN may divide a single image into multiple patches, and obtain the patch probability corresponding to each patch of the image, and obtain the probability value reflected by the discrimination result based on the patch probabilities corresponding to the multiple patches of the image. For example, the patch probabilities corresponding to all patches are averaged, and the obtained average value is used as the probability value reflected by the discrimination result. It is worth noting that the divided patches should not be too small, because too small patches contain insufficient information, making it difficult for the discriminator to obtain a patch probability with reference significance. As an example only, the resolution of each patch may be 32*32.

[0060] In some embodiments, the channels of the desensitized data and its corresponding image can be merged, and the merged result can be sent to the discriminator as input. Channel merging or fusion refers to unifying the size of two data in the channel dimension, and then merging all channels of the two data to obtain one data. For example, for an RBG (corresponding to 3 channels) image with a resolution of 112*112, the desensitized data can be converted into data with one or more channels at 112*112, that is, the converted desensitized data can include one or more channels, and the resolution of each channel is also 112*112. If the number of elements contained in the desensitized data is less than 112*112, desensitized data with 1 channel can be obtained by expanding preset elements (such as 0), and then the desensitized data with 1 channel and the RGB image are merged to obtain data with 4 channels. If the desensitized data contains more than 112*112 elements, the elements contained in the desensitized data can be divided into at least two groups at 112*112, and each group is a channel. It is understandable that there may be a group whose number of elements is less than 112*112. In this case, a complete (ie, 112*112) channel can be obtained by expanding the preset elements (such as 0).

[0061] In some embodiments, the generator may be pre-trained. In the process of alternating training of the generator and the discriminator, the parameters of the generator may be fine-tuned, and the generator after parameter fine-tuning may be used as an image reconstruction model. It should be noted that fine-tuning is relative to the parameters of the pre-trained generator, and the purpose of fine-tuning may be achieved by setting a smaller learning rate. For example, the learning rate may be set to 0.001 or 0.0001. Fine-tuning may not change the order of magnitude of the generator parameters. For example, assuming that a parameter of the pre-trained generator is 100, the parameter may become 110, 101, 102, 101.1, 101.3, and so on after fine-tuning. Thus, while retaining the "knowledge" that the generator has learned, it is made more suitable for the current task.

[0062] In some embodiments, the generator and the discriminator can be alternately trained according to a preset objective function. Specifically, when training the generator, the generator parameters are adjusted to minimize the objective function when the discriminator parameters are fixed. When training the discriminator, the discriminator parameters are adjusted to maximize the objective function when the generator parameters are fixed. The objective function includes a first part, which is positively correlated with the discrimination result corresponding to the first data pair and negatively correlated with the discrimination result corresponding to the second data pair.

[0063] It can be understood that the training of the generator is to improve its ability to "make the fake look real", that is, to make it difficult for the discriminator 114 to distinguish whether the output of the generator 112 is a real image (i.e., the original image, corresponding to "real") or a generated image (corresponding to "false"). Therefore, when adjusting the parameters of the generator, it is hoped that the probability that the generated image in the first data pair reflected by the discrimination result is the original image is low, and the probability that the generated image in the second data pair reflected by the discrimination result is the original image is high, that is, it is hoped that the objective function is minimized. The training of the discriminator is to improve its ability to "distinguish between true and false", that is, to distinguish whether the image corresponding to the desensitized data in the data pair is a real image (i.e., the original image, corresponding to "real") or a generated image (corresponding to "false"). Therefore, when adjusting the parameters of the discriminator, it is hoped that the probability that the generated image in the first data pair reflected by the discrimination result is the original image is high, and the probability that the generated image in the second data pair reflected by the discrimination result is the original image is low, that is, it is hoped that the objective function is maximized.

[0064] In some embodiments, the objective function may further include a second part, which may reflect the difference between the original image and the generated image in the first data pair and the second data pair corresponding to the same desensitized data.

[0065] Just as an example, the parameters of the generator and discriminator can be adjusted according to the following expressions:

[0066] G * =arg minG maxD L cGAN (G, D)+λL L1 (G) (1)

[0067] Among them, G represents the generator, D represents the discriminator, and λ is the preset coefficient.

[0068] L cGAN (G, D)+λL L1 (G) represents the objective function. cGAN (G, D) is the first part of the objective function (also called the loss function), which may be positively correlated with the discrimination result corresponding to the first data pair and negatively correlated with the discrimination result corresponding to the second data pair. L1(G) is the second part of the objective function (also referred to as a constraint term), which can reflect the difference between the original image and the generated image in the first data pair and the second data pair corresponding to the same desensitized data.

[0069] It can be understood that arg represents the parameters of the generator G when the objective function is minimized (min) (the parameters of the discriminator D are fixed at this time), or the parameters of the discriminator D when the objective function is maximized (max) (the parameters of the generator G are fixed at this time).

[0070] In some embodiments, the loss function L cGAN (G, D) can be shown as follows:

[0071] L cGAN (G, D) = E x，y [logD(x,y)]+E x，z [log(1-D(x,G(x,z))] (2)

[0072] Where x represents the desensitized data, y represents the original image, and z represents the noise. G() represents the output of the generator (i.e., the generated image), and D() represents the output of the discriminator (i.e., the discriminant result). (x, y) represents the first data pair, and (x, G(x, z)) represents the second data pair. x，y It means to find the expectation on the set (x, y), E x，z It means to find the expectation on the set of (x, z), and log means logarithmic operation.

[0073] In some embodiments, the constraint L L1 (G) can be as follows:

[0074] L L1 (G) = E x，y，z [||yG(x,z)||1] (3)

[0075] Among them, E x，y，z represents the expectation on the set (x, y, z), ||yG(x, z)||1 represents the 1-norm of the difference between the original image y and the generated image G(x, z) corresponding to the same desensitized data x. The meaning of the variables in expression (3) can also refer to the relevant description of expression (2).

[0076] Figure 5 It is an exemplary flow chart of an image reconstruction method according to some embodiments of the present specification. It can be understood that process 500 corresponds to the model prediction stage.

[0077] In some embodiments, process 500 may be performed by Figure 7Specifically, the process 500 may be stored in a storage device (such as an internal storage device or an external storage device of a processing device implementing the system 700) in the form of instructions, and when the instructions are executed, the process 500 may be implemented. The process 500 may include the following steps.

[0078] Step 510 , obtaining desensitized data, wherein the desensitized data is the result of performing image desensitization on the original image. In some embodiments, step 510 may be performed by the second obtaining module 710 .

[0079] Step 520 , obtaining generator input data based on the desensitized data. In some embodiments, step 520 may be performed by a third obtaining module 720 .

[0080] Step 530 , the generator processes the generator input data to obtain a generated image output by the generator, and uses the generated image as the reconstruction result of the original image. Step 530 may be performed by the image reconstruction module 730 .

[0081] For more details about process 500, please refer to Figure 2 And its related instructions. For example, the generator can come from StyleGAN or StyleGAN2. For another example, the third acquisition module 720 can encode the desensitized data to obtain the generator input data. Further, the third acquisition module 720 can obtain the generator input data based on the encoded desensitized data and noise. For another example, the encoding can be implemented by a U-Net encoder.

[0082] It should be noted that the above description of the relevant process is only for example and explanation, and does not limit the scope of application of this specification. For those skilled in the art, various modifications and changes can be made to the process under the guidance of this specification. For example, the training module 620 can splice the desensitized data vector and the noise vector obtained by the feedforward network to obtain the generator input data. However, these modifications and changes are still within the scope of this specification.

[0083] Figure 6 It is an exemplary module diagram of a system for training an image reconstruction model based on a generative adversarial network according to some embodiments of the present specification.

[0084] like Figure 6 As shown, the system 600 may include a first obtaining module 610 and a training module 620 .

[0085] In some embodiments, the first obtaining module 610 can be used to obtain multiple first data pairs, each first data pair including an original image and desensitized data corresponding thereto.

[0086] In some embodiments, the training module 620 can be used to alternately train the generator and the discriminator based on multiple first data pairs, and use the trained generator as at least a part of the image reconstruction model.

[0087] For more details about the system 600 and its modules, please refer to Figure 2 and its related description.

[0088] Figure 7 is an exemplary module diagram of an image reconstruction system according to some embodiments of the present specification.

[0089] like Figure 7 As shown, the system 700 may include a second obtaining module 710 , a third obtaining module 720 , and an image reconstruction module 730 .

[0090] In some embodiments, the second obtaining module 710 may be used to obtain desensitized data, where the desensitized data is a result of desensitizing the original image.

[0091] In some embodiments, the third obtaining module 720 may be used to obtain generator input data based on the desensitized data.

[0092] In some embodiments, the image reconstruction module 730 can be used to process the generator input data through the generator, obtain the generated image output by the generator, and use it as the reconstruction result of the original image.

[0093] For more details about the system 700 and its modules, please refer to Figure 5 and its related description.

[0094] It should be understood that Figure 6 , Figure 7The system and its modules shown can be implemented in various ways. For example, in some embodiments, the system and its modules can be implemented by hardware, software or a combination of software and hardware. Among them, the hardware part can be implemented using dedicated logic; the software part can be stored in a memory and executed by an appropriate instruction execution system, such as a microprocessor or a dedicated design hardware. Those skilled in the art will understand that the above methods and systems can be implemented using computer executable instructions and / or included in a processor control code, such as a carrier medium such as a disk, CD or DVD-ROM, a programmable memory such as a read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. Such code is provided on the system and its modules of this specification. Not only can the hardware circuits such as ultra-large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field programmable gate arrays, programmable logic devices, etc. be implemented, it can also be implemented with software such as executed by various types of processors, and it can also be implemented by a combination of the above hardware circuits and software (for example, firmware).

[0095] It should be noted that the above description of the system and its modules is only for convenience of description and cannot limit this specification to the scope of the embodiments. It is understandable that for those skilled in the art, after understanding the principle of the system, it is possible to arbitrarily combine the modules or form a subsystem connected with other modules without deviating from this principle. For example, in some embodiments, the second acquisition module 710 and the third acquisition module 720 can be different modules in a system, or one module can realize the functions of these two modules. Such variations are all within the protection scope of this specification.

[0096] The beneficial effects that may be brought about by the embodiments of this specification include but are not limited to: (1) a method for learning the mapping from desensitized data to generated images based on a generative adversarial network is provided, and the trained generator can be used as an image reconstruction model, and the image reconstruction model can be used to test and / or optimize the ability of the image desensitization algorithm to resist black box attacks; (2) under the setting of black box attacks, the obtained image reconstruction model is suitable for testing and / or optimizing various image desensitization algorithms; (3) by encoding the desensitized data, the generator input data that can be received by the generator can be obtained; (4) by adding noise, the generator output data (i.e., the generated image) can be made more diverse, for example, more generated data corresponding to the desensitized data is generated, which helps to improve the model effect; (5) the hidden space of the generator of StyleGAN or StyleGAN2 has the characteristic of separable control over the features of the generated image, and can generate images with rich texture details. It should be noted that different embodiments may have different beneficial effects. In different embodiments, the beneficial effects that may be produced may be any one or a combination of the above, or any other beneficial effects that may be obtained.

[0097] The basic concepts have been described above. Obviously, for those skilled in the art, the above detailed disclosure is only for example and does not constitute a limitation on the embodiments of this specification. Although not explicitly stated here, those skilled in the art may make various modifications, improvements and corrections to the embodiments of this specification. Such modifications, improvements and corrections are suggested in the embodiments of this specification, so such modifications, improvements and corrections still belong to the spirit and scope of the exemplary embodiments of this specification.

[0098] At the same time, this specification uses specific words to describe the embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" refer to a certain feature, structure or characteristic related to at least one embodiment of this specification. Therefore, it should be emphasized and noted that "one embodiment" or "an embodiment" or "an alternative embodiment" mentioned twice or more in different locations in this specification does not necessarily refer to the same embodiment. In addition, certain features, structures or characteristics in one or more embodiments of this specification can be appropriately combined.

[0099] In addition, it will be appreciated by those skilled in the art that the various aspects of the embodiments of this specification may be illustrated and described by a number of patentable categories or situations, including any new and useful process, machine, product or combination of substances, or any new and useful improvements thereto. Accordingly, the various aspects of the embodiments of this specification may be performed entirely by hardware, entirely by software (including firmware, resident software, microcode, etc.), or by a combination of hardware and software. The above hardware or software may all be referred to as "data blocks", "modules", "engines", "units", "components" or "systems". In addition, the various aspects of the embodiments of this specification may be represented as a computer product located in one or more computer-readable media, the product including computer-readable program code.

[0100] A computer storage medium may include a propagated data signal containing computer program code, for example, in baseband or as part of a carrier wave. The propagated signal may be in a variety of forms, including electromagnetic, optical, etc., or a suitable combination. A computer storage medium may be any computer-readable medium other than a computer-readable storage medium, which can be connected to an instruction execution system, device or apparatus to communicate, propagate or transmit the program for use. The program code on the computer storage medium may be transmitted via any suitable medium, including radio, cable, fiber optic cable, RF, or similar media, or any combination of the above media.

[0101] The computer program coding required for the operation of each part of the embodiment of this specification can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python, etc., conventional procedural programming languages ​​such as C language, VisualBasic, Fortran2003, Perl, COBOL2002, PHP, ABAP, dynamic programming languages ​​such as Python, Ruby and Groovy, or other programming languages, etc. The program coding can be run completely on the user's computer, or run on the user's computer as an independent software package, or partly run on the user's computer and partly run on the remote computer, or run completely on the remote computer or processing equipment. In the latter case, the remote computer can be connected to the user's computer by any network form, such as a local area network (LAN) or a wide area network (WAN), or connected to an external computer (e.g., by the Internet), or in a cloud computing environment, or used as a service such as software as a service (SaaS).

[0102] In addition, unless explicitly stated in the claims, the order of the processing elements and sequences described in the embodiments of this specification, the use of alphanumeric characters, or the use of other names are not intended to limit the order of the processes and methods of the embodiments of this specification. Although the above disclosure discusses some invention embodiments that are currently considered useful through various examples, it should be understood that such details are only for illustrative purposes, and the attached claims are not limited to the disclosed embodiments. On the contrary, the claims are intended to cover all modifications and equivalent combinations that are consistent with the essence and scope of the embodiments of this specification. For example, although the system components described above can be implemented by hardware devices, they can also be implemented only by software solutions, such as installing the described system on an existing processing device or mobile device.

[0103] Similarly, it should be noted that in order to simplify the description of the embodiments disclosed in this specification, thereby helping to understand one or more embodiments of the invention, in the above description of the embodiments of this specification, multiple features are sometimes combined into one embodiment, figure or description thereof. However, this disclosure method does not mean that the features required by the objects of the embodiments of this specification are more than the features mentioned in the claims. In fact, the features of the embodiments are less than all the features of the single embodiment disclosed above.

[0104] Each patent, patent application, patent application publication, and other materials, such as articles, books, specifications, publications, documents, etc., cited in this specification is hereby incorporated by reference in its entirety. Except for application history documents that are inconsistent with or conflicting with the contents of this specification, documents that limit the broadest scope of the claims of this application (currently or later attached to this application) are also excluded. It should be noted that if the descriptions, definitions, and / or use of terms in the materials attached to this specification are inconsistent or conflicting with the contents described in this specification, the descriptions, definitions, and / or use of terms in this specification shall prevail.

[0105] Finally, it should be understood that the embodiments described in this specification are only used to illustrate the principles of the embodiments of this specification. Other variations may also fall within the scope of the embodiments of this specification. Therefore, as an example and not a limitation, alternative configurations of the embodiments of this specification may be considered consistent with the teachings of this specification. Accordingly, the embodiments of this specification are not limited to the embodiments explicitly introduced and described in this specification.

Claims

1. A method for training an image reconstruction model based on a generative adversarial network, wherein: The generative adversarial network includes a generator and a discriminator, and the method includes: Obtaining a plurality of first data pairs, each of which includes an original image and desensitized data corresponding thereto, wherein the desensitized data is a result of desensitizing the original image; The generator and the discriminator are alternately trained based on a plurality of first data pairs, and the trained generator is used as an image reconstruction model; the method comprises: Obtaining generator input data based on the desensitized data in the first data pair; Processing the generator input data by the generator to obtain a generated image output by the generator; Processing the first data pair or the second data pair through a discriminator to obtain a discrimination result output by the discriminator, the discrimination result reflecting the probability that the image in the corresponding data pair is the original image corresponding to the desensitized data; wherein the second data pair includes the generated image and the corresponding desensitized data; Parameters of the generator and / or the discriminator are adjusted based on an objective function.

2. The method of claim 1, wherein: The step of obtaining generator input data based on the desensitized data in the first data pair includes: The desensitized data in the first data pair is encoded to obtain generator input data.

3. The method of claim 2, wherein: The encoding of the desensitized data in the first data pair to obtain generator input data includes: obtaining the generator input data based on the encoded desensitized data and noise.

4. The method of claim 3, wherein: The step of obtaining generator input data based on the encoded desensitized data and noise includes: splicing the encoded desensitized data and the noise to obtain the generator input data; or, The encoded desensitized data and noise are processed separately through a feedforward network to obtain a desensitized data vector and a noise vector, and the noise vector and the desensitized data vector are concatenated to obtain the generator input data.

5. The method of claim 3, wherein: The step of obtaining generator input data based on the encoded desensitized data and noise includes: The encoded desensitized data and noise are processed separately through the feedforward network to obtain a desensitized data vector and a noise vector; Calculate the mean and variance of each element in the desensitized data vector; The value of each element in the noise vector is adjusted based on the mean and variance to obtain generator input data.

6. The method of claim 2, wherein: The encoding is implemented by the U-Net encoder.

7. The method of claim 1, wherein: The first part of the objective function is positively correlated with the discrimination result corresponding to the first data pair, and negatively correlated with the discrimination result corresponding to the second data pair; The adjusting the parameters of the generator and / or the discriminator based on the objective function comprises: When training the generator, the generator parameters are adjusted to minimize the objective function while the discriminator parameters are fixed; When training the discriminator, with the generator parameters fixed, the discriminator parameters are adjusted to maximize the objective function.

8. The method according to claim 1 or 7, wherein: The objective function also includes a second part, which reflects the difference between the original image and the generated image in the first data pair and the second data pair corresponding to the same desensitized data.

9. The method of claim 1, wherein: The generator comes from StyleGAN or StyleGAN2.

10. The method of claim 1, wherein: The discriminator comes from Patch-GAN; or, the discriminator is implemented based on a multi-layer perceptron.

11. A system for training an image reconstruction model based on a generative adversarial network, wherein: The generative adversarial network includes a generator and a discriminator, and the system includes: A first acquisition module is used to obtain a plurality of first data pairs, each of which includes an original image and desensitized data corresponding thereto, where the desensitized data is a result of desensitizing the original image; A training module is used to alternately train the generator and the discriminator based on a plurality of first data pairs, and use the trained generator as an image reconstruction model; it is specifically used to: Obtaining generator input data based on the desensitized data in the first data pair; Processing the generator input data by the generator to obtain a generated image output by the generator; Processing the first data pair or the second data pair through a discriminator to obtain a discrimination result output by the discriminator, the discrimination result reflecting the probability that the image in the corresponding data pair is the original image corresponding to the desensitized data; wherein the second data pair includes the generated image and the corresponding desensitized data; Parameters of the generator and / or the discriminator are adjusted based on an objective function.

12. A device for training an image reconstruction model based on a generative adversarial network, comprising a processor and a storage device, wherein the storage device is used to store instructions, wherein: When the processor executes the instructions, the method according to any one of claims 1 to 10 is implemented.

13. An image reconstruction method, comprising: Obtaining desensitized data, where the desensitized data is a result of desensitizing the original image; Obtaining generator input data based on the desensitized data; The generator input data is processed by a generator to obtain a generated image output by the generator, and the generated image is used as the reconstruction result of the original image; the generator is trained by the method described in any one of claims 1 to 10.

14. The method of claim 13, wherein: The generator comes from StyleGAN or StyleGAN2.

15. The method of claim 13, wherein: The obtaining generator input data based on the desensitized data includes: encoding the desensitized data to obtain generator input data.

16. The method of claim 15, wherein: The encoding of the desensitized data to obtain generator input data includes: obtaining generator input data based on the encoded desensitized data and noise.

17. The method of claim 15, wherein: The encoding is implemented by the U-Net encoder.

18. An image reconstruction system, comprising: A second acquisition module is used to obtain desensitized data, where the desensitized data is a result of desensitizing the original image; A third obtaining module, used to obtain generator input data based on the desensitized data; An image reconstruction module is used to process the generator input data through a generator to obtain a generated image output by the generator and use it as the reconstruction result of the original image; the generator is trained by the method described in any one of claims 1 to 10.

19. An image reconstruction device, wherein: The method comprises a processor and a storage device, wherein the storage device is used to store instructions. When the processor executes the instructions, the method according to any one of claims 13 to 17 is implemented.

Images