Training method of generative adversarial network, image generation method and electronic device

Abstract

Embodiments of the present application provide a training method of a generative adversarial network, an image generation method and an electronic device. In order to make an illegal network unable to accurately obtain a corresponding image, in the embodiments of the present application, an image generator is expected to generate a high-quality image, and at the same time, the related network is unable to obtain the image. Based on this purpose, in the embodiments of the present application, a high-quality image is expected to be generated, and at the same time, the image is expected to be mistaken by an illegal network as an image that the network does not expect to obtain, so as to achieve the purpose of protecting the image and protecting privacy. In the training phase of the image generator, in addition to using a discriminator to optimize the parameters of the image generator to ensure the image quality, a classifier is additionally used to optimize the neural network parameters of the image generator, so that the generated image can deceive the classifier and be divided into other categories by the classifier. Thus, the illegal network cannot accurately determine the category of the image, and the purpose of deceiving the illegal network is achieved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method for generative adversarial networks, an image generation method, and an electronic device. Background Technology

[0002] With the advent of the big data era, people's reliance on the internet has gradually increased. Once uploaded to the internet, users' private information is vulnerable to illegal collection, leading to privacy leaks. Images, as an important data resource on the internet, typically contain a large amount of private information. Currently, most privacy protection measures for user images prevent the illegal collection of private information by modifying sensitive areas or directly deleting privacy-related areas from the image. This method suffers from low computational efficiency and, to some extent, undermines the usability of the image. Therefore, a more comprehensive and efficient technical solution is lacking in the field to protect the privacy of images. Summary of the Invention

[0003] The purpose of this application is to provide a training method for generative adversarial networks, an image generation method, and an electronic device. This addresses the lack of a more comprehensive and efficient technical solution for protecting the privacy information of images in related technologies.

[0004] In a first aspect, embodiments of this application provide a method for training a generative adversarial network (GAN), wherein the GAN includes a generator, a discriminator, and a trained classifier, and the method includes:

[0005] A first random quantity is generated and input into the generator to obtain the first virtual image generated by the generator;

[0006] Keeping the parameters of the generator and the classifier unchanged, the parameters of the discriminator are updated based on the sample images in the first virtual image and the second training set;

[0007] A second random quantity is generated and input into the generator to obtain a second virtual image generated by the generator. The second virtual image is then input into the discriminator and the classifier respectively to obtain the discrimination result and classification result of the second virtual image.

[0008] The discrimination loss of the second virtual image is determined based on the discrimination result of the second virtual image, and the classification loss is determined based on the classification result and the expected classification result of the second virtual image; wherein the expected classification result is different from the actual classification result of the second virtual image;

[0009] Keeping the parameters of the discriminator and the classifier unchanged, the parameters of the generator are updated based on the discrimination loss and the classification loss of the second virtual image.

[0010] In some possible embodiments, updating the generator parameters based on the discriminative loss and the classification loss of the second virtual image includes:

[0011] The discrimination loss and the classification loss of the second virtual image are weighted and summed to obtain the total loss information;

[0012] The generator parameters are updated based on the total loss information.

[0013] In some possible embodiments, updating the parameters of the discriminator based on the first virtual image and sample images in the second training set includes:

[0014] The first virtual image and the sample image are respectively input into the discriminator to obtain the discrimination result of the first virtual image and the discrimination result of the sample image;

[0015] A first discrimination loss is determined based on the discrimination result of the first virtual image and the first expected discrimination result, and a second discrimination loss is determined based on the discrimination result of the sample image and the second expected discrimination result;

[0016] The parameters of the discriminator are updated based on the first discriminant loss and the second discriminant loss.

[0017] In some possible embodiments, after updating the generator parameters based on the discriminative loss and the classification loss of the second virtual image, the method further includes:

[0018] The generative adversarial neural network is pruned using a pruning algorithm, which includes weight pruning or channel pruning.

[0019] In some possible embodiments, after pruning the generative adversarial neural network using a pruning algorithm, the method further includes:

[0020] The process of regenerating a third random variable and adjusting the parameters of the pruned discriminator and generator based on the third random variable is iterated.

[0021] In some possible embodiments, the category corresponding to the expected classification result and the category corresponding to the actual classification result are categories with a similarity greater than a preset threshold.

[0022] Secondly, embodiments of this application provide an image generation method using a generative adversarial network (GAN), wherein the GAN includes a generator and a discriminator, and the method includes:

[0023] Acquire the target image from the target video stream;

[0024] Preset protection information is detected in the target image;

[0025] If the preset protection information is detected, an occlusion image is generated based on the generator;

[0026] The protection information is obscured by the obscuring image;

[0027] The generator is trained based on any of the methods in the first aspect.

[0028] In some possible embodiments, generating the occlusion image based on the generator includes:

[0029] A fourth random quantity is generated and input into the generator to obtain the candidate protected image generated by the generator;

[0030] The candidate protected image is input into the discriminator. If the discrimination result of the candidate protected image is true, then the candidate protected image is used as the occlusion image.

[0031] In some possible embodiments, if the discrimination result of the candidate protected image is false, the process returns to the step of generating a fourth random quantity and inputting it into the generator to obtain the candidate protected image generated by the generator, until the discrimination result of the obtained candidate protected image is true.

[0032] Thirdly, another embodiment of this application also provides an electronic device, including at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the methods of the first and second aspects provided in the embodiments of this application.

[0033] In this embodiment, during the training phase of the generative adversarial network (GAN), a first virtual image is obtained by inputting a first random variable into the generator, while keeping the parameters of the generator and the trained classifier unchanged. The discriminator is trained based on the first virtual image and sample images in the training set, enabling the trained discriminator to recognize the first virtual image and the sample images. A second random variable is generated and input into the generator to obtain a second virtual image. The second virtual image is then input into the discriminator and the classifier to obtain the discrimination result and classification result of the second virtual image. The discrimination loss of the second virtual image is determined based on the discrimination result, and correspondingly, the classification loss of the second virtual image is determined based on the classification result and the expected classification result. Keeping the parameters of the discriminator and the classifier unchanged, the generator is trained based on the discrimination loss and the classification loss. The GAN trained in this embodiment generates virtual images and uses these virtual images to occlude the privacy regions of user images, thereby addressing the lack of a more comprehensive and efficient technical solution for protecting image privacy information in related technologies.

[0034] Other features and advantages of this application will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description

[0035] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a diagram of an existing generative adversarial network architecture according to an embodiment of this application;

[0037] Figure 2 This is a diagram of an improved generative adversarial network architecture according to an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of an application environment according to an embodiment of this application;

[0039] Figure 4a This is a flowchart of a training method for a generative adversarial network according to an embodiment of this application;

[0040] Figure 4b This is a schematic diagram showing the pruning process before and after according to an embodiment of this application;

[0041] Figure 5 This is a flowchart of an image generation method using a generative adversarial network according to an embodiment of this application;

[0042] Figure 6 This is a schematic diagram of an electronic device provided according to an embodiment of this application. Detailed Implementation

[0043] The technical solutions in the embodiments of this application will be clearly and thoroughly described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in the text is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of this application, "multiple" means two or more.

[0044] In the description of the embodiments of this application, unless otherwise stated, the term "multiple" refers to two or more, and other quantifiers are similarly understood. The preferred embodiments described herein are only for illustration and explanation of this application and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0045] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, more or fewer operation steps may be included in the method based on conventional or non-inventive effort. For steps that do not logically have a necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application. In actual processing or when the control device executes the method, it may be executed sequentially or in parallel according to the method shown in the embodiments or drawings.

[0046] The inventors discovered that generative adversarial networks (GANs) are frequently used in related technologies to protect image privacy. Structurally inspired by game theory, GANs construct two opposing models—a generator and a discriminator—based on the principle that the sum of two elements equals zero, and the gain of one is the loss of the other. The generator learns the distribution of real data, while the discriminator determines whether the input data is real. The generator's goal is to generate virtual images that can deceive the discriminator, causing it to perceive the virtual image as real. Conversely, the discriminator's goal is to identify virtual images as not being real. During training, the goal is to achieve a Nash equilibrium between the generator and discriminator as much as possible. In GANs, Nash equilibrium means that the discriminator cannot accurately determine whether the virtual image generated by the generator is real, thus maintaining the discriminator's recognition ability at 50%. The structure of a GAN is as follows: Figure 1 As shown, it includes a generator and a discriminator. Applied to scenarios involving image privacy protection, random noise is added to the generator to make it produce a virtual image used to protect the user's image. The discriminator is trained by inputting the virtual image and a sample image representing the user's image into it, enabling the trained discriminator to distinguish between the virtual image and the sample image. Furthermore, the generator is trained with the discriminator parameters fixed so that the virtual image generated by the generator can deceive the discriminator, striving to achieve Nash equilibrium as much as possible.

[0047] Considering that illicit networks often target specific categories of information within images when illegally collecting image information, such as facial information in a user's image, if they cannot correctly identify the true category of the image information, they cannot accurately attack the image's privacy information. Based on the above, the inventive concept of this application is: to prevent illicit networks from accurately acquiring the corresponding images, the generator in this application is expected to generate high-quality images that also prevent the relevant network from acquiring them. Based on this principle, the embodiments of this application aim to generate high-quality images that are also intended to be mistaken by illicit networks as images they do not intend to acquire, thereby achieving the purpose of protecting images and privacy. Specifically, as follows... Figure 2 As shown in the embodiments of this application, in addition to using a discriminator to optimize the generator's parameters to ensure image quality during the generator training stage, a classifier is also used to optimize the generator's neural network parameters, so that the images generated by the generator can fool the classifier and be classified into other categories. As a result, the illegal network will also be unable to accurately determine the category of the image, thus achieving the purpose of deceiving the illegal network.

[0048] The training method for the generative adversarial network and the image generation method in the embodiments of this application will be described in detail below with reference to the accompanying drawings. Specifically, as follows... Figure 3The diagram shown is a schematic representation of an application environment according to an embodiment of this application. It includes a network 100, at least one terminal device 101, and a server 102.

[0049] At least one terminal device 101 communicates with the server 102 via a network 100, which can be a wired network 100 or a wireless network 100. The training method and image generation method of the generative adversarial network can be executed by the terminal device 101, the server 102, or by at least one terminal device 101 and the server 102 in concert.

[0050] After training the generative adversarial network (GAN), it is distributed to terminal device 101. In implementation, taking user video as an example, if the user does not want their face to be shown in the video, the system captures and locates the user's face within the video stream. Random noise is added to the generator to produce a virtual image. The generated virtual image then obscures the user's face to ensure that unauthorized networks cannot successfully identify the user's facial privacy information.

[0051] It should also be noted that the terminal device 101 may include, but is not limited to, tablet computers, various desktop computers, and smartphones, etc. Figure 3 The terminal devices shown are 101_1, 101_2, ..., 101_n. Server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud computing services.

[0052] The following is through Figure 4a The training method for generative adversarial networks provided in the embodiments of this application will be described in detail.

[0053] In related technologies, generative adversarial networks (GANs) are trained on the generator and discriminator using a minimax loss function, which is shown in Equation 1 below:

[0054]

[0055] Where pr represents the real data distribution, pz represents the distribution of random noise z, E represents the expectation, x represents the real data, D(x) represents the discrimination result of the real image input to the discriminator, and G(z) represents the virtual image generated by the generator based on random noise.

[0056] In training a generative adversarial network (GAN), the discriminator is trained with the generator's parameters fixed to optimize its discrimination results, ensuring it can recognize real images (i.e., the user images to be protected) and distinguish between virtual images. Similarly, the generator is trained with the discriminator's parameters fixed to ensure that virtual images generated by the generator are recognized as real images by the discriminator. During training, the discriminator is first trained N times using real data and virtual data distributions of virtual images. Then, the generator is trained with the discriminator parameters fixed. This iterative process allows the generator to reach convergence.

[0057] Considering that illicit networks often target specific categories of information when illegally collecting image data, such as facial features in real images, a dataset might contain 1000 photos belonging to 10 individuals, each photo categorized by an individual's ID. When a generative adversarial network (GAN) generates images from this dataset, the resulting classifier labels may differ from the actual labels, meaning the images and individuals cannot be correctly matched. The illicit network cannot extract the corresponding photos of individuals from a single category, thus failing to obtain useful information. Therefore, if the illicit network cannot correctly identify the true category of the image information, it cannot accurately attack the privacy information within the images.

[0058] This application adds a trained classifier to the existing generative adversarial network (GAN) structure, which can identify the category to which a real image belongs. When training the GAN provided in this application, step 401 is first executed: a first random variable is generated and input into the generator to obtain a first virtual image generated by the generator. The first random variable is random noise added to the generator.

[0059] After acquiring the first virtual image, step 402 is executed: keeping the parameters of the generator and the classifier unchanged, the parameters of the discriminator are updated based on the first virtual image and sample images from the second training set. The sample images in the second training set are real images. In practice, real images can be used to prevent illegal collection.

[0060] When training the discriminator, it is expected that the trained discriminator can identify the first input virtual image as a fake image and can identify the images in the second training set as real images.

[0061] In implementation, the first virtual image and the sample image are first input into the discriminator to obtain the discrimination results for the first virtual image and the sample image, respectively. The discriminator is a binary classifier that can output two results: 1 for true and 0 for false. Based on the expectations for the discriminator, the discrimination loss of the first virtual image needs to be determined by comparing the virtual image with a discriminator output of 1 (i.e., the first virtual image is considered a sample image by the discriminator) with the expected discrimination result (the first virtual image is expected to be correctly identified by the discriminator, i.e., the expected output is 0). Correspondingly, the discrimination loss of the sample image also needs to be determined by comparing the sample image with a discriminator output of 0 (i.e., the sample image is considered a fake image by the discriminator) with the expected discrimination result (the sample image is expected to be correctly identified by the discriminator, i.e., the expected output is 1). Finally, the discriminator is trained based on the discrimination loss of the first virtual image and the discrimination loss of the sample image so that the trained discriminator can achieve the expected results.

[0062] After training the discriminator, the generator needs to be trained with the discriminator's parameters fixed. The goal is for the generator to produce virtual images that the discriminator classifies as real (i.e., outputting a value of 1). Ideally, the generator should not correctly classify the generated virtual images as true.

[0063] When training the generator, step 403 is executed: a second random quantity is generated and input into the generator to obtain a second virtual image generated by the generator, and the second virtual image is input into the discriminator and the classifier respectively to obtain the discrimination result and classification result of the second virtual image.

[0064] To ensure the diversity of training samples, random noise can be added to the generator to make it generate a second virtual image different from the first virtual image. After obtaining the discrimination and classification results of the second virtual image, step 404 is executed: the discrimination loss of the second virtual image is determined based on the discrimination result of the second virtual image, and the classification loss is determined based on the classification result and the expected classification result of the second virtual image; wherein, the expected classification result is different from the real classification result of the second virtual image. That is, in order to deceive the classifier, the classification loss is determined primarily based on misclassification during the generator parameter optimization stage. The classifier in this application is pre-trained and is used to classify images of similar categories, for example, it can classify multiple different portraits into multiple categories according to differences in skin color, face shape, etc. The real classification of the second virtual image is the classification of its corresponding real image, i.e., category 1. During the generator optimization stage, category 2, which is different from category 1, can be used as the expected category of the second virtual image. The purpose of training is to make the real category of the second virtual image generated by the generator unable to be correctly identified by the classifier.

[0065] In order to achieve the purpose of deceiving illegal networks, the objective function of the optimized generative adversarial network in this application embodiment is shown in the following formula (2).

[0066] L GAN (D)=E (x，y) [log(D(x))]+E x Formula (2) is [log(1-D(x, G(x)))].

[0067] Among them, L GAN (D) represents the discriminative loss of the generator, E represents the expectation, x represents the real data, y represents the virtual image, D(x) represents the discrimination result of the discriminator, and G(x) represents the function related to x.

[0068] Therefore, based on the optimized objective function, the generator can be directly guided to generate images using real images.

[0069] To ensure that the virtual images generated by the trained generator cannot be identified by the classifier as real categories, the classification loss function used to adjust the generator parameters can be calculated based on the following formula (3).

[0070] L0 = E x [l c Formula (3) [(x, f)]

[0071] Where L0 represents the classification loss of the generator, L c Let f represent the gradient loss of the classifier, E represent the expectation, x represent the real data, and f represent the true classification of x.

[0072] Specifically, the discrimination and classification results of the second virtual image are obtained by inputting the second virtual image into the discriminator and classifier, respectively. Using the above formulas (2) and (3), the discrimination loss is determined based on the desired discrimination result and the discrimination result output by the discriminator for the second virtual image, and the classification loss is calculated based on the classification result of the real data and the classification result output by the classifier for the second virtual image. After obtaining the discrimination and classification losses of the second virtual image, step 405 is executed: keeping the parameters of the discriminator and the classifier unchanged, the parameters of the generator are updated based on the discrimination loss and classification loss of the second virtual image.

[0073] In practice, the discrimination loss and classification loss of the second virtual image can be assigned corresponding weights according to the actual situation. The total loss information used to train the generator can be obtained by weighted summation. The generator can then be trained based on the total loss information to make the generator converge.

[0074] To further incentivize illicit networks to misidentify the category of virtual images, it's crucial to ensure that the real category of the virtual image generated by the generator is similar to the category identified by the classifier. Specifically, a similarity threshold can be added during the generator training process. Similarity is the degree of similarity between the category corresponding to the classifier's expected classification result and the real category of the second virtual image. In this way, the classification result of the virtual image generated by the trained generator will be very close to its real category, making it easier to confuse illicit networks.

[0075] Considering the real-time and continuous nature of video streams, such as live streams, compared to static images, privacy protection for video images demands higher speed from generative adversarial networks (GANs). Given the complexity of GAN structures and the inability of their processing speed to meet the real-time requirements of video scenarios, this application employs a pruning algorithm to prune the GAN. This pruning algorithm may include weight pruning or channel pruning. A schematic diagram of the pruning algorithm is shown below. Figure 4b As shown, the pruning algorithm aims to remove branches that have little impact on the output of the generative adversarial network (GAN), thereby reducing computational load and improving network processing speed. The following sections describe the pruning process of the GAN in this application using weight pruning and channel pruning algorithms respectively:

[0076] Weight pruning primarily achieves its purpose by setting neuron parameters to zero, and then reduces model size through sparse storage. Parameters in generative adversarial networks (GANs) often contain redundancy, making weight pruning highly efficient. When performing weight pruning on a GAN, each layer has weights. Since weights can be positive or negative, directly judging based on weight magnitude and a threshold would result in the removal of some important negative weights. Therefore, the threshold can be chosen as the standard deviation of the weights for each layer. In practice, the standard deviation of the weights is calculated for each layer in the GAN, and this standard deviation is used as the weight threshold. Layers with weights whose absolute values ​​are less than the weight threshold are pruned.

[0077] Channel pruning essentially distinguishes the importance of convolutions in convolutional layers at the channel level. It reduces the total computation and model size by removing convolutional kernels that have a smaller impact on the generative adversarial network. It has the advantages of low computational requirements and high feasibility. When performing channel pruning on generative adversarial networks, L1 penalty can be used to prune the batch normalization (BN) layer. The BN layer normalizes all parameters of the input image in the network to a normal distribution with a mean of 0 and a variance of 1. Then, adjustment factors δ and μ are used to adjust the normalized values ​​to achieve gradient descent. Specifically, it can be shown in the following formulas (4) and (5):

[0078]

[0079] a i =μ i *τ+δ i Formula (5)

[0080] Where, a′ i a represents the image of each channel of the input. i Image adjusted for each channel; u and σ i These are the mean and variance, respectively; μ i and δ i These are the two adjustment factors corresponding to each channel image. Where μ i This can be viewed as the weight of each channel in the BN layer, when μ i When μ = 0, the output channel is a constant 0 and will not affect subsequent operations. Therefore, we subtract μ... i =0 or μ i The method of using the convolution kernels of the upper and lower convolutional layers corresponding to ≈0 is used to complete channel pruning.

[0081] Consider that μ values ​​typically exhibit a normal distribution in post-trained generative adversarial networks. That is, there are not many parameters that are close to or equal to 0. Based on this, the μ values ​​of each BN layer can be sparsed using the L1 gradient method. The formula for the L1 gradient method is shown in the following formula (6):

[0082]

[0083] Where e represents the learning rate of the loss function; η represents the gradient of the original loss function during training, and η is a hyperparameter that determines the magnitude of each gradient descent in the L1 gradient method.

[0084] Pruning a generative adversarial network (GAN) using a pruning algorithm inevitably affects its accuracy. To minimize this accuracy loss, the pruned GAN needs to be retrained. In practice, after pruning the GAN, random noise is regenerated and input into the generator to obtain a new virtual image. The process of adjusting the parameters of the pruned discriminator and generator based on the newly generated virtual image is iterative, repeating steps 401-405.

[0085] Based on the same inventive concept, this application also provides an image generation method using a generative adversarial network trained using embodiments of this application.

[0086] Because privacy protection for real images in video streams requires a higher speed for generative adversarial networks compared to using fixed real images, the image generation method provided in this application embodiment will be described below using privacy protection for real images in video streams as an example. Specifically, as follows... Figure 5 As shown, it includes the following steps:

[0087] For the target video stream to be protected, step 501 is performed: acquire the target image in the target video stream. The target image is the real image used for protection. After acquiring the target image, step 502 is performed: detect preset protection information in the target image. The preset protection information is privacy information in the real image, such as when a user does not want to show their face during a live stream and therefore chooses video effects to cover their face. In this case, the privacy information is the facial position in the real image.

[0088] After detecting the preset protection information, step 503 is executed: generating an occlusion image based on the generator. In practice, random noise is added to the generator to generate candidate protection images. These candidate protection images are input into the discriminator. If the discrimination result is true (i.e., the output is 1), it means that the candidate protection image can be recognized as a real image by the discriminator. In this case, the candidate protection image can be used as an occlusion image. Conversely, if the discrimination result is false (i.e., the output is 0), it means that the candidate protection image is recognized as a non-real image by the discriminator, indicating that the candidate protection image cannot be used as a real image by an illegal network and cannot achieve the purpose of protecting the real image. In this case, random noise is added again to the generator to generate new candidate protection images, and the newly generated candidate protection images are discriminated against until the discrimination result of the obtained candidate protection image is true.

[0089] After acquiring the occluded image, step 504 is executed: the occluded image is used to occlude the protection information. In this way, the generative adversarial network generates candidate protection images that can be recognized as real images by the discriminator as occluded images, and uses these occluded images to occlude the protection information in the real image. This causes malicious networks to mistakenly attack the occluded image when targeting the user's protection information, thus achieving the effect of protecting user privacy.

[0090] After introducing the training method and image generation method of the generative adversarial network provided in this application, the following refers to... Figure 6 This application describes an electronic device 130 according to one embodiment of the present application. The electronic device in this embodiment can be a smart terminal or a server; the smart terminal may be, for example, a smart TV, a smartphone, a smartwatch, etc. Figure 6 The electronic device 130 shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0091] like Figure 6 As shown, the electronic device 130 is presented in the form of a general-purpose electronic device. The components of the electronic device 130 may include, but are not limited to: at least one processor 131, at least one memory 132, and a bus 133 connecting different system components (including memory 132 and processor 131).

[0092] Bus 133 represents one or more of several bus structures, including a memory bus or memory controller, peripheral bus, processor, or local bus using any of the various bus structures.

[0093] The memory 132 may include a readable medium in the form of volatile memory, such as random access memory (RAM) 1321 and / or cache memory 1322, and may further include read-only memory (ROM) 1323.

[0094] The memory 132 may also include a program / utility 1325 having a set (at least one) of program modules 1324, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0095] Electronic device 130 can also communicate with one or more external devices 134 (e.g., keyboard, pointing device, etc.), and with one or more devices that enable a user to interact with electronic device 130, and / or with any device that enables electronic device 130 to communicate with one or more other electronic devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 135. Furthermore, electronic device 130 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 136. As shown, network adapter 136 communicates with other modules used in electronic device 130 via bus 133. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 130, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0096] In some possible implementations, various aspects of the generative adversarial network training method and image generation method or image processing method provided in this application can also be implemented in the form of a program product, which includes program code. When the program product is run on a computer device, the program code is used to cause the computer device to perform the steps in the generative adversarial network training method and image generation method or image processing method according to the various exemplary embodiments of this application described above.

[0097] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0098] The program product for image scaling according to the embodiments of this application can be a portable compact disc read-only memory (CD-ROM) and include program code, and can run on an electronic device. However, the program product of this application is not limited thereto. In this document, the readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0099] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take many forms, including—but not limited to—electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0100] The program code contained on the readable medium may be transmitted using any suitable medium, including—but not limited to—wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0101] Program code for performing the operations of this application can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's electronic device, partially on the user's device, as a standalone software package, partially on the user's electronic device and partially on a remote electronic device, or entirely on a remote electronic device or server. In cases involving remote electronic devices, the remote electronic device can be connected to the user's electronic device via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external electronic device (e.g., via the Internet using an Internet service provider).

[0102] It should be noted that although several units or sub-units of the device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this application, the features and functions of two or more units described above can be embodied in one unit. Conversely, the features and functions of one unit described above can be further divided and embodied by multiple units.

[0103] Furthermore, although the operations of the method of this application are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable image scaling device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable image scaling device, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable image scaling device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions can also be loaded onto a computer or other programmable image scaling device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] Although preferred embodiments of this application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of this application.

[0109] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A training method for generative adversarial networks, characterized in that, The generative adversarial network includes a generator, a discriminator, and a trained classifier, and the method includes: A first random quantity is generated and input into the generator to obtain the first virtual image generated by the generator; Keeping the parameters of the generator and the classifier unchanged, the first virtual image and the sample images in the second training set are respectively input into the discriminator to obtain the discrimination result of the first virtual image and the discrimination result of the sample images; A first discrimination loss is determined based on the discrimination result of the first virtual image and the first expected discrimination result, and a second discrimination loss is determined based on the discrimination result of the sample image and the second expected discrimination result; The parameters of the discriminator are updated based on the first discriminant loss and the second discriminant loss; A second random quantity is generated and input into the generator to obtain a second virtual image generated by the generator. The second virtual image is then input into the discriminator and the classifier respectively to obtain the discrimination result and classification result of the second virtual image. The discrimination loss of the second virtual image is determined based on the discrimination result of the second virtual image, and the classification loss is determined based on the classification result and the expected classification result of the second virtual image; wherein the expected classification result is different from the actual classification result of the second virtual image; Keeping the parameters of the discriminator and the classifier unchanged, the discrimination loss and the classification loss of the second virtual image are weighted and summed to obtain the total loss information; The generator parameters are updated based on the total loss information.

2. The method according to claim 1, characterized in that, After updating the generator parameters based on the discriminative loss and the classification loss of the second virtual image, the method further includes: The generative adversarial neural network is pruned using a pruning algorithm, which includes weight pruning or channel pruning.

3. The method according to claim 2, characterized in that, After pruning the generative adversarial neural network using a pruning algorithm, the method further includes: The process of regenerating a third random variable and adjusting the parameters of the pruned discriminator and generator based on the third random variable is iterated.

4. The method according to claim 1, characterized in that, The categories corresponding to the expected classification results and the categories corresponding to the actual classification results are categories whose category similarity is greater than a preset threshold.

5. An image generation method using generative adversarial networks, characterized in that, The generative adversarial network includes a generator and a discriminator, and the method includes: Acquire the target image from the target video stream; Preset protection information is detected in the target image; If the preset protection information is detected, an occlusion image is generated based on the generator; The protection information is obscured by the obscuring image; The generator is trained based on the method described in any one of claims 1-4.

6. The method according to claim 5, characterized in that, The generation of the occluded image based on the generator includes: A fourth random quantity is generated and input into the generator to obtain the candidate protected image generated by the generator; The candidate protected image is input into the discriminator. If the discrimination result of the candidate protected image is true, then the candidate protected image is used as the occlusion image.

7. The method according to claim 6, characterized in that, If the discrimination result of the candidate protected image is false, then return to the step of generating the fourth random quantity and inputting it into the generator to obtain the candidate protected image generated by the generator, until the discrimination result of the obtained candidate protected image is true.

8. An electronic device, characterized in that, The method includes at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-7.

Images