Training method of image generation model, image generation method, device and equipment

Abstract

The application discloses a training method of an image generation model, an image generation method, an image generation device and equipment, and belongs to the technical field of artificial intelligence. The method comprises the following steps: acquiring a sample image and a neural network model; extracting a first sample feature from the sample image according to the neural network model, wherein the first sample feature is related to the image category of the sample image; selecting a first memory feature from a memory feature set of the neural network model; fusing the first sample feature and the first memory feature according to the neural network model; generating any one prediction image based on a fusion result; the first memory feature is a feature learned by the neural network model and is irrelevant to the image category; and adjusting the neural network model according to at least two prediction images to obtain an image generation model. The prediction image of the application is based on the same image category as that of the sample image, and combines the information irrelevant to the image category learned by the model, so that the prediction image has content diversity.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a training method for an image generation model, an image generation method, an apparatus, and equipment. Background Art

[0002] In the field of artificial intelligence, neural network models are being applied more and more widely. For example, by training a neural network model with sample images, an image generation model can be obtained, which can then generate a target image.

[0003] In related technologies, Gaussian noise is first randomly generated and fed into a neural network model, which then generates a predicted image based on the Gaussian noise. Sample images can also be obtained. The predicted and sample images are used to determine the loss of the neural network model. The model is then adjusted based on the loss, and the adjusted model is used as the image generation model.

[0004] This technique uses the predicted image and the sample image to adjust the model. This ensures that when the model generates a predicted image based on Gaussian noise, the predicted image will increasingly resemble the sample image. This means that the predicted image's content becomes increasingly similar to the sample image's. Consequently, the predicted image, and therefore the image generated by the model, has a single content. Summary of the Invention

[0005] The present application provides a training method, image generation method, device and equipment for an image generation model, which can be used to solve the problem of single content of images generated by the model in related technologies. The technical solution includes the following contents.

[0006] In one aspect, a method for training an image generation model is provided, the method comprising:

[0007] Get sample images and neural network models;

[0008] extracting a first sample feature from the sample image according to the neural network model, where the first sample feature is related to an image category of the sample image;

[0009] Selecting a first memory feature from a memory feature set of the neural network model, fusing the first sample feature and the first memory feature according to the neural network model, and generating any predicted image based on the fusion result, wherein the first memory feature is a feature learned by the neural network model and is unrelated to the image category;

[0010] The neural network model is adjusted according to at least two predicted images to obtain an image generation model.

[0011] In another aspect, a method for generating an image is provided, the method comprising:

[0012] Obtaining a reference image and an image generation model, wherein the image generation model is trained according to any of the above-mentioned image generation model training methods;

[0013] extracting target features from the reference image according to the image generation model, wherein the target features are related to the image category of the reference image;

[0014] Selecting a target memory feature from the memory feature set of the image generation model, wherein the target memory feature is a feature learned by the image generation model and is independent of the image category;

[0015] The target feature and the target memory feature are fused according to the image generation model, and a target image is generated based on the fusion result.

[0016] In another aspect, a training device for an image generation model is provided, the device comprising:

[0017] Acquisition module, used to obtain sample images and neural network models;

[0018] an extraction module, configured to extract a first sample feature from the sample image according to the neural network model, where the first sample feature is related to an image category of the sample image;

[0019] a selection and generation module, configured to select a first memory feature from the memory feature set of the neural network model, fuse the first sample feature with the first memory feature according to the neural network model, and generate any predicted image based on the fusion result, wherein the first memory feature is a feature learned by the neural network model and is unrelated to the image category;

[0020] An adjustment module is used to adjust the neural network model according to at least two predicted images to obtain an image generation model.

[0021] In a possible implementation, the number of the first memory features is at least two;

[0022] The selection and generation module is configured to perform interpolation processing on at least two first memory features to obtain interpolation features; fuse the first sample features and the interpolation features, and generate any one of the predicted images based on the fusion result.

[0023] In a possible implementation, the acquisition module is further configured to acquire the image category of the sample image and the image category of each predicted image;

[0024] The adjustment module is used to adjust the neural network model according to at least two predicted images, the image category of the sample image and the image category of each predicted image to obtain an image generation model.

[0025] In a possible implementation, the acquisition module is further configured to acquire authenticity labeling results and authenticity prediction results of each predicted image;

[0026] The adjustment module is used to adjust the neural network model according to at least two predicted images, the authenticity labeling results of each predicted image, and the authenticity prediction results to obtain an image generation model.

[0027] In a possible implementation, the extraction module is further configured to extract a second sample feature from the sample image according to the neural network model, where the second sample feature is irrelevant to the image category of the sample image;

[0028] The selection and generation module is further configured to generate a reconstructed image based on the first sample feature and the second sample feature;

[0029] The adjustment module is used to adjust the neural network model according to at least two predicted images, the reconstructed image and the sample image to obtain an image generation model.

[0030] In one possible implementation, the selection and generation module is used to encode the second sample feature to obtain encoding information; based on the encoding information, select a second memory feature from the memory feature set of the neural network model, where the second memory feature is a feature learned by the neural network model, and the second memory feature is independent of the image category and is related to the second sample feature; and generate a reconstructed image based on the second memory feature and the first sample feature.

[0031] In one possible implementation, the adjustment module is used to determine a first loss value based on the reconstructed image and the sample image; determine a second loss value based on the at least two predicted images; and adjust the neural network model based on the first loss value and the second loss value to obtain an image generation model.

[0032] In a possible implementation, the acquisition module is further configured to acquire the authenticity annotation result and the authenticity prediction result of the reconstructed image, and the authenticity annotation result and the authenticity prediction result of each predicted image;

[0033] The adjustment module is used to adjust the neural network model according to the authenticity labeling result and authenticity prediction result of the reconstructed image, the authenticity labeling results and authenticity prediction results of each predicted image, the reconstructed image, the sample image and the at least two predicted images to obtain an image generation model.

[0034] In another aspect, an image generating apparatus is provided, the apparatus comprising:

[0035] an acquisition module, configured to acquire a reference image and an image generation model, wherein the image generation model is trained according to any of the above-mentioned image generation model training methods;

[0036] an extraction module, configured to extract target features from the reference image according to the image generation model, wherein the target features are related to the image category of the reference image;

[0037] a selection module, configured to select a target memory feature from the memory feature set of the image generation model, wherein the target memory feature is a feature learned by the image generation model and is independent of the image category;

[0038] A generation module is used to fuse the target feature and the target memory feature according to the image generation model, and generate a target image based on the fusion result.

[0039] On the other hand, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above-mentioned image generation model training methods or any of the above-mentioned image generation methods.

[0040] On the other hand, a computer-readable storage medium is also provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image generation model training methods or any of the above-mentioned image generation methods.

[0041] On the other hand, a computer program or computer program product is also provided, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image generation model training methods or any of the above-mentioned image generation methods.

[0042] The technical solution provided by this application brings at least the following beneficial effects:

[0043] The technical solution provided by the present application is to extract a first sample feature from a sample image, fuse the first sample feature and the first memory feature, and generate a predicted image based on the fusion result. Since the first sample feature is related to the image category of the sample image, and the first memory feature is a feature learned by the neural network model and is unrelated to the image category, the predicted image combines the information learned by the model that is unrelated to the image category on the basis of ensuring that the image category is the same as the image category of the sample image, so that the content of the predicted image has random diversity, that is, the content of the image generated by the model has random diversity. In the process of adjusting the neural network model according to at least two predicted images to obtain an image generation model, the model continuously learns features that are unrelated to the image category, that is, continuously enriches features that are unrelated to the image category, further improving the content diversity of the predicted image. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0045] Figure 1 Schematic diagram of an implementation environment of a training method for an image generation model provided in an embodiment of the present application;

[0046] Figure 2 This is a flowchart of a method for training an image generation model provided in an embodiment of the present application;

[0047] Figure 3 This is a flowchart of an image generation method provided by an embodiment of the present application;

[0048] Figure 4 Schematic diagram of a training method for an image generation model provided in an embodiment of the present application;

[0049] Figure 5 This is a schematic diagram of generating a predicted image provided by an embodiment of the present application;

[0050] Figure 6 is a schematic diagram of a target image generated by a different image generation model provided in an embodiment of the present application;

[0051] Figure 7 Schematic diagram of a training device for an image generation model provided in an embodiment of the present application;

[0052] Figure 8 is a structural diagram of an image generating device provided in an embodiment of the present application;

[0053] Figure 9 This is a schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0054] Figure 10 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0056] Figure 1 This is a schematic diagram of an implementation environment of a training method for an image generation model provided in an embodiment of the present application, such as Figure 1 As shown, the implementation environment includes a terminal device 101 and a server 102. The training method of the image generation model in the embodiment of the present application can be executed by the terminal device 101, or by the server 102, or by the terminal device 101 and the server 102 together.

[0057] The terminal device 101 can be a smart phone, a game console, a desktop computer, a tablet computer, a laptop computer, a smart TV, a smart car device, an intelligent voice interaction device, a smart home appliance, etc. The server 102 can be a single server, or a server cluster consisting of multiple servers, or any one of a cloud computing platform and a virtualization center, which is not limited in the embodiments of the present application. The server 102 can be connected to the terminal device 101 via a wired network or a wireless network. The server 102 can have functions such as data processing, data storage, and data transmission and reception, which are not limited in the embodiments of the present application. The number of terminal devices 101 and servers 102 is not limited and can be one or more.

[0058] The training method or image generation method of the image generation model provided in the embodiment of the present application can be implemented based on artificial intelligence technology. Artificial Intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0059] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, as well as machine learning / deep learning, autonomous driving, and smart transportation.

[0060] Computer vision (CV) is the science of making machines "see." Specifically, it refers to machine vision, where cameras and computers replace the human eye in identifying and measuring objects, performing further image processing to create images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, and smart transportation.

[0061] Based on the above implementation environment, the present invention provides a training method for an image generation model. Figure 2 Taking the flowchart of a training method of an image generation model provided in the embodiment of the present application as an example, the method can be performed by Figure 1 The terminal device 101 or server 102 in the embodiment of the present application can also execute the training method of the image generation model. For the convenience of description, the terminal device 101 or server 102 that executes the training method of the image generation model in the embodiment of the present application is referred to as an electronic device, and the method can be executed by an electronic device. Figure 2 As shown, the method includes steps 201 to 204.

[0062] Step 201: Obtain a sample image and a neural network model.

[0063] The embodiments of the present application do not limit the method, content, or quantity of obtaining sample images. For example, a sample image is any frame image extracted from a video, or a sample image is an image taken by a user, or a sample image is captured from the Internet, or a generated image. The number of sample images is at least one, and a sample image can be an image randomly sampled from a sample image set, wherein the sample image set includes multiple sample images. The sample image can be a photographic image of an animal, a building, a road, or the like, or a scanned image.

[0064] The embodiment of the present application does not limit the model structure, size, etc. of the neural network model. The neural network model is used to output at least one predicted image corresponding to any sample image in the input sample image set.

[0065] Step 202: extract a first sample feature from the sample image according to the neural network model, where the first sample feature is related to the image category of the sample image.

[0066] Optionally, the neural network model includes a first encoder. After the sample image is input into the neural network model, the first encoder extracts features of the sample image (e.g., at least one downsampling process) to obtain a first sample feature. A sample image x randomly sampled from the sample image set is input into the first encoder E. cr Afterwards, the first sample feature f is obtained cr Optionally, the first encoder is a residual network (such as a Res Net network).

[0067] The first sample feature includes, but is not limited to, at least one of a contour feature of an object in the sample image (used to describe the contour of the object in the sample image and can be used to identify the image category of the sample image), a texture feature of an object in the sample image (used to describe the texture of the object in the sample image and can be used to identify the image category of the sample image), etc. Therefore, the first sample feature is related to the image category of the sample image and can be used to identify the image category of the sample image.

[0068] Step 203: Select a first memory feature from the memory feature set of the neural network model, fuse the first sample feature and the first memory feature according to the neural network model, and generate any predicted image based on the fusion result. The first memory feature is a feature learned by the neural network model and is unrelated to the image category.

[0069] In an embodiment of the present application, the neural network model stores a memory feature set, which includes at least one memory feature. For any memory feature, the memory feature is a feature shared by at least one sample image in the sample image set learned by the neural network model, and the memory feature is independent of the image category. Because the memory feature is a feature shared by at least one sample image, the memory feature can be a feature shared by images of different image categories (i.e., a reusable feature between categories).

[0070] For example, information such as the posture of the object in the sample image (e.g., sitting upright, lying on its side, tilted to the left, etc.), the color of the sample image (e.g., red, gray, green, etc.), the expression of the object in the sample image (e.g., smiling, crying, etc.), and the number of objects in the sample image cannot be used to identify the image category of the sample image. Therefore, the neural network model can learn at least one memory feature that is shared by at least one sample image in the sample image set and is independent of the image category. Any of the memory features can be used to describe the color of the at least one sample image (i.e., the memory feature is a color feature), or the posture of the object in at least one sample image (i.e., the memory feature is a posture feature), or the expression of the object in at least one sample image (i.e., the memory feature is an expression feature), or the number of objects in at least one sample image (i.e., the memory feature is a quantity feature), etc. The neural network model can store at least one memory feature, wherein the set consisting of the at least one memory feature can be referred to as a memory feature set.

[0071] For example, in sample image A, the subject's head is tilted 30 degrees to the left, while in sample image B, it is tilted 28 degrees to the left. The neural network model can learn a common memory feature from sample images A and B, independent of the image category: the posture feature corresponding to the leftward tilt. The posture feature corresponding to the leftward tilt is stored, resulting in a memory feature in the memory feature set.

[0072] In this embodiment of the present application, the neural network model further includes a first generator. At least one memory feature can be randomly selected from the memory feature set, and any randomly selected memory feature can be recorded as a first memory feature. The first generator can generate a predicted image based on the first sample feature and the at least one first memory feature.

[0073] Optionally, the number of the first memory feature is one, and the first generator can generate a predicted image based on the first sample feature and the first memory feature. For example, the first generator generates a predicted image based on the first sample feature f cr and first memory characteristics Generate a predicted image The first generator is based on the first sample feature f cr and first memory characteristics Generate a predicted image

[0074] In the embodiment of the present application, the first generator may generate a predicted image according to the following steps:

[0075] Step 1: Concatenate the first sample feature and the first memory feature to obtain a concatenated feature.

[0076] Step 2: normalize the concatenated feature based on the first memory feature to obtain a first normalized feature, and upsample the first normalized feature to obtain a first upsampled feature.

[0077] In step 3, in response to the dimension of the first upsampled feature being smaller than the target dimension (such as 32*32), the first upsampled feature is used as the concatenated feature for the next cycle, and step 2 is executed repeatedly until the dimension of the first upsampled feature is not smaller than the target dimension.

[0078] Step 4: In response to the dimension of the first upsampled feature being not less than the target dimension, the first upsampled feature is normalized based on the first sample feature to obtain a second normalized feature, and the second normalized feature is upsampled to obtain a second upsampled feature.

[0079] In step 5, in response to the second up-sampled feature not being the feature obtained by the last up-sampling process, the second up-sampled feature is used as the first up-sampled feature of the next cycle, and step 4 is executed repeatedly until the second up-sampled feature is the feature obtained by the last up-sampling process.

[0080] In step 6, in response to the second up-sampled feature being the feature obtained by the last up-sampling process, the second up-sampled feature may be used as a fusion result, and a predicted image is determined based on the second up-sampled feature (ie, based on the fusion result).

[0081] Optionally, in step 2, the concatenated features are normalized based on the first memory features to obtain the first normalized features, wherein the first memory features are used as parameters for adaptive instance normalization, and the concatenated features are subjected to adaptive instance normalization based on the parameters for adaptive instance normalization to obtain the first normalized features. Similarly, in step 4, the first upsampled features are normalized based on the first sample features to obtain the second normalized features, wherein the first sample features are used as parameters for adaptive instance normalization, and the first upsampled features are subjected to normalization based on the parameters for adaptive instance normalization to obtain the second normalized features.

[0082] Taking the splicing feature as an example, the implementation principle of "performing adaptive instance normalization on the splicing feature based on the parameters of the adaptive instance normalization process to obtain the first standardized feature" is explained below. The implementation principle is the same as the implementation principle of "performing adaptive instance normalization on the first upsampled feature based on the parameters of the adaptive instance normalization process to obtain the second standardized feature".

[0083] First, the variance and mean of the spliced ​​features are determined. Based on the variance and mean of the spliced ​​features, the spliced ​​features are normalized to obtain a normalized result. Next, using the parameters of the adaptive instance normalization process, a variance (denoted as the predicted variance) and a mean (denoted as the predicted mean) are determined. Based on the predicted variance and the predicted mean, the normalized result is denormalized to obtain a denormalized result, which is the first normalized feature.

[0084] It should be noted that when the dimension of the first up-sampling feature is smaller than the target dimension, the embodiment of the present application obtains the first up-sampling feature based on the first memory feature. Since the dimension of the first up-sampling feature is small, the first memory feature can be better utilized to change the content, structure, etc. of the image, so that the predicted image has random diversity. When the dimension of the first up-sampling feature is not smaller than the target dimension, the embodiment of the present application obtains the second up-sampling feature based on the first sample feature. Since the dimension of the first up-sampling feature is large, it is difficult to use the first memory feature to change the content, structure, etc. of the image. Therefore, the first sample feature is used to optimize the texture, contour, etc. of the image, so as to ensure that the image category of the predicted image is consistent with the image category of the sample image.

[0085] Optionally, there are at least two first memory features; the first sample feature and the first memory feature are fused, and any predicted image is generated based on the fusion result, including: interpolating at least two first memory features to obtain interpolated features; fusing the first sample feature and the interpolated feature, and generating any predicted image based on the fusion result.

[0086] In the embodiment of the present application, any interpolation algorithm, such as the nearest neighbor interpolation algorithm, the bilinear interpolation algorithm, etc., can be used to interpolate at least two first memory features to obtain interpolation features. The first generator generates any predicted image based on the first sample feature and the interpolation feature. For example, for the first memory feature and first memory characteristics Perform interpolation processing to obtain interpolation features The first generator is based on the first sample feature f cr and interpolation features Generate a predicted image

[0087] Optionally, one of the at least two first memory features is recorded as the first memory feature A, and each of the at least two first memory features other than the first memory feature A is recorded as each first memory feature B. For any first memory feature B, the difference between the first memory feature A and the first memory feature B is determined, and the ratio of the difference to the target data is used as the unit feature between the first memory feature A and the first memory feature B. In this way, the unit feature between the first memory feature A and each first memory feature B can be determined. Based on the first memory feature A and the unit features between the first memory feature A and each first memory feature B, an interpolation feature is determined. By performing interpolation processing on at least two first memory features, the representation capability of the features is enriched, thereby improving the diversity of the predicted image.

[0088] In the embodiment of the present application, the interpolation feature corresponds to the first memory feature in steps 1 to 6. The first generator generates a predicted image based on the first sample feature and the interpolation feature in the manner of steps 1-6, which will not be repeated here.

[0089] Step 204: Adjust the neural network model according to at least two predicted images to obtain an image generation model.

[0090] After acquiring at least two predicted images, a loss value between any two of the at least two predicted images can be determined. For example, a norm (e.g., a 1-norm, a 2-norm, etc.) of the difference between the two predicted images is determined, and the loss value between the two predicted images is determined based on the norm and an image difference parameter. The image difference parameter is used to control the image difference between the two predicted images.

[0091] Optionally, the loss value between any two predicted images is determined according to the mean square loss function shown in formula (1).

[0092]

[0093] Among them, L ds Represents the predicted image With the predicted image The loss value between α and α is the image difference parameter used to control the predicted image With the predicted image The image difference between E x is the symbol of the mean square loss function. ‖A‖1 represents the 1-norm of A, where A is the independent variable.

[0094] In an embodiment of the present application, the loss value of the neural network model can be determined using the loss value between each two predicted images in the at least two predicted images. For example, the loss value of the neural network model can be obtained by performing a weighted summation of the loss values ​​between each two predicted images in the at least two predicted images. The model parameters of the neural network model can be adjusted using the loss value of the neural network model to obtain an adjusted neural network model.

[0095] If the training end conditions are met, such as reaching the target number of training times, the gradient of the loss value of the neural network model no longer decreases, etc., the adjusted neural network model is used as the image generation model. If the training end conditions are not met, the adjusted neural network model is used as the neural network model for the next training, and the model parameters of the neural network model are adjusted again in accordance with steps 201 to 204 until the training end conditions are met to obtain the image generation model. Among them, the embodiment of the present application does not limit the target number of training times. For example, the target number of training times N = 15000.

[0096] Optionally, a loss value between the sample image and any predicted image is determined using the sample image and any predicted image. In this way, the loss value between the sample image and each predicted image can be determined. Based on the loss value between the sample image and each predicted image, the loss value of the neural network model is determined, and the neural network model is adjusted based on the loss value of the neural network model to obtain an image generation model.

[0097] In one possible implementation, before adjusting the neural network model according to at least two predicted images to obtain the image generation model, it also includes: obtaining the image category of the sample image and the image category of each predicted image; adjusting the neural network model according to at least two predicted images to obtain the image generation model, including: adjusting the neural network model according to at least two predicted images, the image category of the sample image, and the image category of each predicted image to obtain the image generation model.

[0098] In an embodiment of the present application, the sample image can be subjected to image classification processing to obtain the image category of the sample image. Similarly, each predicted image can be subjected to image classification processing to obtain the image category of each predicted image. The embodiment of the present application does not limit the method of image classification processing. Exemplarily, taking the sample image as an example, at least one image feature of the sample image, such as texture features, contour features, and color features, is first determined, and then the image category of the sample image is determined based on the image features of the sample image. Among them, the method of performing image classification processing on the predicted image is similar to the method of performing image classification processing on the sample image, and will not be repeated here.

[0099] Optionally, for any predicted image, the loss value between the image category of the sample image and the image category of the predicted image is determined based on the image category of the sample image and the image category of the predicted image, and the detailed determination method is not limited here. In this way, the loss value between the image category of the sample image and the image category of each predicted image can be determined. In addition, the loss value between each two predicted images can be determined based on at least two predicted images. Based on the loss value between the image category of the sample image and the image category of each predicted image, and the loss value between each two predicted images, the loss value of the neural network model is calculated, and the neural network model is adjusted based on the loss value of the neural network model to obtain an image generation model.

[0100] In one possible implementation, before adjusting the neural network model based on at least two predicted images to obtain the image generation model, it also includes: obtaining the authenticity labeling results and authenticity prediction results of each predicted image; adjusting the neural network model based on at least two predicted images to obtain the image generation model, including: adjusting the neural network model based on at least two predicted images, the authenticity labeling results and the authenticity prediction results of each predicted image to obtain the image generation model.

[0101] In an embodiment of the present application, any predicted image can be input into the discriminator, and the discriminator determines and outputs the authenticity prediction result of the predicted image. The authenticity prediction result of the predicted image is used to characterize the probability that the predicted image is a real image. Therefore, the value of the authenticity prediction result can be greater than or equal to 0 and less than or equal to 1. In addition, the predicted image can be labeled to obtain the authenticity labeling result of the predicted image. The authenticity labeling result of the predicted image is used to characterize whether the predicted image is a real image. Therefore, the value of the authenticity labeling result can be 0 or 1. Among them, 0 represents that the predicted image is a non-real image, and 1 represents that the predicted image is a real image.

[0102] The loss value between the authenticity labeling result and the authenticity prediction result of any predicted image can be determined based on the authenticity labeling result of the predicted image and the authenticity prediction result of the predicted image. The determination method is not limited here. In this way, the loss value between the authenticity labeling result and the authenticity prediction result of each predicted image can be determined. In addition, the loss value between each two predicted images can be determined based on at least two predicted images. Based on the loss value between each two predicted images and the loss value between the authenticity labeling result and the authenticity prediction result of each predicted image, the loss value of the neural network model is calculated, and the neural network model is adjusted based on the loss value of the neural network model to obtain an image generation model.

[0103] It should be noted that when training the neural network model, the discriminator can be connected in series after the neural network model, and the neural network model and the discriminator can be trained multiple times using the authenticity prediction results of each predicted image output by the neural network model and each predicted image output by the discriminator until the training end conditions are met, and the trained neural network model is used as the image generation model.

[0104] When training the neural network model and the discriminator, on the one hand, the discriminator can be fixed, and the loss value of the neural network model can be determined based on at least two predicted images, the authenticity labeling results of each predicted image, and the authenticity prediction results of each predicted image, and the neural network model can be adjusted based on the loss value of the neural network model. At this time, the authenticity labeling result of the predicted image is 1. On the other hand, the neural network model can be fixed, and the loss value of the discriminator can be determined based on at least two predicted images, the authenticity labeling results of each predicted image, and the authenticity prediction results of each predicted image, based on the same method as "determining the loss value of the neural network model", and the discriminator can be adjusted based on the loss value of the discriminator. At this time, the authenticity labeling result of the predicted image is 0. Based on the above two methods, the neural network model and the discriminator are trained multiple times until the training end conditions are met, thereby obtaining a trained neural network model and a trained discriminator. The trained neural network model is the image generation model of the embodiment of the present application.

[0105] Optionally, the neural network model can be adjusted based on at least two predicted images, the authenticity labeling results of each predicted image, the authenticity prediction results of each predicted image, the image category of the sample image, and the image category of each predicted image to obtain the image generation model. Specifically, the loss value of the neural network model is determined based on the loss value between each pair of predicted images, the loss value between the authenticity labeling results and the authenticity prediction results of each predicted image, and the loss value between the image category of the sample image and the image category of each predicted image. The neural network model is then adjusted based on the loss value of the neural network model to obtain the image generation model.

[0106] In one possible implementation, after obtaining the sample image and the neural network model, it also includes: extracting a second sample feature from the sample image according to the neural network model, where the second sample feature is independent of the image category of the sample image; generating a reconstructed image according to the first sample feature and the second sample feature; adjusting the neural network model according to at least two predicted images to obtain an image generation model, including: adjusting the neural network model according to at least two predicted images, the reconstructed image and the sample image to obtain an image generation model.

[0107] Optionally, the neural network model includes a second encoder, wherein the first encoder and the second encoder are two different encoders. Optionally, the second encoder is a residual network (such as a Res Net network). After the sample image is input into the neural network model, the second encoder extracts features of the sample image to obtain a second sample feature. A sample image x randomly sampled from the sample image set is input into the second encoder E. ci Afterwards, the second sample feature is obtained

[0108] The second sample feature includes, but is not limited to, at least one of a posture feature of an object in the sample image (used to describe the posture of the object in the sample image), a color feature (used to describe the color of the sample image), an expression feature (used to describe the expression of the object in the sample image), and a quantity feature (used to describe the number of objects in the sample image). The second sample feature may be a feature that is irrelevant to the image category of the sample image and cannot be used to identify the image category of the sample image.

[0109] For example, if the head of the object in sample image A is tilted 30 degrees to the left, the second sample feature obtained after the second encoder extracts features from the sample image can be used to describe that the head of the object in sample image A is tilted 30 degrees to the left.

[0110] Optionally, the neural network model includes a second generator, wherein the first generator and the second generator can be the same generator or two different generators. In the embodiment of the present application, the second generator can generate a reconstructed image based on the first sample feature and the second sample feature. The reconstructed image is the reconstructed sample image.

[0111] In this embodiment of the present application, the second sample feature corresponds to the first memory feature in steps 1 to 6, the reconstructed image corresponds to the predicted image in steps 1 to 6, and the second generator generates the reconstructed image based on the first sample feature and the second sample feature in accordance with the above steps 1-6, which will not be repeated here.

[0112] In one possible implementation, a reconstructed image is generated based on the first sample feature and the second sample feature, including: encoding the second sample feature to obtain encoding information; based on the encoding information, selecting a second memory feature from a memory feature set of a neural network model, where the second memory feature is a feature learned by the neural network model, the second memory feature is independent of the image category, and is related to the second sample feature; and generating a reconstructed image based on the second memory feature and the first sample feature.

[0113] Optionally, the neural network model includes an addresser. The addresser can encode the second sample feature to obtain encoded information. The embodiment of the present application does not limit the method for encoding the second sample feature. Exemplarily, any encoding algorithm such as one-hot encoding or multi-hot encoding is used to encode the second sample feature to obtain encoded information.

[0114] Taking one-hot encoding as an example, in an embodiment of the present application, the second sample feature is encoded based on the Gumbel-normalized exponential function (Gumbel-Softmax) to obtain the encoded information after the second sample feature is one-hot encoded, wherein the use of Gumbel-Softmax can ensure that the neural network model is differentiable and trainable. The second sample feature can be normalized (i.e., Softmax processing) to obtain the normalized second sample feature. The second sample feature is randomly sampled according to the Gumbel distribution to obtain a random variable. Based on the normalized second sample feature and the random variable, the encoding information is determined. Please refer to formula (2) shown below.

[0115]

[0116] Among them, π i is the encoded information after one-hot encoding of the second sample feature. exp is the symbol of the exponential function. is the second sample feature. Characterizes the second sample feature after normalization. g i Characterizes the random variable obtained by randomly sampling the second sample feature according to the Gumbel distribution. τ is a hyperparameter used to adjust π i k is the number of the second sample features.

[0117] The addresser may select a second memory feature from the memory feature set of the neural network model based on the encoding information. Optionally, the addresser multiplies the memory feature set of the neural network model by the encoding information as the second memory feature, as shown in Formula (3) below.

[0118] f ci =π i ·M Formula (3)

[0119] Among them, f ci is the second memory feature. i is the encoded information after one-hot encoding of the second sample feature. M is the memory feature set of the neural network model.

[0120] Next, the second generator can generate a reconstructed image based on the first sample feature and the second memory feature, as shown in formula (4) below.

[0121] x gen =G(f ci ,f cr ) Formula (4)

[0122] Among them, x gen Characterize the reconstructed image. f ci Characterize the second memory characteristics. cr Characterizes the first sample feature. G characterizes the second generator.

[0123] In an embodiment of the present application, the reconstructed image is used to adjust the neural network model. Optionally, adjusting the neural network model based on at least two predicted images, the reconstructed image, and the sample image to obtain an image generation model includes: determining a first loss value based on the reconstructed image and the sample image; determining a second loss value based on the at least two predicted images; and adjusting the neural network model based on the first loss value and the second loss value to obtain the image generation model.

[0124] The first loss value may be determined according to the reconstructed image and the sample image according to the mean square loss function shown in formula (5).

[0125] L R =E x ‖xx gen ‖1 Formula (5)

[0126] Among them, L R Represents the first loss value, E x represents the symbol of the mean square loss function, x represents the sample image, x gen Represents the reconstructed image, ‖A‖1 represents the 1-norm of A, and A is the independent variable.

[0127] A loss value between any two of the at least two predicted images can be determined. A second loss value can be determined based on the loss value between each pair of predicted images. Optionally, a weighted sum of the loss values ​​between each pair of predicted images is performed to obtain the second loss value. Subsequently, a loss value of the neural network model is determined based on the first loss value and the second loss value, so as to adjust the neural network model and obtain an image generation model.

[0128] Optionally, image classification processing can be performed on the sample image to obtain the image category of the sample image. Similarly, image classification processing can be performed on the reconstructed image to obtain the image category of the reconstructed image. The image classification processing method for the reconstructed image is similar to the image classification processing method for the sample image, and will not be repeated here.

[0129] Optionally, a loss value between the image category of the sample image and the image category of the reconstructed image is determined based on the image category of the sample image and the image category of the reconstructed image. The detailed determination method is not limited here. In addition, a first loss value can be determined based on the reconstructed image and the sample image, and a second loss value can be determined based on at least two predicted images. Based on the loss value between the image category of the sample image and the image category of the reconstructed image, the first loss value, and the second loss value, the loss value of the neural network model is calculated, and the neural network model is adjusted based on the loss value of the neural network model to obtain an image generation model.

[0130] In an embodiment of the present application, the reconstructed image can be input into a discriminator, which determines and outputs a authenticity prediction result of the reconstructed image. The authenticity prediction result of the reconstructed image is used to characterize the probability that the reconstructed image is a real image. Therefore, the value of the authenticity prediction result can be greater than or equal to 0 and less than or equal to 1. In addition, the reconstructed image can be annotated to obtain the authenticity annotation result of the reconstructed image. The authenticity annotation result of the reconstructed image is used to characterize whether the reconstructed image is a real image. Therefore, the value of the authenticity annotation result can be 0 or 1. Among them, 0 represents that the reconstructed image is not a real image, and 1 represents that the reconstructed image is a real image.

[0131] The loss value between the authenticity labeling result of the reconstructed image and the authenticity prediction result of the reconstructed image can be determined based on the authenticity labeling result of the reconstructed image and the authenticity prediction result of the reconstructed image. The determination method is not limited here. In addition, a first loss value can be determined based on the reconstructed image and the sample image, and a second loss value can be determined based on at least two predicted images. Based on the loss value between the authenticity labeling result of the reconstructed image and the authenticity prediction result, the first loss value, and the second loss value, the loss value of the neural network model is calculated, and the neural network model is adjusted based on the loss value of the neural network model to obtain an image generation model.

[0132] It should be noted that when training the neural network model, the discriminator can be connected in series after the neural network model, and the neural network model and the discriminator can be trained multiple times using the reconstructed image output by the neural network model and the authenticity prediction results of the reconstructed image output by the discriminator until the training end conditions are met, and the trained neural network model is used as the image generation model.

[0133] Among them, when training the neural network model and the discriminator, on the one hand, the discriminator can be fixed, and the loss value of the neural network model can be determined based on the loss value, the first loss value, and the second loss value between the authenticity labeling result and the authenticity prediction result of the reconstructed image, and the neural network model is adjusted based on the loss value of the neural network model. At this time, the authenticity labeling result of the reconstructed image is 1. On the other hand, the neural network model can be fixed, and the loss value of the discriminator can be determined based on the same method as "determining the loss value of the neural network model" based on the loss value, the first loss value, and the second loss value between the authenticity labeling result and the authenticity prediction result of the reconstructed image, and the discriminator can be adjusted based on the loss value of the discriminator. At this time, the authenticity labeling result of the reconstructed image is 0. Based on the above two methods, the neural network model and the discriminator are trained multiple times until the training end conditions are met, and a trained neural network model and a trained discriminator are obtained. Among them, the trained neural network model is the image generation model of the embodiment of the present application.

[0134] In one possible implementation, the neural network model is adjusted according to at least two predicted images, the reconstructed image and the sample image to obtain the image generation model, which also includes: obtaining the authenticity labeling result of the reconstructed image and the authenticity labeling result of each predicted image; obtaining the authenticity prediction result of the reconstructed image and the authenticity prediction result of each predicted image; adjusting the neural network model according to at least two predicted images, the reconstructed image and the sample image to obtain the image generation model, including: adjusting the neural network model according to the authenticity labeling result and the authenticity prediction result of the reconstructed image, the authenticity labeling result and the authenticity prediction result of each predicted image, the reconstructed image, the sample image and at least two predicted images to obtain the image generation model.

[0135] In an embodiment of the present application, a discriminator can be used to determine the authenticity prediction results of each predicted image and the authenticity prediction results of the reconstructed image. In addition, the authenticity annotation results of each predicted image and the authenticity annotation results of the reconstructed image can also be obtained by annotation. According to the loss value between the authenticity annotation result and the authenticity prediction result of the reconstructed image, the loss value between the authenticity annotation result and the authenticity prediction result of each predicted image, the loss value between every two predicted images, and the loss value between the reconstructed image and the sample image (i.e., the first loss value), the loss value of the neural network model is determined to adjust the neural network model to obtain an image generation model.

[0136] It should be noted that the loss value of the neural network model in the embodiment of the present application is calculated based on at least one of the following loss values: the loss value between the sample image and each predicted image, the loss value between every two predicted images (a second loss value can be further calculated), the loss value between the image category of the sample image and the image category of each predicted image, the loss value between the authenticity labeling result of each predicted image and the authenticity prediction result, the loss value between the reconstructed image and the sample image (that is, the first loss value mentioned above), the loss value between the image category of the sample image and the image category of the reconstructed image, and the loss value between the authenticity labeling result of the reconstructed image and the authenticity prediction result.

[0137] It can be understood that when the loss value of the neural network model is calculated based on at least one or two of the following: the loss value between the authenticity labeling results and the authenticity prediction results of each predicted image, and the loss value between the authenticity labeling results and the authenticity prediction results of the reconstructed image. On the one hand, the discriminator can be fixed, and the neural network model can be adjusted based on the loss value of the neural network model. At this time, the authenticity labeling results of the reconstructed image and the predicted image are both 1. On the other hand, the neural network model can be fixed, and the loss value of the discriminator can be determined based on the same method as "determining the loss value of the neural network model". The discriminator can be adjusted based on the loss value of the discriminator. At this time, the authenticity labeling results of the reconstructed image and the predicted image are both 0. Based on the above two methods, the neural network model and the discriminator are trained multiple times until the training end conditions are met.

[0138] It should be noted that the neural network model includes a memory feature set. During the training process of the neural network model, the memory feature set is continuously updated, so that the neural network model continuously learns features that cannot be used to identify image categories, thereby improving the representation ability of each memory feature in the memory feature set.

[0139] The above method is to extract a first sample feature from a sample image, fuse the first sample feature with the first memory feature, and generate a predicted image based on the fusion result. Since the first sample feature is related to the image category of the sample image, and the first memory feature is a feature learned by the neural network model and is unrelated to the image category, the predicted image combines information unrelated to the image category learned by the model while ensuring that the image category is the same as the image category of the sample image, so that the content of the predicted image has random diversity, that is, the content of the image generated by the model has random diversity. In the process of adjusting the neural network model according to at least two predicted images to obtain an image generation model, the model continuously learns features unrelated to the image category, that is, continuously enriches features unrelated to the image category, further improving the content diversity of the predicted image.

[0140] Based on the above implementation environment, the embodiment of the present application provides an image generation method. Figure 3 As an example, the flowchart of an image generation method provided in the embodiment of the present application is shown. The method can be performed by Figure 1 The terminal device 101 or server 102 in the embodiment of the present application can also execute the training method of the image generation model. For the convenience of description, the terminal device 101 or server 102 that executes the training method of the image generation model in the embodiment of the present application is referred to as an electronic device, and the method can be executed by an electronic device. Figure 3 As shown, the method includes steps 301 to 304.

[0141] Step 301: Obtain a reference image and an image generation model.

[0142] Among them, the image generation model is based on Figure 2 The image generation model is trained using the training method shown in FIG.

[0143] The embodiments of this application do not limit the method, content, or number of reference images. For example, the reference image is any frame extracted from a video, or a user-captured image, or an image captured from the internet, or a generated image. The number of reference images is at least one. The reference image can be a photographic image of an animal, a building, a road, or a scanned image.

[0144] In the embodiment of the present application, the image generation model is obtained by training the neural network model. Therefore, the model structure, size, etc. of the image generation model are consistent with the model structure, size, etc. of the neural network model.

[0145] Step 302 : extracting target features from the reference image according to the image generation model, where the target features are related to the image category of the reference image.

[0146] For the description of step 302 , please refer to the description of step 202 above, and the implementation principles of the two are similar.

[0147] Optionally, the image generation model includes a first encoder, and the first encoder included in the image generation model is trained by the first encoder included in the neural network model. After the reference image is input into the image generation model, the first encoder extracts features from the reference image to obtain target features.

[0148] The target feature includes, but is not limited to, at least one of a contour feature of an object in the reference image (used to describe the contour of the object in the reference image and can be used to identify the image category of the reference image), a texture feature of an object in the reference image (used to describe the texture of the object in the reference image and can be used to identify the image category of the reference image), etc. The target feature can be referred to as a feature related to the image category of the reference image and can be used to identify the image category of the reference image.

[0149] Step 303: Select a target memory feature from the memory feature set of the image generation model. The target memory feature is a feature learned by the image generation model and is independent of the image category.

[0150] For the description of step 303 , please refer to the description of step 203 above, and the implementation principles of the two are similar.

[0151] In an embodiment of the present application, the image generation model stores a memory feature set, which includes at least one memory feature. The memory feature set stored by the image generation model is trained using the memory feature set stored by the neural network model. For each memory feature, the memory feature is learned by the image generation model from multiple sample images, is shared by at least one sample image, and is independent of the image category.

[0152] Optionally, the number of target memory features is at least one. At least one target memory feature can be randomly selected from the memory feature set of the image generation model.

[0153] Step 304: fuse the target features and the target memory features according to the image generation model, and generate a target image based on the fusion result.

[0154] In an embodiment of the present application, the image generation model also includes a first generator, and the first generator included in the image generation model is trained by the first generator included in the neural network model. When the number of target memory features is one, the first generator can generate a target image based on the target feature and the target memory feature. When the target memory features are at least two, the first generator can interpolate the at least two target memory features to obtain an interpolation feature, and generate a target image based on the target feature and the interpolation feature. For the description of step 304, please refer to the relevant description of step 203 above, and the implementation principles of the two are similar.

[0155] The above method extracts target features from a reference image and generates a target image based on the target features and target memory features. Since the target features are related to the image category of the reference image, and the target memory features are features learned by the image generation model and are independent of the image category, the target image, while ensuring that the image category is the same as that of the reference image, combines the information learned by the model that is independent of the image category, so that the content of the target image has random diversity, that is, the content of the image generated by the model has random diversity.

[0156] The above is a detailed description of the training method and image generation method of the image generation model of the embodiment of the present application from the perspective of method steps. Figure 4 To explain. Figure 4 This is a schematic diagram of a training method for an image generation model provided in an embodiment of the present application.

[0157] In an embodiment of the present application, a neural network model is trained to obtain an image generation model. The neural network model includes a first encoder, a second encoder, an addresser, a first generator, a second generator, and a memory feature set. A sample image is input into the neural network model, and the first encoder extracts a first sample feature from the sample image, while the second encoder extracts a second sample feature from the sample image.

[0158] On the one hand, at least one first memory feature can be randomly selected from the memory feature set (wherein, Figure 4 At least three first memory features are randomly selected from the memory feature set), the first sample feature and at least one first memory feature are spliced ​​to obtain a spliced ​​feature, and the first generator generates at least two predicted images based on the spliced ​​feature (wherein, Figure 4 shows the generation of two predicted images using the first generator).

[0159] On the other hand, the addresser encodes the second sample feature to obtain encoded information. The encoded information is multiplied by the memory feature set to obtain a second memory feature. The first sample feature and the second memory feature are concatenated to obtain a concatenated feature, and the second generator generates a reconstructed image based on the concatenated feature.

[0160] Afterwards, a first loss value is determined based on the reconstructed image and the sample image, a second loss value is determined based on at least two predicted images, and the neural network model is adjusted based on the first loss value and the second loss value to obtain an image generation model.

[0161] As mentioned above, the memory feature set includes at least one memory feature. Any memory feature is a feature shared by at least one sample image in the sample image set learned by the neural network model, and the memory feature is independent of the image category. Figure 5 , Figure 5 This is a schematic diagram of generating a predicted image provided in an embodiment of the present application.

[0162] In an embodiment of the present application, the sample image is an image of a sitting panda. By continuously training the neural network model, the neural network model can learn the common features of multiple animal images in a bamboo forest in the sample image set, thereby fusing the first sample feature of the sample image (related to the image category of the sample image) with the common features of multiple animal images in a bamboo forest, and obtaining an image of a panda in a bamboo forest based on the fusion result. Based on the same principle, by continuously training the neural network model, the neural network model can learn the common features of multiple standing animal images in the sample image set, thereby fusing the first sample feature of the sample image with the common features of multiple standing animal images, and obtaining an image of a standing panda based on the fusion result.

[0163] In the embodiment of the present application, two sample image sets are used, according to Figure 4 The neural network model is trained in the manner shown to obtain the image generation model of the embodiment of the present application, which is denoted as image generation model 1. The two sample image sets can also be used to train the neural network model according to the training method of the image generation model of the related art to obtain four image generation models, which are denoted as image generation models 2 to 5. Among them, the two sample image sets are an animal image set and a bird image set. The images corresponding to the 119 image categories of the animal image set constitute a sample image set, which is used to train the neural network model, and the images corresponding to the other 30 image categories constitute a reference image set, which is used to test the image generation model. The images corresponding to 3 / 4 of the image categories of the bird image set constitute a sample image set, which is used to train the neural network model, and the images corresponding to the remaining 1 / 4 of the image categories constitute a reference image set, which is used to test the image generation model.

[0164] Optionally, when testing the image generation model, each image in the reference image set is used as a reference image, and image generation models 1-5 generate a target image based on the reference image. The Fréchet Inception Distance (FID) between the target image and the reference image is then calculated, resulting in Table 1.

[0165] Table 1

[0166]

[0167] It can be seen from Table 1 that the FID corresponding to image generation model 1 is significantly smaller than the FIDs corresponding to image generation models 2 to 5, indicating that the target image generated by image generation model 1 is significantly different from the reference image.

[0168] See Figure 6 , Figure 6 Schematic diagram of a target image generated by a different image generation model provided in an embodiment of the present application. Figure 6 It can be seen that the difference between the target image generated by image generation model 1 and the reference image is large, while the difference between the target images generated by image generation models 3 and 4 and the reference image is small. The content of the target image generated by image generation model 1 has random diversity, while the content of the target images generated by image generation models 3 and 4 is relatively simple.

[0169] In the embodiments of the present application, target images generated by different image generation models can be used as sample images for the image classification model. The image classification model is trained using the sample images, and the accuracy of the image classification model is tested, resulting in the following Table 2. Here, the accuracy is A±B, where A and B are both positive numbers, A represents the average accuracy, and B represents the variance of the accuracy.

[0170] Table 2

[0171] Image classification models Image generation model Accuracy Image Classification Model 1 - 77.93±0.67 Image Classification Model 1 Image Generation Model 4 76.28±0.63 Image Classification Model 1 Image Generation Model 6 78.72±0.64 Image Classification Model 1 Image Generation Model 7 77.94±0.67 Image Classification Model 1 Image Generation Model 1 79.02±0.61 Image Classification Model 2 Image Generation Model 4 79.21±0.63 Image Classification Model 2 Image Generation Model 4+6 79.52±0.62 Image Classification Model 2 Image Generation Model 1 82.98±0.60

[0172] It can be seen from Table 2 that, after the target image generated by the image generation model 1 is used as the sample image of the image classification model 1 and the image classification model 1 is trained using the sample image, the average accuracy of the image classification model 1 is significantly higher than the average accuracy corresponding to other image classification models 1, and the accuracy variance of the image classification model 1 is significantly lower than the accuracy variance corresponding to other image classification models 1. Similarly, after the target image generated by the image generation model 1 is used as the sample image of the image classification model 2 and the image classification model 2 is trained using the sample image, the average accuracy of the image classification model 2 is significantly higher than the average accuracy corresponding to other image classification models 2, and the accuracy variance of the image classification model 2 is significantly lower than the accuracy variance corresponding to other image classification models 2. It can be concluded that, using the target image generated by the image generation model of the embodiment of the present application, an image after data enhancement processing can be seen. Using the image after data enhancement processing to train other image processing models (such as image classification models) can improve the accuracy of the image processing model.

[0173] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the reference images and sample images involved in this application were obtained with full authorization.

[0174] Figure 7 FIG. 1 is a schematic diagram of a structure of a training device for an image generation model provided in an embodiment of the present application. Figure 7 As shown, the device includes:

[0175] An acquisition module 701 is used to acquire a sample image and a neural network model;

[0176] An extraction module 702 is configured to extract a first sample feature from the sample image based on a neural network model, where the first sample feature is related to an image category of the sample image;

[0177] A selection and generation module 703 is configured to select a first memory feature from the memory feature set of the neural network model, fuse the first sample feature with the first memory feature according to the neural network model, and generate any predicted image based on the fusion result, wherein the first memory feature is a feature learned by the neural network model and is unrelated to the image category;

[0178] The adjustment module 704 is used to adjust the neural network model according to at least two predicted images to obtain an image generation model.

[0179] In one possible implementation, the first memory characteristics are at least two;

[0180] A generation module 703 is selected to perform interpolation processing on at least two first memory features to obtain interpolation features; fuse the first sample features and the interpolation features, and generate any predicted image based on the fusion result.

[0181] In a possible implementation, the acquisition module 701 is further configured to acquire the image category of the sample image and the image category of each predicted image;

[0182] The adjustment module 704 is configured to adjust the neural network model according to the at least two predicted images, the image category of the sample image, and the image category of each predicted image to obtain an image generation model.

[0183] In a possible implementation, the acquisition module 701 is further configured to acquire authenticity labeling results and authenticity prediction results of each predicted image;

[0184] The adjustment module 704 is used to adjust the neural network model according to the at least two predicted images, the authenticity labeling results of each predicted image, and the authenticity prediction results to obtain an image generation model.

[0185] In a possible implementation, the extraction module 702 is further configured to extract a second sample feature from the sample image according to the neural network model, where the second sample feature is irrelevant to the image category of the sample image.

[0186] The selection generating module 703 is further configured to generate a reconstructed image based on the first sample feature and the second sample feature;

[0187] The adjustment module 704 is used to adjust the neural network model according to at least two predicted images, the reconstructed image and the sample image to obtain an image generation model.

[0188] In one possible implementation, a generation module 703 is selected to encode the second sample feature to obtain encoding information; based on the encoding information, a second memory feature is selected from the memory feature set of the neural network model, where the second memory feature is a feature learned by the neural network model, and the second memory feature is independent of the image category and is related to the second sample feature; based on the second memory feature and the first sample feature, a reconstructed image is generated.

[0189] In one possible implementation, the adjustment module 704 is used to determine a first loss value based on the reconstructed image and the sample image; determine a second loss value based on at least two predicted images; and adjust the neural network model based on the first loss value and the second loss value to obtain an image generation model.

[0190] In a possible implementation, the acquisition module 701 is further configured to acquire the authenticity annotation result and the authenticity prediction result of the reconstructed image, and the authenticity annotation result and the authenticity prediction result of each predicted image;

[0191] The adjustment module 704 is used to adjust the neural network model according to the authenticity labeling result and authenticity prediction result of the reconstructed image, the authenticity labeling result and authenticity prediction result of each predicted image, the reconstructed image, the sample image and at least two predicted images to obtain an image generation model.

[0192] The above-mentioned device extracts a first sample feature from a sample image, fuses the first sample feature with the first memory feature, and generates a predicted image based on the fusion result. Since the first sample feature is related to the image category of the sample image, and the first memory feature is a feature learned by the neural network model and is unrelated to the image category, the predicted image combines information unrelated to the image category learned by the model while ensuring that the image category is the same as the image category of the sample image, so that the content of the predicted image has random diversity, that is, the content of the image generated by the model has random diversity. In the process of adjusting the neural network model according to at least two predicted images to obtain an image generation model, the model continuously learns features unrelated to the image category, that is, continuously enriches features unrelated to the image category, further improving the content diversity of the predicted image.

[0193] It should be understood that the above Figure 7 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0194] Figure 8 FIG. 1 is a schematic diagram of the structure of an image generating device provided in an embodiment of the present application. Figure 8 As shown, the device includes:

[0195] An acquisition module 801 is configured to acquire a reference image and an image generation model, where the image generation model is trained according to any of the above-mentioned image generation model training methods;

[0196] An extraction module 802 is configured to extract target features from a reference image based on an image generation model, where the target features are related to an image category of the reference image;

[0197] A selection module 803 is configured to select a target memory feature from the memory feature set of the image generation model, where the target memory feature is a feature learned by the image generation model and is unrelated to the image category;

[0198] The generation module 804 is used to fuse the target features and the target memory features according to the image generation model, and generate a target image based on the fusion result.

[0199] The above-mentioned device extracts target features from a reference image and generates a target image based on the target features and target memory features. Since the target features are related to the image category of the reference image, and the target memory features are features learned by the image generation model and are independent of the image category, the target image, while ensuring that the image category is the same as that of the reference image, combines information learned by the model that is independent of the image category, thereby making the content of the target image have random diversity, that is, the content of the image generated by the model has random diversity.

[0200] It should be understood that the above Figure 8 The provided device is illustrated only by the division of the above-mentioned functional modules when implementing its functions. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0201] Figure 9 FIG2 shows a block diagram of a terminal device 900 provided by an exemplary embodiment of the present application. The terminal device 900 includes a processor 901 and a memory 902 .

[0202] The processor 901 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0203] The memory 902 may include one or more computer-readable storage media, which may be non-transitory. The memory 902 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 902 is used to store at least one computer program, which is used to be executed by the processor 901 to implement the training method or image generation method of the image generation model provided in the method embodiment of the present application.

[0204] In some embodiments, the terminal device 900 may optionally include a peripheral device interface 903 and at least one peripheral device. The processor 901, memory 902, and peripheral device interface 903 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, and a power supply 908.

[0205] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0206] The RF circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 904 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 904 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the RF circuit 904 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The RF circuit 904 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 904 may also include circuits related to Near Field Communication (NFC), which is not limited in this application.

[0207] The display screen 905 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 905 is a touch screen, it is also capable of collecting touch signals on or above the surface of the display screen 905. These touch signals can be input as control signals to the processor 901 for processing. In this case, the display screen 905 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 905, located on the front panel of the terminal device 900. In other embodiments, there can be at least two display screens 905, located on different surfaces of the terminal device 900 or in a foldable design. In other embodiments, the display screen 905 can be a flexible display, located on a curved or foldable surface of the terminal device 900. Furthermore, the display screen 905 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 905 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0208] The camera assembly 906 is used to capture images or videos. Optionally, the camera assembly 906 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 906 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0209] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 901 for processing, or input into the radio frequency circuit 904 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there can be multiple microphones, which are respectively arranged in different parts of the terminal device 900. The microphone can also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The speaker can be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signals into sound waves audible to humans, but also convert the electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 907 may also include a headphone jack.

[0210] The power supply 908 is used to power the various components in the terminal device 900. The power supply 908 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When the power supply 908 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0211] In some embodiments, the terminal device 900 further includes one or more sensors 909 , including but not limited to: an acceleration sensor 911 , a gyroscope sensor 912 , a pressure sensor 913 , an optical sensor 914 , and a proximity sensor 915 .

[0212] The accelerometer 911 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal device 900. For example, the accelerometer 911 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 901 can control the display screen 905 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 911. The accelerometer 911 can also be used to collect game or user motion data.

[0213] The gyroscope sensor 912 can detect the body orientation and rotation angle of the terminal device 900. The gyroscope sensor 912 can work with the acceleration sensor 911 to collect the user's 3D movements of the terminal device 900. Based on the data collected by the gyroscope sensor 912, the processor 901 can implement the following functions: motion sensing (such as changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0214] The pressure sensor 913 can be set on the side frame of the terminal device 900 and / or the lower layer of the display screen 905. When the pressure sensor 913 is set on the side frame of the terminal device 900, it can detect the user's grip signal of the terminal device 900, and the processor 901 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 913. When the pressure sensor 913 is set on the lower layer of the display screen 905, the processor 901 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 905. The operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0215] Optical sensor 914 is used to detect ambient light intensity. In one embodiment, processor 901 can control the display brightness of display screen 905 based on the ambient light intensity detected by optical sensor 914. Specifically, when the ambient light intensity is high, the display brightness of display screen 905 is increased; when the ambient light intensity is low, the display brightness of display screen 905 is decreased. In another embodiment, processor 901 can also dynamically adjust the shooting parameters of camera assembly 906 based on the ambient light intensity detected by optical sensor 914.

[0216] The proximity sensor 915, also known as a distance sensor, is typically located on the front panel of the terminal device 900. The proximity sensor 915 is used to detect the distance between the user and the front of the terminal device 900. In one embodiment, when the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually decreasing, the processor 901 controls the display screen 905 to switch from the screen-on state to the screen-off state. When the proximity sensor 915 detects that the distance between the user and the front of the terminal device 900 is gradually increasing, the processor 901 controls the display screen 905 to switch from the screen-off state to the screen-on state.

[0217] Those skilled in the art will understand that Figure 9 The structure shown in the figure does not constitute a limitation on the terminal device 900, and the terminal device 900 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0218] Figure 10This is a schematic diagram of the structure of the server provided in the embodiment of the present application. The server 1000 may have relatively large differences due to different configurations or performances, and may include one or more processors 1001 and one or more memories 1002, wherein the one or more memories 1002 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1001 to implement the image generation model training method or image generation method provided in the above-mentioned various method embodiments. Exemplarily, the processor 1001 is a CPU. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1000 may also include other components for implementing device functions, which will not be described in detail here.

[0219] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image generation model training methods or image generation methods.

[0220] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0221] In an exemplary embodiment, a computer program or a computer program product is also provided, in which at least one computer program is stored, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement any of the above-mentioned image generation model training methods or image generation methods.

[0222] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0223] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0224] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an image generation model, characterized in that: The method comprises: Get sample images and neural network models; extracting a first sample feature from the sample image according to the neural network model, where the first sample feature is related to an image category of the sample image; Selecting a first memory feature from a set of memory features included in the neural network model, fusing the first sample feature and the first memory feature according to the neural network model, and generating any predicted image based on the fusion result, wherein the first memory feature is a feature common to images of different image categories learned by the neural network model and is unrelated to the image category; extracting a second sample feature from the sample image according to the neural network model, where the second sample feature is irrelevant to the image category of the sample image; Normalizing the second sample feature to obtain a normalized second sample feature, randomly sampling the second sample feature to obtain a random variable, and determining encoding information based on the normalized second sample feature and the random variable; Based on the encoding information, select a second memory feature from the memory feature set, where the second memory feature is a feature learned by the neural network model, and the second memory feature is independent of the image category and is related to the second sample feature; generating a reconstructed image based on the second memory feature and the first sample feature; The neural network model is adjusted according to at least two predicted images, the reconstructed image and the sample image to obtain an image generation model, wherein the image generation model includes an adjusted memory feature set.

2. The method according to claim 1, characterized in that The first memory characteristics are at least two; The fusing the first sample feature and the first memory feature, and generating any predicted image based on the fusion result, includes: Performing interpolation processing on at least two first memory features to obtain an interpolation feature; The first sample feature and the interpolation feature are fused, and any one of the predicted images is generated based on the fusion result.

3. The method according to claim 1, characterized in that Before adjusting the neural network model according to the at least two predicted images to obtain the image generation model, the method further includes: Obtaining the image category of the sample image and the image category of each predicted image; The step of adjusting the neural network model according to the at least two predicted images to obtain an image generation model includes: The neural network model is adjusted according to at least two predicted images, the image category of the sample image, and the image category of each predicted image to obtain an image generation model.

4. The method according to claim 1, wherein Before adjusting the neural network model according to the at least two predicted images, the reconstructed image and the sample image to obtain the image generation model, the method further includes: Obtain the authenticity annotation results and authenticity prediction results of each predicted image; The step of adjusting the neural network model according to the at least two predicted images, the reconstructed image, and the sample image to obtain an image generation model includes: The neural network model is adjusted according to at least two predicted images, the reconstructed image, the sample image, the authenticity labeling results of each predicted image, and the authenticity prediction results to obtain an image generation model.

5. The method according to claim 1, wherein The step of adjusting the neural network model according to the at least two predicted images, the reconstructed image, and the sample image to obtain an image generation model includes: determining a first loss value according to the reconstructed image and the sample image; determining a second loss value based on the at least two predicted images; Based on the first loss value and the second loss value, the neural network model is adjusted to obtain an image generation model.

6. The method according to claim 1, characterized in that Before adjusting the neural network model according to the at least two predicted images, the reconstructed image and the sample image to obtain the image generation model, the method further includes: Obtaining the authenticity annotation result and authenticity prediction result of the reconstructed image, and the authenticity annotation result and authenticity prediction result of each predicted image; The step of adjusting the neural network model according to the at least two predicted images, the reconstructed image, and the sample image to obtain an image generation model includes: The neural network model is adjusted according to the authenticity labeling result and the authenticity prediction result of the reconstructed image, the authenticity labeling result and the authenticity prediction result of each predicted image, the reconstructed image, the sample image and the at least two predicted images to obtain an image generation model.

7. An image generation method, characterized in that: The method comprises: Acquire a reference image and an image generation model, wherein the image generation model is trained according to the image generation model training method according to any one of claims 1 to 6; extracting target features from the reference image according to the image generation model, wherein the target features are related to the image category of the reference image; Selecting a target memory feature from the memory feature set of the image generation model, wherein the target memory feature is a feature learned by the image generation model and is independent of the image category; The target feature and the target memory feature are fused according to the image generation model, and a target image is generated based on the fusion result.

8. A training device for an image generation model, characterized in that: The device comprises: Acquisition module, used to obtain sample images and neural network models; an extraction module, configured to extract a first sample feature from the sample image according to the neural network model, where the first sample feature is related to an image category of the sample image; a selection and generation module, configured to select a first memory feature from a set of memory features included in the neural network model, fuse the first sample feature and the first memory feature according to the neural network model, and generate any predicted image based on the fusion result, wherein the first memory feature is a feature common to images of different image categories learned by the neural network model and is independent of the image category; The extraction module is further configured to extract a second sample feature from the sample image according to the neural network model, where the second sample feature is irrelevant to the image category of the sample image; The selection and generation module is further configured to perform normalization processing on the second sample feature to obtain a normalized second sample feature, perform random sampling on the second sample feature to obtain a random variable, and determine encoding information based on the normalized second sample feature and the random variable; select a second memory feature from a memory feature set of the neural network model based on the encoding information, where the second memory feature is a feature learned by the neural network model, the second memory feature is unrelated to the image category, and is related to the second sample feature; and generate a reconstructed image based on the second memory feature and the first sample feature; An adjustment module is used to adjust the neural network model according to at least two predicted images, the reconstructed image and the sample image to obtain an image generation model, wherein the image generation model includes an adjusted memory feature set.

9. The device according to claim 8, characterized in that The first memory characteristics are at least two; The selection and generation module is configured to perform interpolation processing on at least two first memory features to obtain interpolation features; fuse the first sample features and the interpolation features, and generate any one of the predicted images based on the fusion result.

10. The device according to claim 8, characterized in that The acquisition module is further configured to acquire the image category of the sample image and the image category of each predicted image; The adjustment module is used to adjust the neural network model according to at least two predicted images, the image category of the sample image and the image category of each predicted image to obtain an image generation model.

11. The device according to claim 8, characterized in that The acquisition module is further used to obtain the authenticity labeling results and authenticity prediction results of each predicted image; The adjustment module is used to adjust the neural network model according to at least two predicted images, the reconstructed image, the sample image, the authenticity labeling results of each predicted image and the authenticity prediction results to obtain an image generation model.

12. The device according to claim 8, characterized in that The adjustment module is used to determine a first loss value based on the reconstructed image and the sample image; determine a second loss value based on the at least two predicted images; and adjust the neural network model based on the first loss value and the second loss value to obtain an image generation model.

13. The device according to claim 8, characterized in that The acquisition module is further configured to acquire the authenticity annotation result and authenticity prediction result of the reconstructed image, and the authenticity annotation result and authenticity prediction result of each predicted image; The adjustment module is used to adjust the neural network model according to the authenticity labeling result and authenticity prediction result of the reconstructed image, the authenticity labeling results and authenticity prediction results of each predicted image, the reconstructed image, the sample image and the at least two predicted images to obtain an image generation model.

14. An image generating device, characterized in that: The device comprises: an acquisition module, configured to acquire a reference image and an image generation model, wherein the image generation model is trained according to the image generation model training method according to any one of claims 1 to 6; an extraction module, configured to extract target features from the reference image according to the image generation model, wherein the target features are related to the image category of the reference image; a selection module, configured to select a target memory feature from the memory feature set of the image generation model, wherein the target memory feature is a feature learned by the image generation model and is independent of the image category; A generation module is used to fuse the target feature and the target memory feature according to the image generation model, and generate a target image based on the fusion result.

15. An electronic device, characterized in that: The electronic device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the training method of the image generation model as described in any one of claims 1 to 6 or implements the image generation method as described in claim 7.

16. A computer-readable storage medium, characterized in that At least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by the processor to enable the electronic device to implement the training method of the image generation model as described in any one of claims 1 to 6 or the image generation method as described in claim 7.

17. A computer program product, characterized in that The computer program product stores at least one computer program, which is loaded and executed by a processor so that the electronic device implements the training method of the image generation model as described in any one of claims 1 to 6 or the image generation method as described in claim 7.

Images