Image generation method, device, computer equipment and storage medium

Abstract

The present application relates to an image generation method, device, computer equipment and storage medium. The method comprises: extracting input image features of an input source image through an image generation model; obtaining a generated image according to the input image features and random noise parameters using the image generation model; wherein the image generation model is a neural network model obtained by training to maximize the generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value of the generated image. The use of this method can make the difference between the generated images large, and improve the diversity between the generated images.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image generation method, apparatus, computer equipment and storage medium. Background Art

[0002] With the development of image processing technology, there has emerged a technology that generates new images based on input images through neural networks. This technology is increasingly being used in people's daily lives, such as disguising people's appearance, predicting the appearance of a couple's offspring, and so on.

[0003] In traditional technology, image generation inputs two images, one as the source image and the other as the reference image. After the two images pass through different encoders, the features encoded by the encoders are fused and passed through a deconvolution network to generate an image of the same category as the reference image.

[0004] However, the images generated by traditional technologies have little visual difference, and the generated image patterns are single, with only one or several specific patterns, and the image generation does not have diversity. Summary of the invention

[0005] Based on this, it is necessary to provide an image generation method, apparatus, computer device and storage medium to address the above technical problems.

[0006] A method for generating an image, the method comprising:

[0007] Extracting input image features of an input source image through an image generation model;

[0008] The image generation model is used to obtain a generated image according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize a generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images.

[0009] In one embodiment, the image generation model includes a discriminant sub-model and a generative sub-model, and before obtaining the generated image according to the input image features and random noise parameters using the image generation model, it includes:

[0010] Obtain at least two training generated images of the training source image through the initial generation sub-model;

[0011] The initial discriminant sub-model and the initial generative sub-model are trained according to the training generated image and the target image to obtain the discriminant sub-model and the generative sub-model; wherein the initial discriminant sub-model is used to judge whether the input image is a true image according to the true image probability of the input image being a true image; and the target image is an image of the same type as the generated image.

[0012] In one embodiment, training an initial discriminant sub-model and an initial generative sub-model according to the training generated image and the target image to obtain the discriminant sub-model and the generative sub-model includes:

[0013] The initial discriminant sub-model is trained according to the probability difference to obtain an intermediate discriminant sub-model; wherein the probability difference is obtained according to the generation probability that the training generated image is a true image and the target probability that the target image is a true image;

[0014] The initial generation sub-model is trained according to the generation loss function and the generation probability to obtain an intermediate generation sub-model;

[0015] When the probability difference is less than the preset difference, the generated loss function is greater than the preset loss value and the generated probability difference is greater than the preset probability difference, the intermediate discriminant sub-model is used as the discriminant sub-model, and the intermediate generated sub-model is used as the generated sub-model; wherein the generated probability difference is obtained according to the difference between the generated probability and the standard probability of the false image.

[0016] In one embodiment, the training of the initial discriminant sub-model according to the probability difference to obtain the intermediate discriminant sub-model includes:

[0017] Inputting the training generated image and the target image into the initial discriminant sub-model to obtain the generation probability and the target probability;

[0018] Obtaining an average value of the difference between the generation probability and the false image standard probability as the generation probability difference;

[0019] Obtaining an average value of the difference between the target probability and the true image standard probability as the target probability difference;

[0020] Obtaining an average value of the probability difference between the generated probability difference and the target probability difference to obtain the probability difference value;

[0021] The discriminant parameters in the initial discriminant sub-model are adjusted in a direction in which the probability difference is reduced to obtain the intermediate discriminant sub-model.

[0022] In one embodiment, the training of the initial generation sub-model according to the generation loss function and the generation probability to obtain an intermediate generation sub-model includes:

[0023] Extracting image features of the training generated image to obtain training feature values;

[0024] Obtaining a training image content difference value between the training generated images according to a second norm of a square difference of two training feature values;

[0025] Using a Gram matrix to obtain a training image style difference value between the training generated images;

[0026] Obtaining the sum of the training image content difference value and the training image style difference value to obtain an image difference value;

[0027] The image difference value is used as the generation loss function, and the generation parameters in the initial generation sub-model are adjusted in a direction in which the generation loss function and the generation probability difference increase, so as to obtain the intermediate generation sub-model.

[0028] In one embodiment, the step of obtaining the sum of the training image content difference value and the training image style difference value to obtain the image difference value includes:

[0029] Multiplying the image content weight by the initial training image content difference value to obtain the training image content difference value;

[0030] Multiplying the image style weight by the initial training image style difference value to obtain the training image style difference value;

[0031] The training image content difference value is added to the training image style difference value to obtain the image difference value.

[0032] In one embodiment, the step of using a Gram matrix to obtain a training image style difference value between the training generated images includes:

[0033] Obtaining the product of training feature values ​​at different dimensions of the training generated image to obtain a training feature product;

[0034] Obtain the corresponding Gram matrix according to the training feature product of each training generated image;

[0035] The training image style difference values ​​between the training images are obtained according to the norm values ​​of the Gram matrix square differences between the training generated images.

[0036] An image generating device, the device comprising:

[0037] A feature extraction module, for extracting input image features of an input source image through an image generation model;

[0038] A target generation module is used to use the image generation model to obtain a generated image according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize the generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images.

[0039] A computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the following steps are implemented:

[0040] Extracting input image features of an input source image through an image generation model;

[0041] The image generation model is used to obtain a generated image according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize a generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0043] Extracting input image features of an input source image through an image generation model;

[0044] The image generation model is used to obtain a generated image according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize a generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images.

[0045] The above-mentioned image generation method, device, computer equipment and storage medium use the sum of the image content difference value and the image style difference value between the generated images as the generation loss function, and the image generation model obtained by maximizing the generation loss function extracts the input image features of the input source image, and obtains the generated image based on the input image features and random noise parameters, thereby making the difference between the generated images large and improving the diversity between the generated images. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 is a schematic flow chart of an image generating method in one embodiment;

[0047] Figure 2 A schematic diagram of a process flow of a training process of an image generation model in one embodiment;

[0048] Figure 3 A schematic diagram of a flow chart of a training process of an image generation model in another embodiment;

[0049] Figure 4 A schematic diagram of a process of training an intermediate discriminant sub-model in one embodiment;

[0050] Figure 5 A schematic diagram of a process of training an intermediate generation sub-model in one embodiment;

[0051] Figure 6 A schematic diagram of a process for obtaining a style difference value of a training image in one embodiment;

[0052] Figure 7 A schematic diagram of a process for obtaining an image difference value in one embodiment;

[0053] Figure 8 A schematic diagram of a flow chart of a training process of an image generation model in another embodiment;

[0054] Fig. 9 is a structural block diagram of an image generating device in one embodiment;

[0055] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0057] In one embodiment, Figure 1 As shown, an image generation method is provided. This embodiment uses the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. The terminal can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, and portable wearable devices. The server can be implemented as an independent server or a server cluster composed of multiple servers. The method includes the following steps:

[0058] S110 , extracting input image features of the input source image through an image generation model.

[0059] The input source image is a base image for generating an image, that is, a new image is generated based on the input source image. In the task of generating a dog image from a cat image, the cat image is the input source image.

[0060] Specifically, the computer device receives the input source image, and extracts the image content features and image texture features of the input source image through a decoder in the image generation model as the input image features. The image content features are features used to reflect the image content and spatial structure, such as shape contour features, and the image style features are features used to reflect the overall style of the image, such as color and texture features.

[0061] S120: Utilize the image generation model to obtain a generated image according to the input image features and random noise parameters.

[0062] The image generation model includes a neural network model obtained by training a maximum generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images. During the training process of the image generation model, the sum of the image content difference value and the image style difference value between the images output each time is used as the generation loss function, and the generation loss function is trained to be maximum or greater than a preset loss value to obtain the image generation model.

[0063] Furthermore, the random noise parameter is vector data that obeys a normal distribution.

[0064] Specifically, the computer device uses the encoder in the image generation model to encode and fuse the acquired input image features with the random noise parameters to obtain the generated image. The input image features are encoded and fused with one of the random noise parameters to obtain one generated image, and the input image features are encoded and fused with two of the random noise parameters to obtain two generated images, that is, the number of the generated images obtained is the same as the number of the random noise parameters that are encoded and fused.

[0065] In this embodiment, the computer device uses the sum of the image content difference value and the image style difference value between the output images as the generation loss function, and uses the image generation model that makes the generation loss function maximum or greater than the preset loss value to extract the input image features of the input source image, and obtains the generated image based on the input image features and the random noise parameters, so that the difference between the generated images is large, thereby improving the diversity between the generated images.

[0066] In one embodiment, Figure 2 As shown, before S120, using the image generation model to obtain the generated image according to the input image features and the random noise parameters, a training process of the image generation model is also included, the image generation model includes a discriminant sub-model and a generative sub-model, and the training process includes the following steps:

[0067] S210, obtaining at least two training generated images of the training source image through the initial generation sub-model.

[0068] Specifically, the computer device obtains an image of the same image type as the input source image as the training source image. For example, if the input source image is a cat image, then the training source image is also a cat image. The computer device receives the training source image, and obtains the image content features and image texture features of the training source image through the decoder in the initial generation sub-model as the training image features. The computer device encodes and fuses the training image features and at least two random noise parameters respectively through the encoder in the initial generation sub-model to obtain at least two training generated images. For example, the training image features and the first random noise parameter are encoded and fused to obtain a first training generated image, and the training image features and the second random noise parameter are encoded and fused to obtain a second training generated image.

[0069] S220. Train the initial discriminant sub-model and the initial generative sub-model according to the training generated image and the target image to obtain the discriminant sub-model and the generative sub-model.

[0070] The initial discriminant submodel is used to determine whether the input image is a true image according to the true image probability of the input image being a true image. The true image described in this embodiment is a real photographed image, and the false image is a machine-generated image. In theory, the training source image and the target image are both true images, and the training generated images are all false images. The target image is an image of the same type as the generated image. For example, in the above-mentioned task of generating a dog image from a cat image, the dog image is the generated image, and the target image is an image of a dog.

[0071] Specifically, the computer device uses the training generated image and the target image to train the initial discriminant sub-model to obtain the discriminant sub-model, and trains the initial generative sub-model according to the true image probability of the training generated image output by the initial discriminant sub-model each time and the image content difference and image style difference between the training generated images to obtain the generative sub-model, thereby realizing the training of the image generation model.

[0072] In this embodiment, the computer device inputs the training source image into the initial generation sub-model to obtain the training generation image, and inputs the training generation image and the target image into the initial discrimination sub-model to obtain the probability that the training generation image is a true image and the probability that the target image is a true image. The computer device trains the initial discrimination sub-model according to the probability that the training generation image is a true image and the probability that the target image is a true image to obtain the discrimination sub-model, and trains the initial generation sub-model according to the image content difference and image style difference between the training generation images and the probability that the training generation image is a true image to obtain the generation sub-model. The above method is used to achieve simultaneous training of the two sub-models in the image generation model, thereby improving the training efficiency.

[0073] In one embodiment, Figure 3 As shown, the step S220, training the initial discriminant sub-model and the initial generative sub-model according to the training generated image and the target image to obtain the discriminant sub-model and the generative sub-model, includes:

[0074] S310 , training the initial discriminant sub-model according to the probability difference to obtain an intermediate discriminant sub-model.

[0075] The probability difference is obtained according to the generation probability that the training generated image is a true image and the target probability that the target image is a true image.

[0076] Furthermore, if Figure 4 As shown, S310 specifically includes:

[0077] S410, inputting the training generated image and the target image into the initial discriminant sub-model to obtain the generation probability and the target probability.

[0078] Specifically, the computer device inputs the obtained training generated images and the same number of the target images into the initial discriminant sub-model, and obtains the generation probability that the corresponding number of the training generated images are true images, and the target probability that the target images are true images. Taking two training generated images and two target images as an example, the two training generated images and the two target images are simultaneously input into the initial discriminant sub-model, and the generation probabilities of the training generated images being true images are 40% and 50%, and the target probabilities of the target images being true images are 80% and 70%, respectively.

[0079] S420, obtaining an average value of the difference between the generation probability and the false image standard probability as the generation probability difference.

[0080] S430, obtaining an average value of the difference between the target probability and the true image standard probability as the target probability difference.

[0081] S440: Obtain an average probability difference between the generated probability difference and the target probability difference to obtain the probability difference value.

[0082] Among them, the standard probability of the false image is 0, and the standard probability of the true image is 1.

[0083] Specifically, the computer device obtains the average value of the difference between each of the generated probabilities and the standard probability of the false image as the generated probability difference, that is, the average value of the above 40%-0 and 50%-0, and the generated probability difference is 45%. The computer device obtains the average value of the difference between each of the target probabilities and the standard probability of the true image as the target probability difference, that is, the average value of the above 80%-1 and 70%-1, and the target probability difference is 25%. The computer device obtains the average value of the probability difference between the generated probability difference and the target probability difference to obtain the probability difference value, that is, the average value of the above generated probability difference 45% and the target probability difference 25%, and the probability difference value is 35%.

[0084] S450, adjusting the discriminant parameters in the initial discriminant sub-model in a direction in which the probability difference is reduced, to obtain the intermediate discriminant sub-model.

[0085] Specifically, the computer device adjusts the discriminant parameters in the initial discriminant sub-model according to the probability difference, and adjusts the discriminant parameters in the initial discriminant sub-model in a direction in which the obtained probability difference increases. Each time the discriminant parameters are adjusted, the discriminant parameters are updated to obtain a new intermediate discriminant sub-model.

[0086] S320. Train the initial generation sub-model according to the generation loss function and the generation probability to obtain an intermediate generation sub-model.

[0087] Specifically, the computer device further extracts the training generated features of the training generated images to obtain training feature values, calculates the image content difference values ​​between the training generated images according to the difference between the training feature values ​​of the training generated images, and calculates the image style difference values ​​between the training generated images obtained in this training according to the difference between the Gram matrix values ​​of the training generated images obtained by the training feature values. The computer device uses the sum of the image content difference value and the image style difference value as the generation loss function, adjusts the generation parameters in the initial generation sub-model, increases the generation parameters in the initial generation sub-model toward the obtained generation loss function, and obtains the direction of increasing the generation probability difference according to the difference between the generation probability and the standard probability of the false image, and obtains an intermediate generation sub-model each time.

[0088] S330. When the probability difference is less than the preset difference, the generated loss function is greater than the preset loss value, and the generated probability difference is greater than the preset probability difference, the intermediate discriminant sub-model is used as the discriminant sub-model, and the intermediate generated sub-model is used as the generated sub-model.

[0089] Specifically, the computer device trains the discriminant parameters in the initial discriminant sub-model according to the probability difference, and trains the generation parameters in the initial generation sub-model according to the generation loss function and the generation probability, until the probability difference obtained by the training generated image is minimum or less than the preset difference, the generation loss function is maximum or greater than the preset loss value, and the generation probability difference is maximum or greater than the preset probability difference, the training is stopped, and the intermediate discriminant sub-model obtained at this time is used as the discriminant sub-model, and the intermediate generation sub-model is used as the generation sub-model.

[0090] Furthermore, the maximum value of the generated loss function can be determined by judging whether the inverse of the image difference value is less than or equal to a minimum threshold.

[0091] Specifically, the computer device can obtain the reciprocal of the image difference value obtained in each training, and determine whether the image difference value is the largest according to whether the reciprocal of the image difference value is less than or equal to a minimum threshold. When the reciprocal of the image difference value is infinitely close to 0, such as less than or equal to a minimum threshold of 0.001, it is determined that the image difference value is the largest to simplify the data.

[0092] In this embodiment, the computer device trains both the initial discriminant sub-model and the initial generative sub-model each time a training source image is input, and the discriminant parameters in the initial discriminant sub-model are adjusted in the direction of decreasing the probability difference, and the generative parameters in the initial generative sub-model are adjusted in the direction of increasing the generative loss function and increasing the generative probability difference. When the probability difference is less than the preset difference, the generative loss function is greater than the preset loss value, and the generative probability difference is greater than the preset probability difference, the adjustment is stopped to obtain the discriminant sub-model and the generative sub-model, and the discriminant sub-model and the generative sub-model constitute the image generative model, thereby realizing the training of the image generative model. In the above training, the process of adjusting the discriminant parameters in the initial discriminant sub-model in the direction of decreasing the probability difference enables the image generative model to generate an image of the same type as the target image, thereby improving the consistency with the target image; adjusting the generative parameters in the initial generative sub-model in the direction of increasing the generative loss function and increasing the generative probability difference improves the diversity between the generated images.

[0093] In one embodiment, Figure 5 As shown, the S320, training the initial generation sub-model according to the generation loss function and the generation probability to obtain an intermediate generation sub-model, includes:

[0094] S510: Extract image features of the training generated image to obtain training feature values.

[0095] S520: Obtain a training image content difference value between the training generated images according to a second norm of a square difference of two training feature values.

[0096] Specifically, the computer device uses the discriminator in the initial generation sub-model to extract the training feature value of the training generated image, and obtains the training image content difference value between the training generated images according to the content difference formula of the square difference of the training feature value. The content difference formula satisfies the following formula:

[0097]

[0098] Among them, C, H, W represent the dimensions of feature extraction by the discriminator, φ(y) represents the training feature value, and j represents the jth convolutional layer of the discriminator.

[0099] S530: Using a Gram matrix to obtain a training image style difference value between the training generated images.

[0100] The Gram matrix is ​​used to calculate the correlation between two features on the feature map extracted by the discriminator in the image generation model, reflecting the overall style of the image.

[0101] Specifically, the computer device uses the discriminator in the initial discriminant sub-model to obtain the training image style difference value between the training generated images according to the style difference formula of the Gram matrix square difference norm value between the training generated images.

[0102] Furthermore, if Figure 6 As shown, S530 specifically includes:

[0103] S610: Obtain the product of the training feature values ​​of the training generated image in different dimensions to obtain a training feature product.

[0104] S620: Obtain the corresponding Gram matrix according to the training feature product of each training generated image.

[0105] S630: Obtain training image style difference values ​​between the training images according to the norm values ​​of the Gram matrix square differences between the training generated images.

[0106] Specifically, the computer device uses the discriminator in the initial discriminant sub-model to obtain the training image style difference value between the training generated images according to the style difference formula. The style difference formula satisfies the following formula:

[0107]

[0108] Among them, C, H, and W represent the dimensions of feature extraction by the discriminator, G(y) represents the Gram matrix, F is the norm value, which is generally 1 or 2, φ(x) represents the training feature value, and j represents the jth convolutional layer of the discriminator.

[0109] S540: Obtain a sum of the training image content difference value and the training image style difference value to obtain an image difference value.

[0110] S550, taking the image difference value as the generation loss function, adjusting the generation parameters in the initial generation sub-model in a direction in which the generation loss function and the generation probability difference increase, to obtain the intermediate generation sub-model.

[0111] Specifically, the computer device obtains the sum of the image content difference value and the image style difference value to obtain the image difference value, uses the image difference value as the generation loss function, and adjusts the generation parameters in the initial generation sub-model in a direction in which the generation loss function and the generation probability difference increase. Each time the adjustment is made, the generation parameters are updated to obtain the new intermediate generation sub-model.

[0112] In this embodiment, the computer device obtains the training image content difference value between the training generated images according to the content difference formula for the training eigenvalue square difference norm, obtains the training image style difference value between the training generated images according to the style difference formula for the Gram matrix square difference norm value between the training generated images, takes the sum of the training image content difference value and the training image style difference value as the generation loss function, adjusts the generation parameters in the initial generation sub-model in the direction of increasing the generation loss function and the generation probability difference, and obtains the intermediate generation sub-model. By adjusting the generation parameters in the initial generation sub-model in the direction of increasing the generation loss function and the generation probability difference, the training of the initial generation sub-model is achieved to ensure that the difference between the training generated images is large and the diversity between the generated images is improved.

[0113] In one embodiment, Figure 7 As shown, the step S540, obtaining the sum of the training image content difference value and the training image style difference value to obtain the image difference value, includes:

[0114] S710: Multiply the image content weight by the initial training image content difference value to obtain the training image content difference value.

[0115] S720: Multiply the image style weight by the initial training image style difference value to obtain the training image style difference value.

[0116] S730: Add the training image content difference value to the training image style difference value to obtain the image difference value.

[0117] Specifically, since the image content features and image style features reflect the image differences at different levels, different weights can be set in different image generation tasks to focus on different levels of the generated image. The image content weight is multiplied by the initial training image content difference value to obtain the training image content difference, and the image style weight is multiplied by the initial training image style difference value to obtain the training image style difference value, and then the sum is used to obtain the image difference value. For example, in an image generation task that focuses on image contour features, the image content weight can be set to be greater than the image style weight, and in an image generation task that focuses on image color and texture features, the image style weight can be set to be greater than the image content weight.

[0118] In this embodiment, the computer device obtains the image difference value by setting the image content weight and the image style weight, which ensures the diversity between the generated images and reflects the different levels of concern in different image generation tasks.

[0119] In one embodiment, Figure 8 As shown, in order to enable the image generation model to complete the above-mentioned task of generating dog images from cat images, the generation of two training generated images is taken as an example for explanation. The image generation model includes a generation sub-model and a discrimination sub-model. The training process of the image generation model includes:

[0120] Use cat images as training source images (input) and input them into the initial generation sub-model.

[0121] The decoder (Encode) in the initial generation sub-model extracts training image features from the training source image (input).

[0122] The encoder (Decode) in the initial generation sub-model receives the training image features, and fuses the training image features with the first random noise parameter Z1 to generate a first training generated image I1, fuses the training image features with the second random noise parameter Z2 to generate a second training generated image I2, and outputs the first training generated image I1 and the second training generated image I2 as training generated images (output).

[0123] A dog image is used as a target image, and the target image and the obtained training generated image are input into a discriminator in an initial discriminant sub-model. The discriminator obtains the generation probability that the training generated image is a true image and the target probability that the target image is a true image based on the image features of the training generated image and the target image features of the target image. The average value of the difference between the generation probability and the standard probability of a fake image (0) is taken as the generation probability difference, the average value of the difference between the target probability and the standard probability of a true image (1) is taken as the target probability difference, and the average value of the probability difference between the generation probability difference and the target probability difference is taken to obtain the probability difference. The computer device adjusts the discriminant parameters in the initial discriminant sub-model in the direction of reducing the probability difference until the probability difference is minimum or less than the preset probability difference, stops training, and obtains the discriminant sub-model. The discriminator can also determine the authenticity (real / fake) of the training generated image based on the generation probability of each training generated image. If the generation probability of the training generated image is ≥50%, the training generated image is a true image; if the generation probability of the training generated image is <50%, the training generated image is a false image.

[0124] The discriminator in the initial discriminant sub-model is also used to calculate the training image content difference value and the training image style difference value between the training generated images based on feature matching according to the training feature value of the image feature extracted from the training generated image, and use the sum of the training image content difference value and the training image style difference value as the generation loss function of the initial generation sub-model. The computer device adjusts the generation parameters (parameters in the decoder and the encoder) in the initial generation sub-model according to the generation loss function and the generation probability difference of the training generated image until the generation loss function is maximum or greater than the preset loss value and the generation probability difference is maximum or greater than the preset probability difference, stops training, and obtains the generation sub-model.

[0125] It should be understood that although Figure 1-8 The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1-8At least part of the steps may include multiple steps or multiple stages. These steps or stages are not necessarily performed at the same time, but can be performed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed in turn or alternately with other steps or at least part of the steps or stages in other steps.

[0126] In one embodiment, Fig. 9 As shown, an image generation device is provided, including: a feature extraction module 910 and a target generation module 920, wherein:

[0127] The feature extraction module 910 is used to extract input image features of the input source image through an image generation model;

[0128] The target generation module 920 is used to utilize the image generation model to obtain a generated image according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize a generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images.

[0129] In one embodiment, the image generation device further includes a target training module, the image generation model includes a discriminant sub-model and a generative sub-model, and the target training module is used to:

[0130] Obtain at least two training generated images of the training source image through the initial generation sub-model;

[0131] The initial discriminant sub-model and the initial generative sub-model are trained according to the training generated image and the target image to obtain the discriminant sub-model and the generative sub-model; wherein the initial discriminant sub-model is used to judge whether the input image is a true image according to the true image probability of the input image being a true image; and the target image is an image of the same type as the generated image.

[0132] In one embodiment, the target training module is further used to:

[0133] The initial discriminant sub-model is trained according to the probability difference to obtain an intermediate discriminant sub-model; wherein the probability difference is obtained according to the generation probability that the training generated image is a true image and the target probability that the target image is a true image;

[0134] The initial generation sub-model is trained according to the generation loss function and the generation probability to obtain an intermediate generation sub-model;

[0135] When the probability difference is less than the preset difference, the generated loss function is greater than the preset loss value and the generated probability difference is greater than the preset probability difference, the intermediate discriminant sub-model is used as the discriminant sub-model, and the intermediate generated sub-model is used as the generated sub-model; wherein the generated probability difference is obtained according to the difference between the generated probability and the standard probability of the false image.

[0136] In one embodiment, the target training module is further used to:

[0137] Inputting the training generated image and the target image into the initial discriminant sub-model to obtain the generation probability and the target probability;

[0138] Obtaining an average value of the difference between the generation probability and the false image standard probability as the generation probability difference;

[0139] Obtaining an average value of the difference between the target probability and the true image standard probability as the target probability difference;

[0140] Obtaining an average value of the probability difference between the generated probability difference and the target probability difference to obtain the probability difference value;

[0141] The discriminant parameters in the initial discriminant sub-model are adjusted in a direction in which the probability difference is reduced to obtain the intermediate discriminant sub-model.

[0142] In one embodiment, the target training module is further used to:

[0143] Extracting image features of the training generated image to obtain training feature values;

[0144] Obtaining a training image content difference value between the training generated images according to a second norm of a square difference of two training feature values;

[0145] Using a Gram matrix to obtain a training image style difference value between the training generated images;

[0146] Obtaining the sum of the training image content difference value and the training image style difference value to obtain an image difference value;

[0147] The image difference value is used as the generation loss function, and the generation parameters in the initial generation sub-model are adjusted in a direction in which the generation loss function and the generation probability difference increase, so as to obtain the intermediate generation sub-model.

[0148] In one embodiment, the target training module is further used to:

[0149] Multiplying the image content weight by the initial training image content difference value to obtain the training image content difference value;

[0150] Multiplying the image style weight by the initial training image style difference value to obtain the training image style difference value;

[0151] The training image content difference value is added to the training image style difference value to obtain the image difference value.

[0152] In one embodiment, the target training module is further used to:

[0153] Obtaining the product of training feature values ​​at different dimensions of the training generated image to obtain a training feature product;

[0154] Obtain the corresponding Gram matrix according to the training feature product of each training generated image;

[0155] The training image style difference values ​​between the training images are obtained according to the norm values ​​of the Gram matrix square differences between the training generated images.

[0156] For the specific definition of the image generation device, please refer to the definition of the image generation method above, which will not be repeated here. Each module in the above-mentioned image generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0157] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10 As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be achieved through WIFI, an operator network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an image generation method is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0158] Those skilled in the art will understand that Fig.10The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0159] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0160] Extracting input image features of an input source image through an image generation model;

[0161] The image generation model is used to obtain a generated image according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize a generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images.

[0162] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0163] Obtain at least two training generated images of the training source image through the initial generation sub-model;

[0164] The initial discriminant sub-model and the initial generative sub-model are trained according to the training generated image and the target image to obtain the discriminant sub-model and the generative sub-model; wherein the initial discriminant sub-model is used to judge whether the input image is a true image according to the true image probability of the input image being a true image; and the target image is an image of the same type as the generated image.

[0165] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0166] The initial discriminant sub-model is trained according to the probability difference to obtain an intermediate discriminant sub-model; wherein the probability difference is obtained according to the generation probability that the training generated image is a true image and the target probability that the target image is a true image;

[0167] The initial generation sub-model is trained according to the generation loss function and the generation probability to obtain an intermediate generation sub-model;

[0168] When the probability difference is less than the preset difference, the generated loss function is greater than the preset loss value and the generated probability difference is greater than the preset probability difference, the intermediate discriminant sub-model is used as the discriminant sub-model, and the intermediate generated sub-model is used as the generated sub-model; wherein the generated probability difference is obtained according to the difference between the generated probability and the standard probability of the false image.

[0169] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0170] Inputting the training generated image and the target image into the initial discriminant sub-model to obtain the generation probability and the target probability;

[0171] Obtaining an average value of the difference between the generation probability and the false image standard probability as the generation probability difference;

[0172] Obtaining an average value of the difference between the target probability and the true image standard probability as the target probability difference;

[0173] Obtaining an average value of the probability difference between the generated probability difference and the target probability difference to obtain the probability difference value;

[0174] The discriminant parameters in the initial discriminant sub-model are adjusted in a direction in which the probability difference is reduced to obtain the intermediate discriminant sub-model.

[0175] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0176] Extracting image features of the training generated image to obtain training feature values;

[0177] Obtaining a training image content difference value between the training generated images according to a second norm of a square difference of two training feature values;

[0178] Using a Gram matrix to obtain a training image style difference value between the training generated images;

[0179] Obtaining the sum of the training image content difference value and the training image style difference value to obtain an image difference value;

[0180] The image difference value is used as the generation loss function, and the generation parameters in the initial generation sub-model are adjusted in a direction in which the generation loss function and the generation probability difference increase, so as to obtain the intermediate generation sub-model.

[0181] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0182] Multiplying the image content weight by the initial training image content difference value to obtain the training image content difference value;

[0183] Multiplying the image style weight by the initial training image style difference value to obtain the training image style difference value;

[0184] The training image content difference value is added to the training image style difference value to obtain the image difference value.

[0185] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0186] Obtaining the product of training feature values ​​at different dimensions of the training generated image to obtain a training feature product;

[0187] Obtain the corresponding Gram matrix according to the training feature product of each training generated image;

[0188] The training image style difference values ​​between the training images are obtained according to the norm values ​​of the Gram matrix square differences between the training generated images.

[0189] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0190] Extracting input image features of an input source image through an image generation model;

[0191] The image generation model is used to obtain a generated image according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize a generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images.

[0192] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0193] Obtain at least two training generated images of the training source image through the initial generation sub-model;

[0194] The initial discriminant sub-model and the initial generative sub-model are trained according to the training generated image and the target image to obtain the discriminant sub-model and the generative sub-model; wherein the initial discriminant sub-model is used to judge whether the input image is a true image according to the true image probability of the input image being a true image; and the target image is an image of the same type as the generated image.

[0195] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0196] The initial discriminant sub-model is trained according to the probability difference to obtain an intermediate discriminant sub-model; wherein the probability difference is obtained according to the generation probability that the training generated image is a true image and the target probability that the target image is a true image;

[0197] The initial generation sub-model is trained according to the generation loss function and the generation probability to obtain an intermediate generation sub-model;

[0198] When the probability difference is less than the preset difference, the generated loss function is greater than the preset loss value and the generated probability difference is greater than the preset probability difference, the intermediate discriminant sub-model is used as the discriminant sub-model, and the intermediate generated sub-model is used as the generated sub-model; wherein the generated probability difference is obtained according to the difference between the generated probability and the standard probability of the false image.

[0199] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0200] Inputting the training generated image and the target image into the initial discriminant sub-model to obtain the generation probability and the target probability;

[0201] Obtaining an average value of the difference between the generation probability and the false image standard probability as the generation probability difference;

[0202] Obtaining an average value of the difference between the target probability and the true image standard probability as the target probability difference;

[0203] Obtaining an average value of the probability difference between the generated probability difference and the target probability difference to obtain the probability difference value;

[0204] The discriminant parameters in the initial discriminant sub-model are adjusted in a direction in which the probability difference is reduced to obtain the intermediate discriminant sub-model.

[0205] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0206] Extracting image features of the training generated image to obtain training feature values;

[0207] Obtaining a training image content difference value between the training generated images according to a second norm of a square difference of two training feature values;

[0208] Using a Gram matrix to obtain a training image style difference value between the training generated images;

[0209] Obtaining the sum of the training image content difference value and the training image style difference value to obtain an image difference value;

[0210] The image difference value is used as the generation loss function, and the generation parameters in the initial generation sub-model are adjusted in a direction in which the generation loss function and the generation probability difference increase, so as to obtain the intermediate generation sub-model.

[0211] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0212] Multiplying the image content weight by the initial training image content difference value to obtain the training image content difference value;

[0213] Multiplying the image style weight by the initial training image style difference value to obtain the training image style difference value;

[0214] The training image content difference value is added to the training image style difference value to obtain the image difference value.

[0215] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0216] Obtaining the product of training feature values ​​at different dimensions of the training generated image to obtain a training feature product;

[0217] Obtain the corresponding Gram matrix according to the training feature product of each training generated image;

[0218] The training image style difference values ​​between the training images are obtained according to the norm values ​​of the Gram matrix square differences between the training generated images.

[0219] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0220] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0221] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. An image generation method, characterized in that: The method comprises: Extracting input image features of an input source image through an image generation model; The generated image is obtained by using the image generation model according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize the generation loss function, and the generation loss function is the sum of the image content difference value and the image style difference value between the generated images; the input source image is the base image of the generated image; The image generation model includes a discriminant sub-model and a generative sub-model. Before obtaining a generated image according to the input image features and random noise parameters using the image generation model, the method includes: Obtain at least two training generated images of the training source image through the initial generation sub-model; Inputting the training generated image and the target image into the initial discriminant sub-model to obtain a generation probability and a target probability; Obtaining an average value of the difference between the generation probability and the false image standard probability as the generation probability difference; Obtaining an average value of the difference between the target probability and the true image standard probability as the target probability difference; Obtaining an average value of the probability difference between the generated probability difference and the target probability difference to obtain a probability difference value; The discriminant parameters in the initial discriminant sub-model are adjusted in a direction in which the probability difference is reduced to obtain an intermediate discriminant sub-model; wherein the probability difference is obtained according to the generation probability that the training generated image is a true image and the target probability that the target image is a true image; The initial generation sub-model is trained according to the generation loss function and the generation probability to obtain an intermediate generation sub-model; When the probability difference is less than a preset difference, the generated loss function is greater than a preset loss value and the generated probability difference is greater than a preset probability difference, the intermediate discriminant sub-model is used as the discriminant sub-model, and the intermediate generated sub-model is used as the generated sub-model; wherein the generated probability difference is obtained according to the difference between the generated probability and the standard probability of a false image; wherein the initial discriminant sub-model is used to determine whether the input image is a true image according to the true image probability of the input image being a true image; and the target image is an image of the same type as the generated image.

2. The method according to claim 1, characterized in that The step of training the initial generation sub-model according to the generation loss function and the generation probability to obtain an intermediate generation sub-model includes: Extracting image features of the training generated image to obtain training feature values; Obtaining a training image content difference value between the training generated images according to a second norm of a square difference of two training feature values; Using a Gram matrix to obtain a training image style difference value between the training generated images; Obtaining the sum of the training image content difference value and the training image style difference value to obtain an image difference value; The image difference value is used as the generation loss function, and the generation parameters in the initial generation sub-model are adjusted in a direction in which the generation loss function and the generation probability difference increase, so as to obtain the intermediate generation sub-model.

3. The method according to claim 2, characterized in that The obtaining the sum of the training image content difference value and the training image style difference value to obtain the image difference value includes: Multiplying the image content weight by the initial training image content difference value to obtain the training image content difference value; Multiplying the image style weight by the initial training image style difference value to obtain the training image style difference value; The training image content difference value is added to the training image style difference value to obtain the image difference value.

4. The method according to claim 2, characterized in that: The step of using the Gram matrix to obtain the training image style difference value between the training generated images includes: Obtaining the product of training feature values ​​at different dimensions of the training generated image to obtain a training feature product; Obtain the corresponding Gram matrix according to the training feature product of each training generated image; The training image style difference values ​​between the training images are obtained according to the norm values ​​of the Gram matrix square differences between the training generated images.

5. An image generating device, characterized in that: The device comprises: A feature extraction module, for extracting input image features of an input source image through an image generation model; A target generation module, used to obtain a generated image using the image generation model according to the input image features and random noise parameters; wherein the image generation model includes a neural network model trained to maximize a generation loss function, wherein the generation loss function is the sum of an image content difference value and an image style difference value between the generated images; and the input source image is a base image of the generated image; The image generation model includes a discriminant sub-model and a generative sub-model. Before obtaining a generated image according to the input image features and random noise parameters using the image generation model, the method includes: The target training module is used to obtain at least two training generated images of the training source image through the initial generation sub-model; input the training generated image and the target image into the initial discriminant sub-model to obtain the generation probability and the target probability; obtain the average value of the difference between the generation probability and the standard probability of the false image as the generation probability difference; obtain the average value of the difference between the target probability and the standard probability of the true image as the target probability difference; obtain the average value of the probability difference between the generation probability difference and the target probability difference to obtain the probability difference; adjust the discriminant parameters in the initial discriminant sub-model in the direction of reducing the probability difference to obtain an intermediate discriminant sub-model; wherein the probability difference is calculated based on the generation probability of the training generated image being the true image and the target image. The target probability that the marked image is a true image is obtained; the initial generation sub-model is trained according to the generation loss function and the generation probability to obtain an intermediate generation sub-model; when the probability difference is less than the preset difference, the generation loss function is greater than the preset loss value and the generation probability difference is greater than the preset probability difference, the intermediate discriminant sub-model is used as the discriminant sub-model, and the intermediate generation sub-model is used as the generation sub-model; wherein the generation probability difference is obtained according to the difference between the generation probability and the standard probability of a false image; wherein the initial discriminant sub-model is used to judge whether the input image is a true image according to the true image probability of the input image being a true image; the target image is an image of the same type as the generated image.

6. The device according to claim 5, characterized in that The target training module is further used to extract image features of the training generated images to obtain training feature values; obtain a training image content difference value between the training generated images according to a second norm of a square difference of two training feature values; and obtain a training image style difference value between the training generated images using a Gram matrix; The sum of the content difference value of the training image and the style difference value of the training image is obtained to obtain an image difference value; the image difference value is used as the generation loss function, and the generation parameters in the initial generation sub-model are adjusted in a direction in which the generation loss function and the generation probability difference increase to obtain the intermediate generation sub-model.

7. The device according to claim 6, characterized in that The target training module is further used to multiply the image content weight by the initial training image content difference value to obtain the training image content difference value; multiply the image style weight by the initial training image style difference value to obtain the training image style difference value; and add the training image content difference value to the training image style difference value to obtain the image difference value.

8. The device according to claim 6, characterized in that The target training module is also used to obtain the product of the training feature values ​​of the training generated image in different dimensions to obtain the training feature product; obtain the corresponding Gram matrix according to the training feature product of each training generated image; and obtain the training image style difference value between the training images according to the norm value of the Gram matrix square difference between the training generated images.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 4 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 4 are implemented.

Images