Method, device and computer-readable storage medium for generating target image

Abstract

The present disclosure relates to a method, device and computer-readable storage medium for generating a target image, and relates to the field of computer technology. The method comprises: according to a residual learning method, using a generative model to determine a first feature map and an attention feature map of an image to be processed; according to a convolution processing method, using a generative model to determine a second feature map of the image to be processed; according to the second feature map and the attention feature map, using a generative model to determine a third feature map; according to the third feature map, the first feature map and feature information of a first additional condition, using a generative model to generate a first target image that meets the first additional condition.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a method for generating a target image, a device for generating a target image, and a computer-readable storage medium. Background Art

[0002] With the development of computer vision technology, it is possible to generate target images that meet additional conditions based on existing images according to business needs. For example, additional conditions can be to change the person in the existing image to a specified race, gender, age, etc.; or to change the scene in the existing image to a specified time period, season, weather, etc.

[0003] In recent years, the face aging generation task has attracted more and more attention. This task has a wide range of application scenarios. For example, in the film and television entertainment industry, when playing the same person at different ages, the use of face aging algorithms can greatly improve the efficiency of film and television production and reduce production costs; in the field of identity verification, the use of face aging algorithms can also perform cross-age face verification and help find children who have been missing for many years.

[0004] In the related art, sample groups that meet different additional conditions can be used to simulate the change of additional conditions. For example, the difference between different age groups can be used to simulate the age change pattern to generate an aging image. Summary of the invention

[0005] The inventors of the present disclosure have discovered that the above-mentioned related technologies have the following problems: the image is processed indiscriminately as a whole, resulting in poor quality of the generated target image and failure to meet the additional conditions.

[0006] In view of this, the present disclosure proposes a technical solution for generating a target image, which can improve the effect of the target image and better meet the additional conditions.

[0007] According to some embodiments of the present disclosure, a method for generating a target image is provided, including: according to a residual learning method, using a generative model to determine a first feature map and an attention feature map of an image to be processed; according to a convolution processing method, using a generative model to determine a second feature map of the image to be processed; according to the second feature map and the attention feature map, using a generative model to determine a third feature map; according to the third feature map, the first feature map and feature information of the first additional condition, using the generative model to generate a first target image that meets the first additional condition.

[0008] In some embodiments, using a generative model to determine a first feature map and an attention feature map of an image to be processed includes: determining the first feature map and the attention feature map according to a first additional condition; using a generative model to determine a second feature map of an image to be processed includes: determining the second feature map according to the first additional condition.

[0009] In some embodiments, according to the first additional condition, determining the first feature map and the attention feature map includes: adding the first additional condition to the residual learning method by means of conditional instance regularization; according to the first additional condition, determining the second feature map includes: adding the first additional condition to the convolution processing method by means of conditional instance regularization.

[0010] In some embodiments, determining the second feature map of the image to be processed by using the generative model includes:

[0011] Convolution processing is performed on each image region of the image to be processed to determine a second feature map. In some embodiments, determining the second feature map of the image to be processed using the generative model includes: using conditional instance regularization to add feature information of the first additional condition to each image region of the image to be processed; and convolution processing is performed on each image region to which the feature information of the first additional condition is added to determine the second feature map.

[0012] In some embodiments, the feature information of the first additional condition is a feature map of the first additional condition; using the generative model to generate a first target image that meets the first additional condition includes: fusing the third feature map, the first feature map and the feature map of the first additional condition into data to be processed; and generating the first target image based on the data to be processed.

[0013] In some embodiments, the generative model is trained using a loss function, and the loss function is determined by the following steps: according to the second additional condition, using the generative model to generate a second target image that satisfies the second additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a third target image that satisfies the third additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a fourth target image that satisfies the third additional condition based on the second target image; and determining the loss function based on the difference between the fourth target image and the third target image, and the difference between the image to be processed and the first target image.

[0014] In some embodiments, the generative model is trained using an objective function until the objective function reaches a maximum value. The objective function takes the first target image as a variable, is positively correlated with the target similarity, is negatively correlated with the sum of each real similarity, and is negatively correlated with the sum of each generated similarity.

[0015] In some embodiments, the target similarity is the similarity between the first target image and the real image that meets the first additional condition. The real similarity is the similarity between the first target image and the real image that meets other additional conditions. The generated similarity is the similarity between the first target image and the generated image that meets the first additional condition and other additional conditions. The generated image is an image generated by the generation model according to the additional conditions.

[0016] In some embodiments, the target similarity is the similarity between the feature information of the first real data distribution and the feature information of the first target image. The first real data distribution is the data distribution of the real image that meets the first additional condition.

[0017] In some embodiments, the real similarity is the similarity between the feature information of other real data distributions and the feature information of the first target image. The other real data distributions are data distributions of real images that meet other additional conditions.

[0018] In some embodiments, the generated similarity is the similarity between the feature information of the generated data distribution and the feature information of the first target image. The generated data distribution is the data distribution of the generated image that meets the first additional condition and other additional conditions.

[0019] In some embodiments, the generative model is trained based on the discrimination results of the discriminative model on the first target image and the real image satisfying the first additional condition.

[0020] According to some other embodiments of the present disclosure, a device for generating a target image is provided, including: a residual learning unit, for determining a first feature map and an attention feature map of an image to be processed by using a generative model according to a residual learning method; a convolution processing unit, for determining a second feature map of the image to be processed by using a generative model according to a convolution processing method; a determination unit, for determining a third feature map by using a generative model according to the second feature map and the attention feature map; and a generation unit, for generating a first target image that meets the first additional condition by using a generative model according to feature information of the third feature map, the first feature map and the first additional condition.

[0021] In some embodiments, the residual learning unit determines the first feature map and the attention feature map according to the first additional condition; the convolution processing unit determines the second feature map according to the first additional condition.

[0022] In some embodiments, the residual learning unit adds a first additional condition to the residual learning method by using conditional instance regularization; the convolution processing unit adds a first additional condition to the convolution processing method by using conditional instance regularization.

[0023] In some embodiments, the convolution processing unit performs convolution processing on each image region of the image to be processed respectively to determine the second feature map.

[0024] In some embodiments, the convolution processing unit uses conditional instance regularization to add feature information of the first additional condition to each image region of the image to be processed; convolution processing is performed on each image region to which the feature information of the first additional condition is added to determine a second feature map.

[0025] In some embodiments, the feature information of the first additional condition is a feature map of the first additional condition; the generating unit fuses the third feature map, the first feature map and the feature map of the first additional condition into data to be processed; and generates a first target image based on the data to be processed.

[0026] In some embodiments, the generative model is trained using a loss function, and the loss function is determined by the following steps: according to the second additional condition, using the generative model to generate a second target image that satisfies the second additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a third target image that satisfies the third additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a fourth target image that satisfies the third additional condition based on the second target image; and determining the loss function based on the difference between the fourth target image and the third target image, and the difference between the image to be processed and the first target image.

[0027] In some embodiments, the generative model is trained using an objective function until the objective function reaches a maximum value. The objective function takes the first target image as a variable, is positively correlated with the target similarity, is negatively correlated with the sum of each real similarity, and is negatively correlated with the sum of each generated similarity.

[0028] In some embodiments, the target similarity is the similarity between the first target image and the real image that satisfies the first additional condition, the real similarity is the similarity between the first target image and the real image that satisfies other additional conditions, the generated similarity is the similarity between the first target image and the generated image that satisfies the first additional condition and other additional conditions, and the generated image is an image generated by the generation model according to the additional conditions.

[0029] In some embodiments, the target similarity is the similarity between the feature information of the first real data distribution and the feature information of the first target image. The first real data distribution is the data distribution of the real image that meets the first additional condition.

[0030] In some embodiments, the real similarity is the similarity between the feature information of other real data distributions and the feature information of the first target image. The other real data distributions are data distributions of real images that meet other additional conditions.

[0031] In some embodiments, the generated similarity is the similarity between the feature information of the generated data distribution and the feature information of the first target image, and the generated data distribution is the data distribution of the generated image that satisfies the first additional condition and other additional conditions.

[0032] According to some further embodiments of the present disclosure, there is provided a device for generating a target image, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the method for generating a target image in any one of the above embodiments based on instructions stored in the memory device.

[0033] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for generating a target image in any of the above embodiments is implemented.

[0034] In the above embodiment, the convolution processing result is weighted by using the attention feature map output by the residual learning method, so that the generation method can focus on the key areas in the image to be processed. In this way, a target image that can better meet the additional conditions can be generated, thereby improving the effect of the target image. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0036] The present disclosure may be more clearly understood from the following detailed description with reference to the accompanying drawings:

[0037] Figure 1 A flowchart showing some embodiments of the method for generating a target image of the present disclosure;

[0038] Figure 2 Schematic diagrams showing some embodiments of the method for generating a target image disclosed herein;

[0039] Figure 3 Schematic diagrams showing other embodiments of the method for generating a target image disclosed herein;

[0040] Figure 4 A block diagram showing some embodiments of a device for generating a target image of the present disclosure;

[0041] Figure 5 A block diagram showing some other embodiments of the device for generating a target image of the present disclosure;

[0042] Figure 6 A block diagram showing some further embodiments of the device for generating a target image according to the present disclosure. DETAILED DESCRIPTION

[0043] Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present disclosure unless otherwise specifically stated.

[0044] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0045] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the present disclosure, its application, or uses.

[0046] Technologies, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but where appropriate, the technologies, methods and equipment should be considered part of the authorization specification.

[0047] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0048] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0049] Figure 1 A flowchart showing some embodiments of the method for generating a target image of the present disclosure.

[0050] like Figure 1 As shown, the method includes: step 110, performing residual processing; step 120, performing convolution processing; step 130, determining a third feature map; and step 140, generating a first target image.

[0051] In step 110, according to the residual learning method, the generative model is used to determine the first feature map and the attention feature map of the image to be processed. For example, the image to be processed is a portrait image, and the user needs to perform aging processing on the image to be processed.

[0052] In step 120, according to the convolution processing method, a second feature map of the image to be processed is determined using a generative model.

[0053] In some embodiments, the first feature map and the attention feature map are determined according to the first additional condition; the second feature map is determined according to the first additional condition. For example, the first additional condition can be added to the residual learning method by using CIN (Conditional Instance Normalization); the first additional condition can be added to the convolution processing method by using conditional instance regularization.

[0054] In some embodiments, convolution processing is performed on each image region of the image to be processed to determine the second feature map. For example, by using conditional instance regularization, feature information of the first additional condition is added to each image region of the image to be processed; convolution processing is performed on each image region to which the feature information of the first additional condition is added to determine the second feature map.

[0055] In step 130, a third feature map is determined using a generative model based on the second feature map and the attention feature map.

[0056] In step 140, a first target image that meets the first additional condition is generated using a generation model based on the third feature map, the first feature map, and feature information of the first additional condition. For example, the first additional condition is designated age information for aging or rejuvenation.

[0057] In some embodiments, the feature information of the first additional condition is a feature map of the first additional condition. The third feature map, the first feature map and the feature map of the first additional condition are merged into data to be processed; and a first target image is generated according to the data to be processed.

[0058] In some embodiments, Figure 2 The embodiment in utilizes a generative model to generate a target image.

[0059] Figure 2 Schematic diagrams showing some embodiments of the method for generating a target image disclosed herein.

[0060] like Figure 2 As shown, the generator (generative model) may include a deep generative network, a sampler group, and a fusion generative network. For example, the generator is used to take a face picture as an input image and generate an aging or rejuvenation image as an output image.

[0061] In some embodiments, the deep generative network is constructed by a series of residual modules. For example, the residual module may include a downsampling residual module, a residual processing module, and an upsampling residual module. The downsampling residual module is used for downsampling processing and residual processing; the residual processing module includes multiple layers for residual processing; the upsampling residual module is used for upsampling processing and residual processing. The deep generative network adopts the structure of upsampling residual module plus downsampling residual module to ensure that the feature size of the output is consistent with the input.

[0062] For example, the input image of the deep generative network can be an original face picture, and the output is the deep features used to generate the face, as well as the attention feature map used to guide the sampler group.

[0063] For example, the age condition information (specified age information) can be embedded in each residual module of the deep generative network by means of CIN. In this way, the age condition information can be better encoded into the deep features, thereby generating facial features of the specified age.

[0064] In some embodiments, the sampler group may adopt a multi-branch parallel structure, each branch including a CIN module for embedding age condition information and an intermediate convolution module for extracting shallow features. After the input image is processed by the pre-convolution module, the processing result may be input into each branch for processing different regions of the input image.

[0065] For example, each intermediate convolutional module can be a shallow network composed of atrous convolutions with different dilation rates. Each intermediate convolutional module is used to obtain pixel information at different positions in the input image.

[0066] For example, the input of the sampler group is the original face image, and the output is a series of shallow features used to generate the face. The shallow features output by each branch target different areas of the face.

[0067] In some embodiments, the attention feature map output by the deep generative network can be used to weight the shallow features input by each branch to obtain weighted shallow features, so that certain specific areas can be focused on.

[0068] For example, when turning a picture of a middle-aged man with a beard into a picture of a child, the generation of the image area where the mouth is located may rely more on distant facial information. In this case, the attention feature map can improve the effect of aging or rejuvenation processing with a large age span.

[0069] In some embodiments, the fusion network obtains the data to be processed by combining the deep features, the weighted shallow features and the feature maps of the age condition information. The data to be processed is sent to the subsequent convolution layer to obtain an aging or rejuvenation image of a specified age. For example, the network can cascade the deep features, the weighted shallow features and the feature maps of the age condition information to obtain the data to be processed.

[0070] In some embodiments, the generative model is trained using a loss function, and the loss function is determined by the following steps: according to the second additional condition, using the generative model to generate a second target image that satisfies the second additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a third target image that satisfies the third additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a fourth target image that satisfies the third additional condition based on the second target image; and determining the loss function based on the difference between the fourth target image and the third target image, and the difference between the image to be processed and the first target image.

[0071] In some embodiments, for the task of processing face aging or rejuvenation, the training of the generator must not only constrain the generator to generate images that meet the target data distribution and input conditions, but also constrain the identity preservation conditions. That is, the training of the generator must also constrain the input face and the output face to be identifiable as the same person.

[0072] For example, the generator needs to generate an output image G(x,y1) that satisfies the age condition y1 based on the input image x. In this case, the loss function L can be determined as follows:

[0073]

[0074] E( ) is the mathematical expectation when each variable satisfies a certain probability distribution, q(X,Y) is the real data distribution (joint probability distribution) of real pictures X that meet age condition Y, q(X) is the real data distribution (probability distribution) of real pictures X, q(Y) is the real data distribution (probability distribution) of age condition Y, and || ||1 is the 1 norm.

[0075] y2 and y3 are two other age conditions different from y1. The generator generates an output image that satisfies age condition y3 based on the input image x; the generator first generates an image G(x, y2) that satisfies age condition y2, and then generates an output image G(G(x, y2), y3) that satisfies age condition y3 based on image G(x, y2).

[0076] In the above formula, the first item is used to evaluate whether the generator can maintain the consistency of input and output (such as identity consistency) when reconstructing based on the input image; the second item is used to evaluate whether the generator can maintain the consistency of direct and indirect processing when performing direct and indirect processing based on the input image.

[0077] In this way, the consistency of the input and output images (such as identity consistency) can be constrained over the entire sequence of additional conditions (such as age conditions). Moreover, such a loss function does not require an additional network to achieve the above functions, which is more flexible and convenient.

[0078] In some embodiments, not only can the generator be used to generate the target image, but the generator can also be trained adversarially using the objective function. For example, the generative model is trained using the objective function until the objective function reaches a maximum value. The objective function takes the first target image as a variable, is positively correlated with the target similarity, is negatively correlated with the sum of each real similarity, and is negatively correlated with the sum of each generated similarity.

[0079] In some embodiments, the target similarity is the similarity between the first target image and the real image satisfying the first additional condition. For example, the target similarity is the similarity between the feature information of the first real data distribution and the feature information of the first target image, and the first real data distribution is the data distribution of the real image satisfying the first additional condition.

[0080] In some embodiments, the real similarity is the similarity between the first target image and the real image that meets other additional conditions. For example, the real similarity is the similarity between the feature information of other real data distributions and the feature information of the first target image. The other real data distribution is the data distribution of the real image that meets other additional conditions.

[0081] In some embodiments, the generated similarity is the similarity between the first target image and the generated image satisfying the first additional condition and other additional conditions, and the generated image is an image generated by the generation model according to the additional conditions. For example, the generated similarity is the similarity between the feature information of the generated data distribution and the feature information of the first target image, and the generated data distribution is the data distribution of the generated image satisfying the first additional condition and other additional conditions.

[0082] In some embodiments, the generative model is trained based on the discrimination results of the discriminative model on the first target image and the real image satisfying the first additional condition.

[0083] In some embodiments, it is possible to use Figure 3 The discriminator and cross classifier in are used to perform adversarial training on the generator.

[0084] Figure 3 Schematic diagrams showing other embodiments of the method for generating a target image of the present disclosure.

[0085] like Figure 3 As shown in the figure, after the input image is processed by the generator, an output image that meets the age condition is generated; the discriminator distinguishes the output image from the real image that meets the corresponding age condition; the generator is adversarially trained by combining the discrimination result of the discriminator and the objective function of the cross classifier.

[0086] In some embodiments, in the task of face aging, because it is necessary to specify the age condition for generating pictures, it can be implemented through CGAN (Conditional Generative Adversarial Nets).

[0087] The special thing about CGANs is that both the generator and the discriminator receive an additional conditional information, so the data generated by the generator must not only conform to the distribution of the real data, but also meet the conditional information (such as age conditions). Therefore, the discriminator needs to distinguish the joint distribution of the real data and the conditional information. In other words, the discriminator must be able to distinguish between the generated pictures and the real pictures, and also determine whether the pictures generated by the generator meet the specified age conditions.

[0088] In some embodiments, the generator wants to generate an output image G(x, y=k) satisfying the condition y=k based on the input image x. The modeling equation of the generator can be a log-linear model in the feature space:

[0089]

[0090] f Φ (G(x,y=k) is the feature vector of the image G(x,y=k) generated by the generator based on the input image x that satisfies the condition information (such as age condition) y=k, and K is the total number of categories of condition information. P(y=k|f Φ (G(x,y=k)) is the conditional probability. The k-th type of generated data (generated images satisfying y=k) of the generator all satisfy the same generated data distribution, that is, they are located on the same classification hyperplane. Generate the normal vector of the classification hyperplane where the k-th category data is located, and T is the transposition operation. The larger the value of is, the closer the generated image G(x, y=k) is to the generated data distribution of the kth category of the generator.

[0091] When this modeling equation reaches the maximum value, the generator completes the generation of G(x, y=k).

[0092] Based on this modeling equation, the k-th category of real data (real images satisfying y=k) all satisfy the same real data distribution, that is, they are located on the same classification hyperplane. is the normal vector of the classification hyperplane where the k-th class of real data is located.

[0093] In some embodiments, the cross classifier of the present disclosure can make f Φ (G(x,y=k)) tries to fit the eigenvector of the kth class of real data, while staying away from the eigenvectors of other classes of real data and the eigenvectors of other classes of generated data.

[0094] In some embodiments, selecting As the fitting direction of the generator, Will use As the gradient, directly direction to fit, thereby improving the efficiency of training and learning.

[0095] For example, the objective function for training the generator in a cross classifier can be configured to have the effect of forcing f Φ (G(x,y=k)) is far away The thrust term and the force f Φ (G(x,y=k)) is close to The tension item.

[0096] In the selection As the fitting direction of the generator, when the generator maximizes the modeling equation, the generator is constrained to generate data G~P(G|y=k), away from the classification hyperplane of other real data

[0097] In some embodiments, in the face aging task, age categories are mutually exclusive. The classification hyperplane of other categories of data can be used as a driving term to improve the performance of the generator, that is, a cross classifier. For example, each category is divided into two subcategories of real data and generated data, then there are 2×K categories in the cross classifier. This cross classifier can be constructed as a model with 1 pull term and 2×K-1 push terms. Therefore, when training the generator, the cross classifier is required to classify the generated data into the corresponding categories, and the objective function of training the generator is as follows:

[0098]

[0099] d is q(X, Y=i) or p(X, Y=i) (the generated data distribution of the generated image X that satisfies the age condition Y). The larger it is, the closer G(x,y=k) is to the data distribution of the real data of the kth category; The larger (such as i≠k) G(x,y=k), the closer it is to the data distribution of the i-th category real data or the data distribution of the i-th category generated data.

[0100] In this way, the generator is trained by the above objective function, not only can it fit the target category of the real data It is also constrained to stay away from other categories in the true data distribution q(X,Y) and all categories in the generated data distribution p(X,Y). The cross classifier expands the number of thrust terms, thereby using more mutually exclusive classification hyperplanes to compress the search space and improve the performance of the model.

[0101] In the above embodiment, a deep generative network with age information is embedded in the whole processing process, so that the generated deep features contain as much conditional information as possible to guide the generation of pictures.

[0102] In the above embodiment, the shallow features and deep features generated by the sampler group complement each other. The attention feature map from the deep generative network weights the shallow features well. The use of a multi-branch dilated convolution sampler solves the difficult problem of changes across large age groups (large changes in additional conditions).

[0103] In the above embodiment, the discriminant network based on the cross classifier increases the number of thrust terms for network convergence under the guidance of mathematical theory, so that more classification hyperplanes of mutually exclusive categories can be effectively used to compress the search space and improve the performance of the model.

[0104] In the above embodiment, the loss function constrains identity consistency over the entire age sequence (sequence of additional conditions), does not require an additional network, and is very flexible and convenient.

[0105] In the above embodiment, the convolution processing result is weighted by using the attention feature map output by the residual learning method, so that the generation method can focus on the key areas in the image to be processed. In this way, a target image that can better meet the additional conditions can be generated, thereby improving the effect of the target image.

[0106] Figure 4 A block diagram showing some embodiments of an apparatus for generating a target image according to the present disclosure.

[0107] like Figure 4 As shown, the target image generating device 4 includes a residual learning unit 41, a convolution processing unit 42, a determination unit 43 and a generation unit 44.

[0108] The residual learning unit 41 determines the first feature map and the attention feature map of the image to be processed using a generative model according to a residual learning method.

[0109] In some embodiments, the residual learning unit 41 determines the first feature map and the attention feature map according to the first additional condition; the convolution processing unit determines the second feature map according to the first additional condition.

[0110] In some embodiments, the residual learning unit 41 adds a first additional condition to the residual learning method by using conditional instance regularization; the convolution processing unit adds a first additional condition to the convolution processing method by using conditional instance regularization.

[0111] The convolution processing unit 42 determines the second feature map of the image to be processed by using the generation model according to the convolution processing method.

[0112] In some embodiments, the convolution processing unit 42 performs convolution processing on each image region of the image to be processed respectively to determine the second feature map.

[0113] In some embodiments, the convolution processing unit 42 uses conditional instance regularization to add feature information of the first additional condition to each image region of the image to be processed; convolution processing is performed on each image region to which the feature information of the first additional condition is added to determine a second feature map.

[0114] The determination unit 43 determines the third feature map using a generation model based on the second feature map and the attention feature map.

[0115] The generating unit 44 generates a first target image that meets the first additional condition by using a generating model according to the third feature map, the first feature map and the feature information of the first additional condition.

[0116] In some embodiments, the feature information of the first additional condition is a feature map of the first additional condition; the generation unit 44 merges the third feature map, the first feature map and the feature map of the first additional condition into data to be processed; and generates a first target image based on the data to be processed.

[0117] In some embodiments, the generative model is trained using a loss function, and the loss function is determined by the following steps: according to the second additional condition, using the generative model to generate a second target image that satisfies the second additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a third target image that satisfies the third additional condition based on the image to be processed; according to the third additional condition, using the generative model to generate a fourth target image that satisfies the third additional condition based on the second target image; and determining the loss function based on the difference between the fourth target image and the third target image, and the difference between the image to be processed and the first target image.

[0118] In some embodiments, the generative model is trained using an objective function until the objective function reaches a maximum value. The objective function takes the first target image as a variable, is positively correlated with the target similarity, is negatively correlated with the sum of each real similarity, and is negatively correlated with the sum of each generated similarity.

[0119] In some embodiments, the target similarity is the similarity between the first target image and the real image that satisfies the first additional condition, the real similarity is the similarity between the first target image and the real image that satisfies other additional conditions, the generated similarity is the similarity between the first target image and the generated image that satisfies the first additional condition and other additional conditions, and the generated image is an image generated by the generation model according to the additional conditions.

[0120] In some embodiments, the target similarity is the similarity between the feature information of the first real data distribution and the feature information of the first target image. The first real data distribution is the data distribution of the real image that meets the first additional condition.

[0121] In some embodiments, the real similarity is the similarity between the feature information of other real data distributions and the feature information of the first target image. The other real data distributions are data distributions of real images that meet other additional conditions.

[0122] In some embodiments, the generated similarity is the similarity between the feature information of the generated data distribution and the feature information of the first target image, and the generated data distribution is the data distribution of the generated image that satisfies the first additional condition and other additional conditions.

[0123] In the above embodiment, the convolution processing result is weighted by using the attention feature map output by the residual learning method, so that the generation method can focus on the key areas in the image to be processed. In this way, a target image that can better meet the additional conditions can be generated, thereby improving the effect of the target image.

[0124] Figure 5 A block diagram showing some other embodiments of the device for generating a target image according to the present disclosure.

[0125] like Figure 5 As shown, the target image generation device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51, and the processor 52 is configured to execute the target image generation method in any embodiment of the present disclosure based on the instructions stored in the memory 51.

[0126] The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0127] Figure 6 A block diagram showing some further embodiments of the device for generating a target image according to the present disclosure.

[0128] like Figure 6 As shown, the target image generation device 6 of this embodiment includes: a memory 610 and a processor 620 coupled to the memory 610, and the processor 620 is configured to execute the target image generation method in any one of the aforementioned embodiments based on the instructions stored in the memory 610.

[0129] The memory 610 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, and other programs.

[0130] The target image generation device 6 may further include an input / output interface 630, a network interface 640, a storage interface 650, etc. These interfaces 630, 640, 650, the memory 610, and the processor 620 may be connected, for example, via a bus 660. The input / output interface 630 provides a connection interface for input / output devices such as a display, a mouse, a keyboard, and a touch screen. The network interface 640 provides a connection interface for various networked devices. The storage interface 650 provides a connection interface for external storage devices such as an SD card and a USB flash drive.

[0131] Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as methods, systems, or computer program products. Therefore, the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media containing computer-usable program code.

[0132] So far, the method for generating a target image, the device for generating a target image, and the computer-readable storage medium according to the present disclosure have been described in detail. In order to avoid obscuring the concept of the present disclosure, some details known in the art are not described. Based on the above description, those skilled in the art can fully understand how to implement the technical solution disclosed herein.

[0133] The method and system of the present disclosure may be implemented in many ways. For example, the method and system of the present disclosure may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present disclosure are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present disclosure may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present disclosure. Therefore, the present disclosure also covers a recording medium storing a program for executing the method according to the present disclosure.

[0134] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for generating a target image, comprising: According to the residual learning method, a first feature map and an attention feature map of the image to be processed are determined using a generative model, wherein the generative model includes a deep generative network, and the deep generative network includes a residual processing module and a sampling residual module, for generating the first feature map and the attention feature map; Determine a second feature map of the image to be processed using the generative model according to a convolution processing method; Determine a third feature map using the generative model according to the second feature map and the attention feature map; Generate a first target image that meets the first additional condition using the generation model according to the third feature map, the first feature map and feature information of the first additional condition; Wherein, determining the third feature map by using the generation model according to the second feature map and the attention feature map includes: weighting the second feature map using the attention feature map to determine the third feature map; Determining the first feature map and the attention feature map of the image to be processed by using the generative model includes: Determining the first feature map and the attention feature map according to the first additional condition; Determining the second feature map of the image to be processed by using the generative model includes: According to the first additional condition, the second feature map is determined.

2. The generation method according to claim 1, wherein: The determining, according to the first additional condition, the first feature map and the attention feature map comprises: Adding the first additional condition to the residual learning method by means of conditional instance regularization; The determining, according to the first additional condition, the second characteristic graph comprises: The first additional condition is added to the convolution processing method by using conditional instance regularization.

3. The generation method according to claim 1, wherein: Determining the second feature map of the image to be processed by using the generative model includes: Convolution processing is performed on each image region of the image to be processed to determine the second feature map.

4. The generation method according to claim 3, wherein: Determining the second feature map of the image to be processed by using the generative model includes: Using conditional instance regularization, respectively adding feature information of the first additional condition to each image region of the image to be processed; Convolution processing is performed on each image region to which the feature information of the first additional condition is added to determine the second feature map.

5. The generation method according to claim 3, wherein: The characteristic information of the first additional condition is a characteristic graph of the first additional condition; The generating a first target image that meets the first additional condition by using the generating model comprises: Merging the third feature map, the first feature map and the feature map of the first additional condition into data to be processed; The first target image is generated according to the data to be processed.

6. The generation method according to claim 1, wherein: The generative model is trained using a loss function, which is determined by the following steps: According to a second additional condition, using the generation model to generate a second target image that satisfies the second additional condition according to the image to be processed; According to a third additional condition, generating a third target image satisfying the third additional condition according to the image to be processed by using the generation model; According to the third additional condition, generating a fourth target image satisfying the third additional condition according to the second target image using the generation model; The loss function is determined according to a difference between the fourth target image and the third target image, and a difference between the image to be processed and the first target image.

7. The generation method according to any one of claims 1 to 5, wherein: The generation model is trained using an objective function until the objective function reaches a maximum value, wherein the objective function takes the first target image as a variable, is positively correlated with the target similarity, is negatively correlated with the sum of each real similarity, and is negatively correlated with the sum of each generated similarity, The target similarity is the similarity between the first target image and the real image that meets the first additional condition. The real similarity is the similarity between the first target image and the real image that meets other additional conditions. The generated similarity is the similarity between the first target image and a generated image that satisfies the first additional condition and the other additional conditions, and the generated image is an image generated by the generation model according to the additional conditions.

8. The generation method according to claim 7, wherein: The target similarity is the similarity between the feature information of the first real data distribution and the feature information of the first target image, and the first real data distribution is the data distribution of the real image that satisfies the first additional condition. The real similarity is the similarity between the feature information of other real data distribution and the feature information of the first target image, and the other real data distribution is the data distribution of the real image that meets the other additional conditions. The generated similarity is the similarity between feature information of generated data distribution and feature information of the first target image, and the generated data distribution is the data distribution of the generated image that meets the first additional condition and the other additional conditions.

9. The generation method according to claim 7, wherein: The generation model is trained based on the discrimination model on the differentiation result of the first target image and the real image satisfying the first additional condition.

10. A device for generating a target image, comprising: A residual learning unit, configured to determine a first feature map and an attention feature map of an image to be processed by using a generative model according to a residual learning method, wherein the generative model comprises a deep generative network, and the deep generative network comprises a residual processing module and a sampling residual module, configured to generate the first feature map and the attention feature map; A convolution processing unit, configured to determine a second feature map of the image to be processed using the generation model according to a convolution processing method; a determining unit, configured to determine a third feature map using the generation model according to the second feature map and the attention feature map; a generating unit, configured to generate a first target image that meets the first additional condition by using the generating model according to the third feature map, the first feature map and feature information of the first additional condition; The determining unit weights the second feature map using the attention feature map to determine the third feature map; The residual learning unit determines the first feature map and the attention feature map according to the first additional condition; The convolution processing unit determines the second feature map according to the first additional condition.

11. A device for generating a target image, comprising: Memory; and A processor coupled to the memory, wherein the processor is configured to execute the method for generating a target image according to any one of claims 1 to 9 based on instructions stored in the memory.

12. A computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for generating a target image according to any one of claims 1 to 9 is implemented.

Images