Image generation method, apparatus, device, and storage medium

Abstract

The present disclosure relates to an image generation method, device, equipment and storage medium. The method comprises: obtaining a first to-be-fused image and a target fused image; calculating a third hidden variable according to a pre-set fusion coefficient, a first hidden variable of the first to-be-fused image and a second hidden variable of the target fused image; and generating a second to-be-fused image based on the third hidden variable. According to the embodiment of the present disclosure, any one to-be-fused image and the fused image can be used to generate another to-be-fused image reversely, so that a new image generation special effect play can be obtained to meet the diversified image generation requirements of users.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image generation method, apparatus, device, and storage medium. Background Technology

[0002] Related technologies enable applications such as short video apps, shooting apps, and live streaming apps to offer image generation effects, allowing users to create the images they want. However, as user demands continue to increase, these image generation effects need to be constantly updated to meet diverse user needs. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides an image generation method, apparatus, device, and storage medium.

[0004] In a first aspect, this disclosure provides an image generation method, the method comprising:

[0005] Obtain the first image to be fused and the target image to be fused;

[0006] The third latent variable is calculated based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fusion image;

[0007] Based on the third latent variable, a second image to be fused is generated.

[0008] Secondly, this disclosure provides an image generation apparatus, the apparatus comprising:

[0009] The image acquisition module is used to acquire the first image to be fused and the target fusion image;

[0010] The third latent variable calculation module is used to calculate the third latent variable based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fused image.

[0011] The second image to be fused generation module is used to generate a second image to be fused based on the third hidden variable.

[0012] Thirdly, embodiments of this disclosure also provide an image generation apparatus, the apparatus comprising:

[0013] One or more processors;

[0014] Storage device for storing one or more programs.

[0015] When one or more programs are executed by one or more processors, the one or more processors implement the image generation method provided in the first aspect.

[0016] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image generation method provided in the first aspect.

[0017] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0018] An image generation method, apparatus, device, and storage medium according to embodiments of this disclosure can acquire a first image to be fused and a target image to be fused. The first image to be fused can be any image to be fused, and the target image to be fused can be a fused image. Based on a preset fusion coefficient, a first latent variable of the first image to be fused, and a second latent variable of the target image to be fused, a third latent variable is calculated. Thus, a third latent variable for generating another image to be fused can be obtained. Then, based on the third latent variable, a second image to be fused is generated. The second image to be fused can be another image to be fused. Therefore, another image to be fused can be generated in reverse using any image to be fused and a fused image, thereby obtaining new image generation effects to meet the diverse image generation needs of users. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0020] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic flowchart of an image generation method provided in this embodiment;

[0022] Figure 2 A schematic flowchart illustrating another image generation method provided in this embodiment of the present disclosure;

[0023] Figure 3 A schematic flowchart illustrating yet another image generation method provided in this disclosure embodiment;

[0024] Figure 4 A logical schematic diagram of an image generation method provided in an embodiment of this disclosure;

[0025] Figure 5 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of the present disclosure;

[0026] Figure 6 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this disclosure. Detailed Implementation

[0027] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0028] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0029] The related technologies provide short video applications, shooting applications, live streaming applications, and other applications that can offer image generation effects, allowing users to use these effects to create the images they want.

[0030] Current image generation effects can perform forward merging of two images to be combined, generating a merged image.

[0031] Taking the generation of a child's photo based on two photos of adults of the opposite sex as an example, the electronic device can acquire photos of an adult male and an adult female, extract the latent variables corresponding to the adult male photo and the adult female photo respectively, fuse the latent variables corresponding to the adult male photo and the adult female photo, and then input the fused latent variables into a pre-trained generative adversarial network (GAN) to generate an ideal child's photo.

[0032] As user demands continue to increase, users want to use an image to be merged and a merged image to reverse-engineer another image to be merged. For example, a user wants to use a given adult photo and an ideal child photo to reverse-engineer another adult photo whose gender is opposite to the given adult photo.

[0033] However, current image generation effects can only perform forward prediction based on two images to be merged, and cannot perform reverse prediction. Therefore, image generation effects need to be continuously updated to meet users' diverse image generation needs.

[0034] To generate another image to be merged from any image to be merged and a merged image, embodiments of this disclosure provide an image generation method, apparatus, device, and storage medium. This method acquires a first image to be merged and a target image to be merged. The first image to be merged is the image to be merged, and the target image to be merged is the merged image. Based on a pre-set fusion coefficient, a first latent variable of the first image to be merged, and a second latent variable of the target image to be merged, a third latent variable is calculated. This third latent variable is the latent variable of the other image to be merged. A second image to be merged is then generated based on this third latent variable and used as the other image to be merged. Therefore, it is possible to generate another image to be merged from any merged image and a merged image, resulting in new image generation effects and satisfying diverse user needs.

[0035] For example, image generation effects can be configured in live streaming applications. During a live stream, the streamer can use these effects to generate a photo of an ideal person of the opposite sex using their own photo and a photo of an ideal child. This allows the live streaming application to match the ideal photo of the opposite sex with the streamer's female fans, thereby increasing the interaction between the streamer and the fans.

[0036] The following is a combination of... Figures 1 to 4 The image generation method provided in the embodiments of this disclosure will be described.

[0037] Figure 1 A schematic flowchart of an image generation method provided in an embodiment of this disclosure is shown.

[0038] In some embodiments of this disclosure, Figure 1 The image generation method shown can be executed by an electronic device or a server. The electronic device can include devices with texture mapping capabilities, such as mobile phones, tablets, desktop computers, laptops, in-vehicle terminals, wearable devices, all-in-one machines, and smart home devices, or devices simulated by virtual machines or simulators. The server can be a cloud server or server cluster, or a device with storage and computing capabilities.

[0039] like Figure 1 As shown, the image generation method may include the following steps.

[0040] S110. Obtain the first image to be fused and the target image to be fused.

[0041] In the disclosed embodiments, when a user uses an image to generate special effects, a first image to be fused and a target image to be fused can be sent to an electronic device, so that the electronic device can acquire the first image to be fused and the target image to be fused, and generate a new image using the first image to be fused and the target image to be fused.

[0042] In this embodiment of the disclosure, the first image to be fused may be an image of a target person that needs to be fused.

[0043] Optionally, the target person can be a person, an animal, or a cartoon character; there are no restrictions.

[0044] In this embodiment of the disclosure, the target fused image can be a fused image of the target person.

[0045] Among them, the target person in the target fusion image and the target person in the first image to be fused belong to the same category.

[0046] In some embodiments, the similarity of partial features between the first image to be fused and the target image to be fused may be greater than a preset threshold. That is, the first image to be fused and the target image to be fused have some similar features.

[0047] Some of the features can be the features corresponding to the target person in the first image to be fused and the target fused image.

[0048] Optionally, some features may include at least one of the following: facial features, skin color, face shape, and hair color.

[0049] The similarity of some features can be the degree of similarity between some features of the first image to be fused and the target image to be fused.

[0050] The preset similarity threshold can be a similarity threshold that is set in advance as needed to determine whether some features are similar or identical.

[0051] Taking a first image to be fused as a photo of an adult woman and a target image to be fused as a photo of a child as an example, some features between the first image to be fused and the target image to be fused may include at least one of facial features, skin color features, face shape features, and hair color features. If the similarity of some features between the first image to be fused and the target image to be fused is greater than a preset similarity threshold, then an adult male photo can be generated in reverse based on some features or some similar features between the first image to be fused and the target image to be fused.

[0052] In other embodiments, the similarity of each feature between the first image to be fused and the target image to be fused is less than a preset threshold. That is, the first image to be fused and the target image to be fused do not have similar features.

[0053] Taking the first image to be fused as a photo of an adult woman and the target image to be fused as a photo of a child as an example again, the similarity of each feature between the first image to be fused and the target image to be fused is less than the preset similarity threshold. That is, an adult male photo can be generated in reverse by using any first image to be fused and any target image to be fused.

[0054] S120. Calculate the third latent variable based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fusion image.

[0055] In this embodiment of the disclosure, after the electronic device acquires the first image to be fused and the target image to be fused, it can find the first fusion coefficient and the second fusion coefficient from the preset fusion coefficients, and calculate the third latent variable based on the first fusion coefficient, the second fusion coefficient, the first latent variable and the second latent variable.

[0056] In this embodiment of the disclosure, the pre-set fusion coefficient can be composed of the first fusion coefficient corresponding to the first latent variable and the second fusion coefficient corresponding to the third latent variable, and the sum of the first fusion coefficient and the second fusion coefficient is 0.

[0057] In this embodiment of the disclosure, optionally, the first fusion coefficient and the first latent variable can be multiplied, and the second latent variable can be subtracted from the product. The result of the subtraction can then be divided by the second fusion coefficient to obtain the third latent variable.

[0058] Optionally, the third hidden variable can be implemented in the following way:

[0059]

[0060] Where P1 is the first latent variable and P2 is the third latent variable. The first fusion coefficient corresponds to the first latent variable. Let C be the second fusion coefficient corresponding to the third latent variable.

[0061] In this embodiment of the disclosure, the first latent variable can be a latent variable generated based on the features of the target person in the first image to be fused. Specifically, the first latent variable can be a latent variable to be fused.

[0062] In this embodiment of the disclosure, the second latent variable can be a latent variable generated based on the features of the target person in the target fused image. Specifically, the second latent variable can be the fused latent variable.

[0063] In some embodiments, after adjusting the parameter values ​​of the preset parameters corresponding to the target fused image to preset values, a second latent variable can be generated based on the first target feature of the target person in the target fused image.

[0064] The preset parameters can be pre-configured parameters used to adjust the first target features of the target fusion image.

[0065] Optionally, the preset parameter can be the age parameter.

[0066] Optionally, the preset value can be an adult age value. For example, the preset value can be 18 years old, 22 years old, etc.

[0067] The first target feature can be a facial feature used to generate the second latent variable.

[0068] Optionally, the first target features may include facial features, face shape features, skin color features, and hair color features, etc.

[0069] Taking the first image to be fused as a photo of an adult woman and the target image to be fused as a photo of a child as an example, the preset parameter is the age parameter, and the parameter value is the age value. After adjusting the age value of the child in the child's photo to 18 years old, the second latent variable can be generated based on the facial features of the child in the child's photo. After adjusting the child's appearance to an adult appearance, the latent variable can be extracted, which can avoid the impact of the age difference between the child and the adult woman on the accuracy of the third latent variable.

[0070] In other embodiments, latent variables can be generated directly based on the features of the target person in the target fused image, such that the second latent variable is a latent variable whose feature value of the first target feature is not adjusted to the target feature value.

[0071] Taking the first image to be fused as a photo of an adult woman and the target image to be fused as a photo of a child as an example again, the second latent variable can be directly generated based on the features of the child in the photo, so that the second latent variable can be extracted directly from the child's appearance features.

[0072] In this embodiment of the disclosure, the third latent variable may be a target latent variable used to fuse with the first latent variable to generate the second latent variable.

[0073] Specifically, the electronic device can calculate the third latent variable based on the first fusion coefficient corresponding to the first latent variable, the second fusion coefficient corresponding to the third latent variable, and the second latent variable.

[0074] S130. Based on the third latent variable, generate the second image to be fused.

[0075] In this embodiment of the disclosure, after the electronic device determines the third hidden variable, it can generate the second image to be fused in reverse based on the third hidden variable.

[0076] In this embodiment of the disclosure, the second image to be fused may be another image to be fused that is fused with the first image to be fused to generate the target fused image.

[0077] In this embodiment of the disclosure, optionally, a third latent variable can be input into the image generation model to generate a second image to be fused.

[0078] In this embodiment, a first image to be fused and a target image to be fused can be obtained. The first image to be fused can be any image that needs to be fused, and the target image to be fused can be a fused image. A third latent variable is calculated based on a pre-set fusion coefficient, a first latent variable of the first image to be fused, and a second latent variable of the target image to be fused. Thus, a third latent variable for generating another image to be fused can be obtained. Then, based on the third latent variable, a second image to be fused is generated. The second image to be fused can be another image that needs to be fused. Therefore, another image to be fused can be generated in reverse using any image to be fused and a fused image. This allows for new image generation effects to meet the diverse image generation needs of users.

[0079] In another embodiment of this disclosure, a pre-trained encoder can be used to generate a first latent variable corresponding to the first image to be fused and a second latent variable corresponding to the target fused image, thereby improving the timeliness of latent variable extraction and further improving the image generation efficiency.

[0080] Figure 2 A schematic flowchart of another image generation method provided in an embodiment of this disclosure is shown.

[0081] like Figure 2 As shown, the image generation method may include the following steps.

[0082] S210. Obtain the first image to be fused and the target image to be fused.

[0083] S210 is similar to S110, and will not be described in detail here.

[0084] S220. Input the first image to be fused into the pre-trained first encoder to obtain the first latent variable.

[0085] In this embodiment of the disclosure, after the electronic device acquires the first image to be fused, it can input the first image to be fused into a pre-trained first encoder to extract the first latent variable corresponding to the first image to be fused.

[0086] The first encoder can be an encoder model trained based on the sample image to be fused and the latent variables corresponding to the sample image to be fused.

[0087] S230. Input the target fused image into the pre-trained second encoder to obtain the second latent variable.

[0088] In this embodiment of the disclosure, after the electronic device acquires the target fused image, it can input the target fused image into a pre-trained second encoder to extract the second latent variable corresponding to the target fused image.

[0089] The second encoder can be an encoder model trained based on the sample fusion image and the latent variables corresponding to the sample fusion image.

[0090] Optionally, in this embodiment of the disclosure, S230 may specifically include the following steps:

[0091] S2301. Input the target fused image into the pre-trained second encoder. Based on the preset parameters in the second encoder, adjust the feature value of the first target feature of the target fused image to the target feature value to obtain the second latent variable.

[0092] Specifically, after the electronic device acquires the target fused image, it can input the target fused image into a pre-trained second encoder. Using the preset parameters in the second encoder, the feature values ​​of the first target feature of the target fused image are adjusted to the target feature values ​​to obtain the second latent variable.

[0093] The first target feature can be a facial feature used to generate the second latent variable. Optionally, the first target feature may include facial features, face shape features, skin color features, and hair color features, etc.

[0094] The preset parameters can be pre-configured parameters used to adjust the feature values ​​of the first target feature in the target fusion image.

[0095] Optionally, the preset parameter can be the age parameter.

[0096] The target feature value can be the feature value corresponding to the first target feature when the preset parameters are adjusted to the preset value.

[0097] Optionally, the preset value can be an adult's age. For example, the preset value could be 18 years old, 22 years old, etc. Correspondingly, the target feature value is the facial feature value of an adult.

[0098] Taking the age parameter as the preset parameter and the facial features as the first target feature as the target fusion image as a child's photo, the child's age is 3 years old, and the age parameter is set to 20 years old. The target fusion image is input into the second encoder. Based on the target fusion encoder and the age parameter, the facial features of the child in the photo are adjusted to the facial features of a 20-year-old. Furthermore, the latent variables corresponding to the facial features of the child at age 20 are determined, and the latent variables corresponding to the facial features of the child at age 20 are used as the second latent variables.

[0099] Therefore, in this embodiment of the disclosure, the age parameter in the second encoder can be used to adjust the first target feature of the target fusion image, so as to avoid the accuracy of the third latent variable being affected by the large age difference between the second latent variable and the first latent variable.

[0100] In summary, in this embodiment of the present disclosure, a trained encoder can be used to extract the first latent variable of the first image to be fused and the second latent variable of the target fused image, respectively. Compared with the traditional latent variable extraction methods such as the Hidden Markov Model (HMM) in the existing methods, the extraction time of latent variables can be shortened, so as to achieve the effect of online extraction of latent variables and further improve the efficiency of image generation.

[0101] S240. Calculate the third latent variable based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fusion image.

[0102] Optionally, in this embodiment, S240 may specifically include the following steps:

[0103] S2401. Find the first fusion coefficient corresponding to the first hidden variable and the second fusion coefficient corresponding to the third hidden variable from the preset fusion coefficients;

[0104] S2402. Calculate the predicted latent variables based on the first latent variable, the second latent variable, the first fusion coefficient, and the second fusion coefficient.

[0105] S2403. Based on the pre-set latent variable intervals and estimated latent variables corresponding to each dimension feature, determine the third latent variable.

[0106] Specifically, after the electronic device determines the first and second latent variables, it can find the first fusion coefficient corresponding to the first latent variable and the second fusion coefficient corresponding to the third latent variable from the pre-set fusion coefficients. Based on the first fusion coefficient, the second fusion coefficient, the first latent variable, and the second latent variable, the predicted latent variable is calculated. Then, based on the relationship between the feature values ​​of the predicted latent variable and the feature values ​​of the latent variable intervals corresponding to the pre-set features of each dimension, the third latent variable is determined.

[0107] Optionally, latent variables can be estimated in the following ways:

[0108]

[0109] Among them, P2 represents the predicted latent variables.

[0110] The intervals of latent variables corresponding to each dimension feature can be reasonable intervals of latent variables corresponding to each dimension feature that are pre-set as needed.

[0111] In some embodiments, S2403 may specifically include the following steps:

[0112] S1. If there is a first latent variable in the latent variables, adjust the feature value corresponding to the first latent variable to the feature value corresponding to the smallest latent variable in the latent variable interval. The feature value corresponding to the first latent variable is less than the feature value corresponding to the smallest latent variable in the latent variable interval. The smallest latent variable is the smallest latent variable in the first latent variable or the smallest latent variable in the second latent variable.

[0113] S2. The first predicted latent variable after eigenvalue adjustment and the second predicted latent variable located within the latent variable interval are taken as the third latent variable.

[0114] Specifically, after the electronic device determines the predicted latent variables, it compares the eigenvalues ​​corresponding to each predicted latent variable with the eigenvalues ​​corresponding to the smallest latent variable in the latent variable interval. The smallest latent variable is either the smallest latent variable in the first latent variable or the smallest latent variable in the second latent variable. If there is a predicted latent variable that is smaller than the smallest latent variable in the first latent variable or the smallest latent variable in the second latent variable, then the predicted latent variable smaller than the smallest latent variable in the first latent variable or the smallest latent variable in the second latent variable is taken as the first predicted latent variable, and the eigenvalue corresponding to the first predicted latent variable is adjusted to the smallest latent variable in the second latent variable. Then, the first predicted latent variable with the adjusted eigenvalue and the second predicted latent variable located in the latent variable interval are taken as the third latent variable.

[0115] Among them, the first predicted latent variable is the abnormal latent variable that is outside the reasonable range of the predicted latent variables.

[0116] Among them, the second predicted latent variable is the reasonable latent variable that is within the reasonable range of the predicted latent variables.

[0117] Optionally, adjusting the first predicted latent variable can be achieved in the following way:

[0118] Latent variable interval: [min(C i P 1i ), max(C i P 1i If P 2i <min(C i P 1i If P, then 2i =C i .

[0119] Among them, C i P is the second hidden variable. 1i P is the first hidden variable. 2i To predict latent variables, min(C) i P 1i ) represents the minimum value of the smallest latent variable among the first latent variables and the smallest latent variable among the second latent variables, max(C) iP 1i ) represents the maximum value of the largest hidden variable in the first hidden variable and the largest hidden variable in the second hidden variable.

[0120] Taking the first latent variable as the latent variable corresponding to the adult female photo and the second latent variable as the latent variable corresponding to the child photo as an example, the predicted latent variable is the latent variable corresponding to the adult male photo. If the first predicted latent variable exists in the predicted latent variables, since the probability that the adult male photo and the child photo have the same features is relatively high, the feature value corresponding to the first predicted latent variable is adjusted to the feature value corresponding to the smallest latent variable in the second latent variable. This makes the feature value corresponding to the first predicted latent variable of the adult male the same as the feature value corresponding to the smallest latent variable in the second latent variable of the child. The first predicted latent variable with the adjusted feature value and the second predicted latent variable located in the latent variable interval are used as the third latent variable, so that more predicted latent variables are retained, further improving the rationality of the second image to be fused.

[0121] In other embodiments, S2403 may specifically include the following steps:

[0122] S3. If there is no first latent variable among the predicted latent variables, the second latent variable that is located within the latent variable interval shall be taken as the second latent variable.

[0123] Specifically, after the electronic device determines the predicted latent variables, it compares the eigenvalues ​​corresponding to each predicted latent variable with the eigenvalues ​​corresponding to the smallest latent variable in the latent variable interval. The smallest latent variable is either the smallest latent variable in the first latent variable or the smallest latent variable in the second latent variable. If there is no predicted latent variable smaller than the smallest latent variable in the first latent variable or the first smallest latent variable in the second latent variable, then the second predicted latent variable located in the latent variable interval is directly used as the third latent variable.

[0124] Therefore, in this embodiment of the disclosure, if a first estimated latent variable exists among the estimated latent variables, the eigenvalue corresponding to the first estimated latent variable is adjusted to the eigenvalue corresponding to the smallest latent variable among the second latent variables, so as to simultaneously retain both the adjusted abnormal latent variables and the reasonable latent variables among the estimated latent variables, thereby retaining more estimated latent variables. If the first estimated latent variable does not exist among the estimated latent variables, then all the estimated latent variables are reasonable latent variables, and the reasonable latent variables are retained. Therefore,

[0125] The above method can make the obtained third latent variable more comprehensive, further improving the rationality of the second image to be fused.

[0126] S250. Based on the third latent variable, generate the second image to be fused.

[0127] S250 is similar to S130, so it will not be described in detail here.

[0128] In another embodiment of this disclosure, the second image to be fused can be generated directly using the first image generation model, or the second image to be fused can be generated by combining the discriminator and the second image generation model.

[0129] Figure 3 A schematic flowchart of another image generation method provided in an embodiment of this disclosure is shown.

[0130] like Figure 3 As shown, the image generation method may include the following steps.

[0131] S310. Obtain the first image to be fused and the target image to be fused.

[0132] S310 is similar to S110, so it will not be described in detail here.

[0133] S320. Calculate the third latent variable based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fusion image.

[0134] S320 is similar to S120, so it will not be described in detail here.

[0135] S330. Input the third latent variable into the pre-trained first generative model to obtain the second image to be fused.

[0136] Specifically, after the electronic device determines the third latent variable, it can use the first generative model to directly generate the second image to be fused based on the third latent variable.

[0137] The first generative model can be a pre-trained image generation model. Specifically, the first generative model can be trained using the latent variables of the samples to be fused and the images of the samples to be fused.

[0138] Optionally, the first generative model can be a generator based on a style-based generative adversarial network (StyleGAN) or other network models, without any restrictions.

[0139] Continuing with the example of the first image to be fused being a photo of an adult woman and the target image to be fused being a photo of a child, after the electronic device obtains the third latent variable, it inputs the third latent variable into the pre-trained first generative model to obtain the second image to be fused, and then directly outputs a photo of an adult man.

[0140] S340. Input the third latent variable into the pre-trained first generative model to obtain candidate images.

[0141] Specifically, after the electronic device determines the third latent variable, it can use the first generative model to generate candidate images based on the third latent variable, and then generate the second image to be fused based on the candidate images.

[0142] The candidate image can be an image generated based on a third latent variable to generate the second image to be fused.

[0143] S350. Input the candidate image into the pre-trained discriminant network to obtain the probability value of the second target feature in the candidate image.

[0144] Specifically, after the electronic device generates a candidate image, it can input the candidate image into a pre-trained discriminant network and use the discriminant network to predict the probability value of the second target feature in the candidate image.

[0145] The discriminant network can be a pre-trained discriminant model used to predict the features of the second target. Specifically, the discriminant network can be trained using sample images and the features of the second target in the samples.

[0146] Optionally, the discriminant network can be a discriminator of StyleGAN or other network models; there are no restrictions here.

[0147] The second target feature can be a feature that is pre-set as needed to distinguish it from the first image to be fused.

[0148] Optionally, the second target feature can be a gender feature.

[0149] In some embodiments, if the second target feature of the first image to be fused is female, then the second target feature of the second image to be fused is male.

[0150] In other embodiments, if the second target feature of the first image to be fused is male, then the second target feature of the second image to be fused is female.

[0151] S360. If the probability value of the second target feature is greater than the preset probability value threshold, the candidate image is used as the second image to be fused.

[0152] Specifically, after the electronic device calculates the probability value of the second target feature, it compares the probability value of the second target feature with a preset probability value threshold. If the probability value of the second target feature is greater than the preset probability value threshold, the candidate image is used as the second image to be fused. If the probability value of the second target feature is less than the preset probability value threshold, the second target feature conversion network is used to output the second image to be fused carrying the second target feature.

[0153] The probability threshold can be a probability value that is preset as needed to determine the features of the second target.

[0154] S370. If the probability value of the second target feature is less than the preset probability value threshold, the third latent variable is input into the pre-trained second generative model to obtain the second image to be fused carrying the second target feature.

[0155] Specifically, after the electronic device calculates the probability value of the second target feature, it compares the probability value of the second target feature with a preset probability value threshold. If the probability value of the second target feature is less than the preset probability value threshold, the third latent variable is input into the pre-trained second generative model to obtain the second image to be fused carrying the second target feature.

[0156] The second generative model can be a pre-trained image generation model. Specifically, the second generative model can be trained using the latent variables of the samples to be fused and the images of the samples to be fused carrying the second target features.

[0157] Optionally, the second generative model can be a StyleGAN generator or other network models; there are no restrictions on this.

[0158] Continuing with the example of the first image to be fused being a photo of an adult woman and the target image to be fused being a photo of a child, after the electronic device obtains the third latent variable, it inputs the third latent variable into the pre-trained first generative model to obtain a candidate image. The candidate image is then input into the pre-trained discriminative network to obtain the probability value of the gender being male. If the probability value is greater than a pre-set probability threshold, the candidate image is used as the second image to be fused. If the probability value is less than the pre-set probability threshold, the third latent variable is input into the pre-trained second generative model, which outputs the second image to be fused, indicating that the gender is male.

[0159] Therefore, in this embodiment, a pre-trained first generative model can be used to directly generate the second image to be fused. Alternatively, the image output by the first generative model can be used as a candidate image, and a discriminative network can be used to calculate the probability value of the second target feature in the candidate image. If the probability value of the second target feature is greater than a preset probability threshold, the candidate image is used as the second image to be fused. If the probability value of the second target feature is less than the preset probability threshold, the second model is used to generate the second image to be fused carrying the second target feature, thereby improving the accuracy of the second image to be fused. Thus, different methods can be used to generate the second image to be fused, improving the flexibility of the second image to be fused generation.

[0160] Figure 4 A logical schematic diagram of an image generation method provided by an embodiment of this disclosure is shown.

[0161] like Figure 4As shown, the first image to be fused is a photo of an adult woman, and the target fusion image is a photo of a child. Based on the adult woman's photo and the child's photo, an adult male photo is generated. First, the first image to be fused is input into a pre-trained first encoder to obtain the first latent variable P1 corresponding to the first image to be fused. Then, the target fusion image is input into a pre-trained second encoder to obtain the second latent variable C. Next, random fusion coefficients are obtained, including the random fusion coefficient corresponding to the first latent variable P1. And based on the random fusion coefficient corresponding to the first latent variable P1, Given the first latent variable P1 and the second latent variable C, calculate the predicted latent variable P2. Next, based on the pre-set latent variable intervals and estimated latent variables corresponding to each dimension of features, the third latent variable P2 is determined. Finally, the third latent variable P2 is input into the pre-trained first generative model to directly obtain the second image to be fused. This first generative model is the generator of StyleGAN. Alternatively, the third latent variable is input into the pre-trained first generative model to obtain a candidate image. The candidate image is then input into a pre-trained discriminant network to obtain the probability value of the second target feature in the candidate image. This discriminant network is the discriminator of StyleGAN. If the probability value of the second target feature is greater than a pre-set probability threshold, the candidate image is used as the second image to be fused. If the probability value of the second target feature is less than the pre-set probability threshold, the third latent variable is input into the pre-trained second generative model to obtain the second image to be fused carrying the second target feature. This second generative model is the generator of StyleGAN.

[0162] This disclosure also provides an image generation apparatus for implementing the above-described image generation method, which will be described below in conjunction with... Figure 5 The following explanation is provided. In this embodiment, the image generation device can be an electronic device or a server. The electronic device can include mobile terminals, tablet computers, vehicle-mounted terminals, wearable electronic devices, virtual reality (VR) all-in-one machines, smart home devices, and other devices with communication functions. The server can be a cloud server or server cluster, or other devices with storage and computing functions.

[0163] Figure 5 A schematic diagram of the structure of an image generation apparatus provided in an embodiment of this disclosure is shown.

[0164] like Figure 5 As shown, the image generation device 500 may include: an image acquisition module 510, a third latent variable calculation module 520, and a second image to be fused generation module 530.

[0165] Image acquisition module 510 is used to acquire the first image to be fused and the target fusion image;

[0166] The third latent variable calculation module 520 is used to calculate the third latent variable based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fused image.

[0167] The second image to be fused generation module 530 is used to generate a second image to be fused based on a third latent variable.

[0168] In this embodiment, a first image to be fused and a target image to be fused can be obtained. The first image to be fused can be any image that needs to be fused, and the target image to be fused can be a fused image. A third latent variable is calculated based on a pre-set fusion coefficient, a first latent variable of the first image to be fused, and a second latent variable of the target image to be fused. Thus, a third latent variable for generating another image to be fused can be obtained. Then, based on the third latent variable, a second image to be fused is generated. The second image to be fused can be another image that needs to be fused. Therefore, another image to be fused can be generated in reverse using any image to be fused and a fused image. This allows for new image generation effects to meet the diverse image generation needs of users.

[0169] Optionally, the device may further include a first latent variable generation module and a second latent variable generation module;

[0170] The first latent variable generation module is used to input the first image to be fused into the pre-trained first encoder to obtain the first latent variable;

[0171] The second latent variable generation module is used to input the target fused image into the pre-trained second encoder to obtain the second latent variable.

[0172] Optionally, the second latent variable generation module is specifically used to input the target fused image into the pre-trained second encoder, adjust the feature value of the first target feature of the target fused image to the target feature value, and obtain the second latent variable.

[0173] Optionally, the third latent variable calculation module 520 includes: a fusion coefficient lookup unit, a latent variable prediction calculation unit, and a third latent variable determination unit;

[0174] The fusion coefficient lookup unit can be used to find the first fusion coefficient corresponding to the first hidden variable and the second fusion coefficient corresponding to the third hidden variable from the preset fusion coefficients;

[0175] The latent variable estimation unit can be used to calculate the latent variable based on the first latent variable, the second latent variable, the first fusion coefficient, and the second fusion coefficient.

[0176] The third latent variable calculation unit can be used to determine the third latent variable based on the latent variable intervals and estimated latent variables corresponding to each dimension feature in a pre-set manner.

[0177] Optionally, the third latent variable determination unit is specifically used to adjust the feature value corresponding to the first latent variable to the feature value corresponding to the smallest latent variable in the second latent variable when there is a first latent variable in the latent variables. The feature value corresponding to the first latent variable is less than the feature value corresponding to the smallest latent variable in the latent variable interval. The smallest latent variable is the smallest latent variable in the first latent variable or the smallest latent variable in the second latent variable.

[0178] The first predicted latent variable after eigenvalue adjustment, and the second predicted latent variable located within the latent variable interval, are taken as the third latent variable.

[0179] Optionally, the third latent variable determination unit is specifically used to, when there is no first latent variable among the latent variables, take the second latent variable located within the latent variable interval as the second latent variable.

[0180] Optionally, the second image to be fused generation module 530 is specifically used to input the third latent variable into the pre-trained first generation model to obtain the second image to be fused.

[0181] Optionally, the second image to be fused generation module 530 includes: a candidate image generation unit, a probability value calculation unit, and a second image to be fused generation unit;

[0182] The candidate image generation unit can be used to input the third latent variable into the pre-trained first generation model to obtain candidate images;

[0183] The probability value calculation unit can be used to input candidate images into a pre-trained discrimination network to obtain the probability value of the second target feature in the candidate image;

[0184] The second image generation unit can be used to generate a candidate image as the second image to be fused if the probability value of the second target feature is greater than a preset probability value threshold.

[0185] Optionally, the second image generation unit is further configured to, if the probability value of the second target feature is less than a preset probability value threshold, input the third latent variable into the pre-trained second generation model to obtain the second image to be fused carrying the second target feature.

[0186] It should be noted that, Figure 5 The image generation device 500 shown can perform... Figures 1 to 4 The various steps in the method embodiment shown are implemented. Figures 1 to 4The processes and effects in the method embodiments shown are not described in detail here.

[0187] Figure 6 A schematic diagram of the structure of an image generation device provided in an embodiment of this disclosure is shown.

[0188] like Figure 6 As shown, the image generating device may include a processor 601 and a memory 602 storing computer program instructions.

[0189] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0190] Memory 602 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway device. In a particular embodiment, memory 602 is a non-volatile solid-state memory. In a particular embodiment, memory 602 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0191] The processor 601 reads and executes computer program instructions stored in the memory 602 to perform the steps of the image generation method provided in the embodiments of this disclosure.

[0192] In one example, the image generating device may further include a transceiver 603 and a bus 604. Wherein, as... Figure 6As shown, the processor 601, memory 602 and transceiver 603 are connected via bus 604 and communicate with each other.

[0193] Bus 604 includes hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0194] The following are embodiments of a computer-readable storage medium provided in this disclosure. This computer-readable storage medium belongs to the same inventive concept as the image generation methods in the above embodiments. For details not described in detail in the embodiments of the computer-readable storage medium, please refer to the embodiments of the above image generation methods.

[0195] This embodiment provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image generation method, the method comprising:

[0196] Obtain the first image to be fused and the target image to be fused;

[0197] The third latent variable is calculated based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fusion image;

[0198] Based on the third latent variable, a second image to be fused is generated.

[0199] Of course, the computer-executable instructions provided in the embodiments of this disclosure are not limited to the above-described method operations, but can also perform related operations in the image generation method provided in any embodiment of this disclosure.

[0200] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this disclosure can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer cloud platform (which may be a personal computer, a server, or a network cloud platform, etc.) to execute the image generation methods provided in the various embodiments of this disclosure.

[0201] Note that the above description is merely a preferred embodiment and the technical principles employed in this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, it is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this disclosure, and the scope of this disclosure is determined by the scope of the appended claims.

Claims

1. An image generation method, characterized in that, include: Obtain the first image to be fused and the target image to be fused; A third latent variable is calculated based on a pre-set fusion coefficient, a first latent variable of the first image to be fused, and a second latent variable of the target fused image. The first latent variable is generated based on the features of the target object in the first image to be fused, and the second latent variable is generated based on the features of the target object in the target fused image after the parameter value of the preset parameter corresponding to the target fused image is adjusted to a preset value. Based on the third latent variable, a second image to be fused is generated.

2. The method according to claim 1, characterized in that, Before calculating the third latent variable based on the preset fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fusion image, the method further includes: The first image to be fused is input into a pre-trained first encoder to obtain the first latent variable; The target fused image is input into a pre-trained second encoder to obtain the second latent variable.

3. The method according to claim 2, characterized in that, The step of inputting the target fused image into a pre-trained second encoder to obtain the second latent variable includes: The target fused image is input into a pre-trained second encoder. Based on the preset parameters in the second encoder, the feature value of the first target feature of the target fused image is adjusted to the target feature value to obtain the second latent variable.

4. The method according to claim 1, characterized in that, The step of calculating the third latent variable based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fusion image includes: Find the first fusion coefficient corresponding to the first hidden variable and the second fusion coefficient corresponding to the third hidden variable from the preset fusion coefficients; Calculate the predicted latent variable based on the first latent variable, the second latent variable, the first fusion coefficient, and the second fusion coefficient; The third latent variable is determined based on the pre-set latent variable intervals corresponding to each dimension feature and the estimated latent variable.

5. The method according to claim 4, characterized in that, The process of determining the third latent variable based on the pre-set latent variable intervals corresponding to each dimension of features and the estimated latent variables includes: If a first latent variable exists among the predicted latent variables, the feature value corresponding to the first latent variable is adjusted to the feature value corresponding to the smallest latent variable among the second latent variables. The feature value corresponding to the first latent variable is less than the feature value corresponding to the smallest latent variable in the latent variable interval. The smallest latent variable is either the smallest latent variable among the first latent variables or the smallest latent variable among the second latent variables. The first estimated latent variable after eigenvalue adjustment and the second estimated latent variable located within the latent variable interval are taken as the third latent variable.

6. The method according to claim 4, characterized in that, The process of determining the third latent variable based on the pre-set latent variable intervals corresponding to each dimension of features and the estimated latent variables includes: If the first estimated latent variable is not present among the estimated latent variables, the second estimated latent variable that is located within the interval of the latent variables is taken as the second latent variable.

7. The method according to claim 1, characterized in that, The step of generating a second image to be fused based on the third latent variable includes: The third latent variable is input into the pre-trained first generative model to obtain the second image to be fused.

8. The method according to claim 1, characterized in that, The step of generating a second image to be fused based on the third latent variable includes: The third latent variable is input into the pre-trained first generative model to obtain candidate images; The candidate image is input into a pre-trained discriminant network to obtain the probability value of the second target feature in the candidate image; If the probability value of the second target feature is greater than a preset probability value threshold, the candidate image is used as the second image to be fused.

9. The method according to claim 8, characterized in that, After inputting the candidate image into a pre-trained discriminant network to obtain the probability value of the second target feature in the candidate image, the method further includes: If the probability value of the second target feature is less than the preset probability value threshold, the third latent variable is input into the pre-trained second generative model to obtain a second image to be fused carrying the second target feature.

10. An image generation apparatus, characterized in that, include: The image acquisition module is used to acquire the first image to be fused and the target fusion image; The third latent variable calculation module is used to calculate the third latent variable based on the pre-set fusion coefficient, the first latent variable of the first image to be fused, and the second latent variable of the target fused image. The first latent variable is generated based on the features of the target object in the first image to be fused, and the second latent variable is generated based on the features of the target object in the target fused image after the parameter value of the preset parameter corresponding to the target fused image is adjusted to the preset value. The second image to be fused generation module is used to generate a second image to be fused based on the third hidden variable.

11. An image generation device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image generation method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the image generation method according to any one of claims 1-9.

Images