Image generation method, apparatus, device, and storage medium

Abstract

The present disclosure relates to an image generation method, device, equipment and storage medium. The method comprises: obtaining a first to-be-fused image and a target fused image; generating a candidate to-be-fused image based on a first hidden variable of the first to-be-fused image and a randomly generated second hidden variable; adjusting the second hidden variable based on a feature difference between the target fused image and the candidate to-be-fused image; and generating a second to-be-fused image based on the adjusted second hidden variable in a case where the feature difference is not greater than a preset threshold. According to the embodiment of the present disclosure, any one to-be-fused image and the fused image can be used to generate another to-be-fused image, thereby obtaining a new image generation special effect play to meet the diversified image generation needs of users.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to an image generation method, apparatus, device, and storage medium. Background Technology

[0002] Related technologies enable applications such as short video apps, shooting apps, and live streaming apps to offer image generation effects, allowing users to create the images they want. However, as user demands continue to increase, these image generation effects need to be constantly updated to meet diverse user needs. Summary of the Invention

[0003] To solve the above-mentioned technical problems, or at least partially solve them, this disclosure provides an image generation method, apparatus, device, and storage medium.

[0004] In a first aspect, this disclosure provides an image generation method, the method comprising:

[0005] Obtain the first image to be fused and the target image to be fused;

[0006] Based on the first latent variable of the first image to be fused and the randomly generated second latent variable, candidate images to be fused are generated.

[0007] The second latent variable is adjusted based on the feature differences between the target fusion image and the candidate images to be fused.

[0008] If the feature difference is no greater than a preset threshold, a second image to be fused is generated based on the adjusted second latent variable.

[0009] Secondly, this disclosure provides an image generation apparatus, the apparatus comprising:

[0010] The image acquisition module is used to acquire the first image to be fused and the target fusion image;

[0011] The candidate image to be fused module is used to generate candidate images to be fused based on the first latent variable of the first image to be fused and the randomly generated second latent variable.

[0012] The second latent variable adjustment module is used to adjust the second latent variable based on the feature differences between the target fused image and the candidate images to be fused.

[0013] The second image to be fused generation module is used to generate a second image to be fused based on the adjusted second latent variable, provided that the feature difference is not greater than a preset threshold.

[0014] Thirdly, embodiments of this disclosure also provide an image generation apparatus, the apparatus comprising:

[0015] One or more processors;

[0016] Storage device for storing one or more programs.

[0017] When the one or more programs are executed by the one or more processors, the one or more processors implement the image generation method provided in the first aspect.

[0018] Fourthly, embodiments of this disclosure also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image generation method provided in the first aspect.

[0019] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0020] An image generation method, apparatus, device, and storage medium disclosed herein can acquire a first image to be fused and a target image to be fused. The first image to be fused can be any image to be fused, and the target image to be fused can be a fused image. Based on a first latent variable of the first image to be fused and a randomly generated second latent variable, a candidate image to be fused is generated. Then, based on the feature differences between the target image to be fused and the candidate image to be fused, the second latent variable is adjusted until the feature difference between the candidate image to be fused and the target image to be fused, generated based on the first latent variable and the adjusted second latent variable, is less than a preset threshold. Further, if the feature difference is not greater than the preset threshold, a second image to be fused is generated based on the adjusted second latent variable. Therefore, any image to be fused and a fused image can be used to generate another image to be fused, resulting in new image generation effects and satisfying diverse image generation needs of users. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0022] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic flowchart of an image generation method provided in this embodiment;

[0024] Figure 2 A schematic flowchart illustrating another image generation method provided in this embodiment of the present disclosure;

[0025] Figure 3 A schematic flowchart illustrating yet another image generation method provided in this disclosure embodiment;

[0026] Figure 4 A logical schematic diagram of an image generation method provided in an embodiment of this disclosure;

[0027] Figure 5 This is a schematic diagram of the structure of an image generation apparatus provided in an embodiment of the present disclosure;

[0028] Figure 6 This is a schematic diagram of the structure of an image generation device provided in an embodiment of this disclosure. Detailed Implementation

[0029] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0030] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0031] The related technologies provide short video applications, shooting applications, live streaming applications, and other applications that can offer image generation effects, allowing users to use these effects to create the images they want.

[0032] Current image generation effects can perform forward merging of two images to be combined, generating a merged image.

[0033] Taking the generation of a child's photo based on two photos of adults of the opposite sex as an example, the electronic device can acquire photos of an adult male and an adult female, extract the latent variables corresponding to the adult male photo and the adult female photo respectively, fuse the latent variables corresponding to the adult male photo and the adult female photo, and then input the fused latent variables into a pre-trained generative adversarial network (GAN) to generate an ideal child's photo.

[0034] As user demands continue to increase, users want to use an image to be merged and a merged image to reverse-engineer another image to be merged. For example, a user wants to use a given adult photo and an ideal child photo to reverse-engineer another adult photo whose gender is opposite to the given adult photo.

[0035] However, current image generation effects can only perform forward prediction based on two images to be merged, and cannot perform reverse prediction. Therefore, image generation effects need to be continuously updated to meet users' diverse image generation needs.

[0036] To generate another image to be merged based on any image to be merged and a merged image, embodiments of this disclosure provide an image generation method, apparatus, device, and storage medium. This method acquires a first image to be merged and a target image to be merged. The first image to be merged is the image to be merged, and the target image to be merged is the merged image. Based on a first latent variable of the first image to be merged and a randomly generated second latent variable, candidate images to be merged are generated. Then, based on the target image to be merged and the candidate images to be merged, the second latent variable is adjusted so that, provided the feature difference is not greater than a preset threshold, a second image to be merged is generated based on the adjusted second latent variable. This second image to be merged is used as the other image to be merged. Therefore, any image to be merged and a merged image can be used to generate another image to be merged, resulting in new image generation effects and satisfying diverse user needs.

[0037] For example, image generation effects can be configured in live streaming applications. During a live stream, the streamer can use these effects to generate a photo of an ideal person of the opposite sex using their own photo and a photo of an ideal child. This allows the live streaming application to match the ideal photo of the opposite sex with the streamer's female fans, thereby increasing the interaction between the streamer and the fans.

[0038] The following is a combination of... Figures 1 to 4 The image generation method provided in the embodiments of this disclosure will be described.

[0039] Figure 1 A schematic flowchart of an image generation method provided in an embodiment of this disclosure is shown.

[0040] In some embodiments of this disclosure, Figure 1 The image generation method shown can be executed by an electronic device or a server. The electronic device can include devices with image generation capabilities such as mobile phones, tablets, desktop computers, laptops, in-vehicle terminals, wearable devices, all-in-one machines, and smart home devices, or devices simulated by virtual machines or simulators. The server can be a cloud server or server cluster, or a device with storage and computing capabilities.

[0041] like Figure 1 As shown, the image generation method may include the following steps.

[0042] S110. Obtain the first image to be fused and the target image to be fused.

[0043] In the disclosed embodiments, when a user uses an image to generate special effects, a first image to be fused and a target image to be fused can be sent to an electronic device, so that the electronic device can acquire the first image to be fused and the target image to be fused, and use the first image to be fused and the target image to generate a new image.

[0044] In this embodiment of the disclosure, the first image to be fused may be an image of a target person that needs to be fused.

[0045] Optionally, the target person can be a person, an animal, or a cartoon character; there are no restrictions.

[0046] In this embodiment of the disclosure, the target fused image can be a fused image of the target person.

[0047] Among them, the target person in the target fusion image and the target person in the first image to be fused belong to the same category.

[0048] In some embodiments, the similarity of some features between the first image to be fused and the target image to be fused may be greater than a preset threshold. That is, the first image to be fused and the target image to be fused have some similar features.

[0049] Some of the features can be the features corresponding to the target person in the first image to be fused and the target fused image.

[0050] Optionally, some features may include at least one of the following: facial features, skin color, face shape, and hair color.

[0051] The similarity of some features can be the degree of similarity between some features of the first image to be fused and the target image to be fused.

[0052] The preset similarity threshold can be a similarity threshold that is set in advance as needed to determine whether some features are similar or identical.

[0053] Taking a first image to be fused as a photo of an adult woman and a target image to be fused as a photo of a child as an example, some features between the first image to be fused and the target image to be fused may include at least one of facial features, skin color features, face shape features, and hair color features. If the similarity of some features between the first image to be fused and the target image to be fused is greater than a preset similarity threshold, then an adult male photo can be generated in reverse based on some features or some similar features between the first image to be fused and the target image to be fused.

[0054] In other embodiments, the similarity of each feature between the first image to be fused and the target image to be fused is less than a preset threshold. That is, the first image to be fused and the target image to be fused do not have similar features.

[0055] Taking the first image to be fused as a photo of an adult woman and the target image to be fused as a photo of a child as an example again, the similarity of each feature between the first image to be fused and the target image to be fused is less than the preset similarity threshold. That is, an adult male photo can be generated in reverse by using any first image to be fused and any target image to be fused.

[0056] S120. Based on the first latent variable of the first image to be fused and the randomly generated second latent variable, generate candidate images to be fused.

[0057] In this embodiment of the disclosure, after the electronic device acquires the first image to be fused and the target image to be fused, it can acquire the first latent variable of the first image to be fused and the randomly generated second latent variable, and generate a candidate image to be fused based on the first latent variable of the first image to be fused and the randomly generated second latent variable.

[0058] In this embodiment of the disclosure, the first latent variable may be a latent variable generated based on the features of the target person in the first image to be fused.

[0059] In this embodiment of the disclosure, the randomly generated second latent variable can be a random latent variable fused with the first latent variable. Specifically, the randomly generated second latent variable can be generated based on the feature dimensions of the first latent variable.

[0060] In this embodiment of the disclosure, the candidate image to be fused can be a fused image generated based on a first latent variable and a randomly generated second latent variable.

[0061] Specifically, the electronic device can fuse the first latent variable and the second latent variable to obtain a fused latent variable, and generate candidate images to be fused based on the fused latent variable.

[0062] S130. Adjust the second latent variable based on the feature differences between the target fusion image and the candidate images to be fused.

[0063] In this embodiment of the disclosure, after the electronic device generates candidate images to be fused, the feature difference between each feature between the target fused image and the candidate images to be fused can be determined based on the image features of the target fused image and the candidate images to be fused. The second latent variable is adjusted according to the feature difference, and the candidate images to be fused are regenerated based on the adjusted second latent variable until the feature difference between the candidate images to be fused generated based on the first latent variable and the adjusted second latent variable and the target fused image is less than a preset threshold, so that the similarity of the features between the target fused image and the candidate images to be fused is high enough, and the adjustment of the second latent variable ends.

[0064] In this embodiment of the disclosure, the feature difference can be based on the difference between the target features of the target person in the target fusion image and the target features of the target person in the candidate image to be fused.

[0065] The target feature can be a feature used to adjust the second latent variable.

[0066] In this embodiment of the disclosure, the preset threshold can be a feature value set as needed to stop adjusting the second latent variable.

[0067] Specifically, the electronic device can determine the features of the target person in the target fused image and the features of the target person in the candidate image to be fused. Based on the features of the target person in the target fused image and the features of the target person in the candidate image to be fused, the feature difference between the target fused image and the candidate image to be fused is calculated. If the feature difference is less than a preset threshold, the adjustment of the second latent variable ends; otherwise, the adjustment of the second latent variable continues until the feature difference is less than the preset threshold.

[0068] Taking a child's photo as the target fusion image and an estimated adult male's photo as the candidate fusion image, the electronic device can calculate the difference in skin color features between the target fusion image and the candidate fusion image based on the skin color features of the child in the target fusion image and the skin color features of the adult male in the candidate fusion image. If the difference in skin color features is less than a preset threshold, the adjustment of the second latent variable ends; otherwise, the adjustment of the second latent variable continues until the difference in skin color features is less than the preset threshold.

[0069] S140. If the feature difference is not greater than a preset threshold, generate a second image to be fused based on the adjusted second latent variable.

[0070] In this embodiment of the disclosure, if the electronic device determines that the feature difference between the candidate image to be fused generated by the first latent variable and the adjusted second latent variable and the target image to be fused is less than a preset threshold, then the adjustment of the second latent variable is ended, the adjusted second latent variable is obtained, and a second image to be fused is generated based on the adjusted second latent variable.

[0071] In this embodiment of the disclosure, the adjusted second latent variable may be a latent variable used to generate another image to be fused.

[0072] In this embodiment of the disclosure, the second image to be fused may be another image to be fused that is fused with the first image to be fused to generate the target fused image.

[0073] Specifically, an image generation model can be used to generate a second image to be fused, based on the adjusted second latent variable.

[0074] In this embodiment, a first image to be fused and a target image to be fused can be obtained. The first image to be fused can be any image that needs to be fused, and the target image to be fused can be a fused image. Based on a first latent variable of the first image to be fused and a randomly generated second latent variable, a candidate image to be fused is generated. Then, based on the feature difference between the target image to be fused and the candidate image to be fused, the second latent variable is adjusted until the feature difference between the candidate image to be fused generated based on the first latent variable and the adjusted second latent variable and the target image to be fused is less than a preset threshold. Further, if the feature difference is not greater than the preset threshold, a second image to be fused is generated based on the adjusted second latent variable. Therefore, any image to be fused and a fused image can be used to generate another image to be fused, thereby obtaining new image generation effects to meet the diverse image generation needs of users.

[0075] In another embodiment of this disclosure, a pre-trained encoder can be used to generate a first latent variable corresponding to the first image to be fused, which improves the timeliness of latent variable extraction and further improves the efficiency of image generation.

[0076] Figure 2 A flowchart illustrating another image generation method provided in an embodiment of this disclosure is shown.

[0077] like Figure 2 As shown, the image generation method may include the following steps.

[0078] S210. Input the first image to be fused into the pre-trained encoder to obtain the first latent variable corresponding to the first image to be fused.

[0079] In this embodiment of the disclosure, after the electronic device acquires the first image to be fused and the target image to be fused, the first image to be fused can be input into a pre-trained encoder to extract the first latent variable corresponding to the first image to be fused.

[0080] The encoder can be an encoder model trained based on the sample image and the latent variables corresponding to the sample image.

[0081] Therefore, in this embodiment of the present disclosure, the first latent variable of the first image to be fused can be extracted using a trained encoder. Compared with the traditional latent variable extraction methods such as the Hidden Markov Model (HMM) in the existing methods, the extraction time of latent variables can be shortened, so as to achieve the effect of online extraction of latent variables and further improve the efficiency of image generation.

[0082] S220. Based on the feature dimension corresponding to the first latent variable, generate a second latent variable that includes the feature dimension.

[0083] In this embodiment of the disclosure, after the electronic device extracts the first latent variable of the first image to be fused, a second latent variable containing the feature dimension can be generated based on the feature dimension of the first latent variable, so that the feature dimension of the second latent variable is the same as that of the first latent variable.

[0084] In this embodiment of the disclosure, the feature dimension may be the number of categories of the feature corresponding to the first latent variable.

[0085] Specifically, the electronic device can randomly generate a latent variable containing the feature dimension corresponding to the first latent variable, and use it as the second latent variable to obtain the initial latent variable of another image to be fused.

[0086] Therefore, in this embodiment of the present disclosure, an initial latent variable for another image to be fused can be randomly generated based on the feature dimension corresponding to the first latent variable, and used as the second latent variable.

[0087] S230. Obtain the first image to be fused and the target image to be fused.

[0088] S230 is similar to S110, so it will not be described in detail here.

[0089] S240. Based on the first latent variable of the first image to be fused and the randomly generated second latent variable, generate candidate images to be fused.

[0090] Optionally, in this embodiment, S240 may specifically include the following steps:

[0091] S2401. Based on the random fusion coefficients, the first latent variable and the second latent variable are weighted and summed to obtain the fused latent variable;

[0092] S2402. Input the latent variables of fusion into the pre-trained first generative model to obtain candidate images to be fused.

[0093] Specifically, while generating the first and second latent variables, the electronic device can also generate random fusion coefficients. Based on the random fusion coefficients, the first and second latent variables are weighted and summed to obtain the fused latent variable. Then, the fused latent variable is input into the pre-trained first generative model to obtain candidate images to be fused.

[0094] Among them, the random fusion coefficient can be the fusion coefficient used to fuse the first latent variable and the second latent variable.

[0095] For example, if the first latent variable is P1 and the second latent variable is P2, the random fusion coefficient corresponding to the first latent variable is... Then the random fusion coefficient corresponding to the second latent variable is Therefore, latent variables are integrated.

[0096] The first generative model can be a pre-trained image generation model. Specifically, the first generative model can be trained using sample fusion latent variables and sample fusion images.

[0097] Optionally, the first generative model can be a generator based on a style-based generative adversarial network (StyleGAN) or other network models, without any restrictions.

[0098] Therefore, in this embodiment of the present disclosure, the first latent variable and the second latent variable can be weighted and summed based on the random fusion coefficient to obtain the fusion latent variable. Then, the first generative model is used, and based on the fusion latent variable, the candidate images to be fused are accurately generated to obtain the predicted fused image.

[0099] S250. Adjust the second latent variable based on the feature differences between the target fusion image and the candidate images to be fused.

[0100] Optionally, in this embodiment, S250 may specifically include the following steps:

[0101] Based on the feature differences between the target fusion image and the candidate images to be fused, the random fusion coefficients and the second latent variable are adjusted.

[0102] Specifically, after the electronic device generates candidate images to be fused, it can determine the feature differences of each feature between the target fused image and the candidate images to be fused based on the image features of the target fused image and the candidate images to be fused. It then adjusts the second latent variable and the random fusion coefficients according to the feature differences, and regenerates the candidate images to be fused based on the adjusted second latent variable and the adjusted random fusion coefficients. This process continues until the feature differences between the candidate images to be fused generated based on the first latent variable and the adjusted second latent variable and the target fused image are less than a preset threshold, making the feature similarity between the target fused image and the candidate images to be fused sufficiently high. At this point, the adjustment of the second latent variable ends. Otherwise, the adjustment of the second latent variable and the random fusion coefficients continues until the feature differences are less than the preset threshold.

[0103] Therefore, in this embodiment of the present disclosure, the second latent variable can be adjusted by adjusting the random fusion coefficient based on the feature differences of each feature between the target fused image and the candidate image to be fused, thereby improving the accuracy of the adjustment of the second latent variable and facilitating the accurate generation of another image to be fused.

[0104] S260. If the feature difference is not greater than a preset threshold, generate a second image to be fused based on the adjusted second latent variable.

[0105] S260 is similar to S140, so it will not be described in detail here.

[0106] In yet another embodiment of this disclosure, the second image to be fused can be generated in a different manner.

[0107] Figure 3 A schematic flowchart of another image generation method provided in an embodiment of this disclosure is shown.

[0108] like Figure 3 As shown, the image generation method may include the following steps.

[0109] S310. Obtain the first image to be fused and the target image to be fused.

[0110] S320. Input the first image to be fused into the pre-trained encoder to obtain the first latent variable corresponding to the first image to be fused.

[0111] S330. Based on the feature dimension corresponding to the first latent variable, generate a second latent variable that includes the feature dimension.

[0112] S340. Based on the random fusion coefficients, the first latent variable and the second latent variable are weighted and summed to obtain the fused latent variable.

[0113] S350. Input the latent variables of fusion into the pre-trained first generative model to obtain candidate images to be fused.

[0114] S360. Based on the feature differences between the target fusion image and the candidate images to be fused, adjust the random fusion coefficient and the second latent variable.

[0115] S310 to S360 are similar to the steps in the aforementioned embodiments, and will not be described in detail here.

[0116] S370. If the feature difference is not greater than a preset threshold, generate a second image to be fused based on the adjusted second latent variable.

[0117] In some embodiments of this disclosure, step S370 may specifically include the following steps:

[0118] S3701. Input the adjusted second latent variable into the pre-trained second generative model to obtain the second image to be fused.

[0119] Specifically, if the electronic device determines that the feature difference between the candidate image to be fused generated by the first latent variable and the adjusted second latent variable and the target fused image is less than a preset threshold, then the adjustment of the second latent variable ends, the adjusted second latent variable is obtained, and the adjusted second latent variable is input into the pre-trained second generation model. Using the second generation model, a second image to be fused is generated. Therefore, another image to be fused can be obtained.

[0120] The second generative model can be a pre-trained image generation model. Specifically, the first generative model can be trained using latent variables and sample images.

[0121] Optionally, the second generative model can be a StyleGAN generator or other network models; there are no restrictions on this.

[0122] Continuing with the example of the first image to be fused being an adult female photo and the target fused image being a child photo, after the electronic device obtains the adjusted second latent variable, it inputs the adjusted second latent variable into the pre-trained second generative model to obtain the second image to be fused, and then directly outputs an adult male photo.

[0123] In other embodiments of this disclosure, S370 may specifically include the following steps:

[0124] S3702. Input the adjusted second latent variable into the pre-trained second generative model to obtain the predicted image;

[0125] S3703. Input the predicted image into the pre-trained discriminant network to obtain the probability value of the target features in the predicted image;

[0126] S3704. If the probability value of the target feature is greater than the preset probability value threshold, the candidate image is used as the second image to be fused.

[0127] Specifically, if the electronic device determines that the feature difference between the candidate image to be fused generated by the first latent variable and the adjusted second latent variable and the target fused image is less than a preset threshold, then the adjustment of the second latent variable ends, the adjusted second latent variable is obtained, and the adjusted second latent variable is input into the pre-trained second generation model to obtain the predicted image. Then, the predicted image is input into the pre-trained discriminator network to obtain the probability value of the target feature in the predicted image. If the probability value of the target feature is greater than the preset probability value threshold, then the candidate image is used as the second image to be fused. If the probability value of the target feature is less than the preset probability value threshold, then the new image generation model is used to output the second image to be fused carrying the target feature.

[0128] The predicted image can be an image initially generated based on the adjusted second latent variable.

[0129] The target feature can be a pre-defined feature used to distinguish it from the first image to be fused. Optionally, the target feature can be a gender feature.

[0130] The discriminant network can be a pre-trained discriminant model used to predict target features. Specifically, the discriminant network can be trained using sample images and sample target features.

[0131] Optionally, the discriminant network can be a discriminator of StyleGAN or other network models; there are no restrictions here.

[0132] The probability threshold can be a probability value that is preset as needed to determine the target feature.

[0133] Continuing with the example of the first image to be fused being a photo of an adult woman and the target fused image being a photo of a child, after the electronic device obtains the adjusted second latent variable, it inputs the adjusted second latent variable into the pre-trained second generative model to obtain a predicted image. The predicted image is then input into the pre-trained discriminative network to obtain the probability value of the gender being male. If the probability value is greater than a pre-set probability threshold, the predicted image is used as the second image to be fused. If the probability value is less than the pre-set probability threshold, a new image generation model is needed to output the second image to be fused that is male.

[0134] Therefore, in this embodiment, a pre-trained second generative model can be used to generate an image corresponding to the adjusted second latent variable. This image can be directly used as the second image to be fused, or it can be used as a predicted image. A discriminative network can be used to calculate the probability value of the target feature in the predicted image. If the probability value of the target feature is greater than a preset probability threshold, the predicted image is used as the second image to be fused. If the probability value of the target feature is less than the preset probability threshold, a new model is used to generate a second image to be fused carrying the target feature, thereby improving the accuracy of the second image to be fused. Thus, different methods can be used to generate the second image to be fused, improving the flexibility of the second image to be fused generation.

[0135] Figure 4 A logical schematic diagram of an image generation method provided by an embodiment of this disclosure is shown.

[0136] like Figure 4 As shown, the first image to be fused is a photo of an adult woman, and the target fusion image is a photo of a child. An adult male photo is generated based on the adult woman's photo and the child's photo. First, the first image to be fused is input into a pre-trained encoder to obtain the first latent variable P1 corresponding to the first image to be fused. Then, based on the feature dimension corresponding to the first latent variable, a second latent variable P2 containing that feature dimension is generated. Second, random fusion coefficients are obtained, including the random fusion coefficient corresponding to the first latent variable P1. The random fusion coefficient corresponding to the second latent variable P2 is Based on the random fusion coefficients, a weighted sum of the first latent variable P1 and the second latent variable P2 is performed to obtain the fusion latent variable P. Next, the fusion latent variable P is input into a pre-trained first generative model, which is a StyleGAN generator, to generate candidate images to be fused. Then, the feature difference between the target fused image and the candidate images to be fused is calculated. Based on this feature difference, the random fusion coefficient and the second latent variable are adjusted until the feature difference between the candidate images to be fused generated based on the first latent variable and the adjusted second latent variable is less than a preset threshold. At this point, the adjustment of the second latent variable ends, resulting in the adjusted second latent variable. This feature difference can be... The loss value between the target fusion image and the candidate images to be fused is obtained as the adjusted latent variable for adult males. Finally, the adjusted second latent variable is input into a pre-trained second generative model, which is the generator of StyleGAN, to generate the second image to be fused. Alternatively, the adjusted second latent variable is input into a pre-trained second generative model to obtain a predicted image. The predicted image is then input into a pre-trained discriminative network to obtain the probability value of the target feature in the predicted image. If the probability value of the target feature is greater than a pre-set probability threshold, the predicted image is used as the second image to be fused.

[0137] This disclosure also provides an image generation apparatus for implementing the above-described image generation method, which will be described below in conjunction with... Figure 5 The following explanation is provided. In this embodiment, the image generation device can be an electronic device or a server. The electronic device can include mobile terminals, tablet computers, vehicle-mounted terminals, wearable electronic devices, virtual reality (VR) all-in-one machines, smart home devices, and other devices with communication functions. The server can be a cloud server or server cluster, or other devices with storage and computing functions.

[0138] Figure 5 A schematic diagram of the structure of an image generation apparatus provided in an embodiment of this disclosure is shown.

[0139] like Figure 5 As shown, the image generation device 500 may include: an image acquisition module 510, a candidate image to be fused generation module 520, a second latent variable adjustment module 530, and a second image to be fused generation module 540.

[0140] The image acquisition module 510 is used to acquire a first image to be fused and a target image to be fused, wherein the similarity of some features between the first image to be fused and the target image to be fused is greater than a preset similarity threshold.

[0141] The candidate image to be fused generation module 520 is used to generate candidate images to be fused based on the first latent variable of the first image to be fused and the randomly generated second latent variable;

[0142] The second latent variable adjustment module 530 is used to adjust the second latent variable based on the feature differences between the target fused image and the candidate images to be fused.

[0143] The second image to be fused generation module 540 is used to generate a second image to be fused based on the adjusted second latent variable, provided that the feature difference is not greater than a preset threshold.

[0144] In this embodiment, a first image to be fused and a target image to be fused can be obtained. The first image to be fused can be any image that needs to be fused, and the target image to be fused can be a fused image. Based on a first latent variable of the first image to be fused and a randomly generated second latent variable, a candidate image to be fused is generated. Then, based on the feature difference between the target image to be fused and the candidate image to be fused, the second latent variable is adjusted until the feature difference between the candidate image to be fused generated based on the first latent variable and the adjusted second latent variable and the target image to be fused is less than a preset threshold. Further, if the feature difference is not greater than the preset threshold, a second image to be fused is generated based on the adjusted second latent variable. Therefore, any image to be fused and a fused image can be used to generate another image to be fused, thereby obtaining new image generation effects to meet the diverse image generation needs of users.

[0145] Optionally, the device further includes: a first latent variable generation module and a second latent variable generation module;

[0146] The first latent variable generation module is used to input the first image to be fused into a pre-trained encoder to obtain the first latent variable corresponding to the first image to be fused.

[0147] The second latent variable generation module is used to generate a second latent variable containing the feature dimension based on the feature dimension corresponding to the first latent variable.

[0148] Optionally, the candidate image generation module 520 includes:

[0149] The weighted summation unit is used to perform a weighted summation of the first latent variable and the second latent variable based on the random fusion coefficients to obtain the fused latent variable;

[0150] The candidate image to be fused generation unit is used to input the latent variables of fusion into the pre-trained first generation model to obtain the candidate image to be fused.

[0151] Optionally, the second latent variable adjustment module 530 is specifically used to adjust the random fusion coefficient and the second latent variable based on the feature differences between the target fusion image and the candidate images to be fused.

[0152] Optionally, the second image to be fused generation module 540 is specifically used for:

[0153] The adjusted second latent variable is input into the pre-trained second generative model to obtain the second image to be fused.

[0154] Optionally, the second image generation module 540 includes:

[0155] The image prediction generation unit is used to input the adjusted second latent variable into the pre-trained second generation model to obtain the predicted image;

[0156] The probability value prediction unit is used to input the estimated image into a pre-trained discriminant network to obtain the probability value of the target feature in the estimated image;

[0157] The second image generation unit is used to generate a candidate image as the second image to be fused if the probability value of the target feature is greater than a preset probability value threshold.

[0158] It should be noted that, Figure 5 The image generation device 500 shown can perform... Figures 1 to 4 The various steps in the method embodiment shown are implemented. Figures 1 to 4 The processes and effects in the method embodiments shown are not described in detail here.

[0159] Figure 6 A schematic diagram of the structure of an image generation device provided in an embodiment of this disclosure is shown.

[0160] like Figure 6 As shown, the image generating device may include a processor 601 and a memory 602 storing computer program instructions.

[0161] Specifically, the processor 601 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0162] Memory 602 may include a large-capacity storage for information or instructions. For example, and not limitingly, memory 602 may include a hard disk drive (HDD), a floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or a Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 602 may include removable or non-removable (or fixed) media. Where appropriate, memory 602 may be internal or external to the integrated gateway device. In a particular embodiment, memory 602 is a non-volatile solid-state memory. In a particular embodiment, memory 602 includes read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (Electrically Programmable ROM, EPROM), an electrically erasable programmable PROM (EEPROM), an electrically alterable ROM (EAROM), or flash memory, or a combination of two or more of these.

[0163] The processor 601 reads and executes computer program instructions stored in the memory 602 to perform the steps of the image generation method provided in the embodiments of this disclosure.

[0164] In one example, the image generating device may further include a transceiver 603 and a bus 604. Wherein, as... Figure 6 As shown, the processor 601, memory 602 and transceiver 603 are connected via bus 604 and communicate with each other.

[0165] Bus 604 includes hardware, software, or both. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses, or a combination of two or more of these. Where appropriate, bus 604 may include one or more buses. Although specific buses are described and illustrated in the embodiments of this application, this application considers any suitable bus or interconnection.

[0166] The following are embodiments of a computer-readable storage medium provided in this disclosure. This computer-readable storage medium belongs to the same inventive concept as the image generation methods in the above embodiments. For details not described in detail in the embodiments of the computer-readable storage medium, please refer to the embodiments of the above image generation methods.

[0167] This embodiment provides a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform an image generation method, the method comprising:

[0168] Obtain the first image to be fused and the target image to be fused;

[0169] Based on the first latent variable of the first image to be fused and the randomly generated second latent variable, candidate images to be fused are generated.

[0170] The second latent variable is adjusted based on the feature differences between the target fusion image and the candidate images to be fused.

[0171] If the feature difference is no greater than a preset threshold, a second image to be fused is generated based on the adjusted second latent variable.

[0172] Of course, the computer-executable instructions provided in the embodiments of this disclosure are not limited to the above-described method operations, but can also perform related operations in the image generation method provided in any embodiment of this disclosure.

[0173] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this disclosure can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer cloud platform (which may be a personal computer, a server, or a network cloud platform, etc.) to execute the image generation methods provided in the various embodiments of this disclosure.

[0174] Note that the above description is merely a preferred embodiment and the technical principles employed in this disclosure. Those skilled in the art will understand that this disclosure is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this disclosure. Therefore, although this disclosure has been described in detail through the above embodiments, it is not limited to the above embodiments. Many other equivalent embodiments may be included without departing from the concept of this disclosure, and the scope of this disclosure is determined by the scope of the appended claims.

Claims

1. An image generation method, characterized in that, include: Obtain the first image to be fused and the target image to be fused; Based on the first latent variable of the first image to be fused and the randomly generated second latent variable, candidate images to be fused are generated; The second latent variable is adjusted based on the feature differences between the target fused image and the candidate images to be fused. If the feature difference is not greater than a preset threshold, a second image to be fused is generated based on the adjusted second latent variable; The step of generating candidate images to be fused based on the first latent variable of the first image to be fused and the randomly generated second latent variable includes: Based on the random fusion coefficients, the first latent variable and the second latent variable are weighted and summed to obtain the fused latent variable; The latent variables for fusion are input into a pre-trained first generative model to obtain the candidate images to be fused.

2. The method according to claim 1, characterized in that, Before generating candidate images to be fused based on the first latent variable of the first image to be fused and the randomly generated second latent variable, the method further includes: The first image to be fused is input into a pre-trained encoder to obtain the first latent variable corresponding to the first image to be fused. Based on the feature dimension corresponding to the first latent variable, a second latent variable containing the feature dimension is generated.

3. The method according to claim 1, characterized in that, The step of adjusting the second latent variable based on the feature differences between the target fused image and the candidate images to be fused includes: Based on the feature differences between the target fused image and the candidate images to be fused, the random fusion coefficients and the second latent variable are adjusted.

4. The method according to claim 1, characterized in that, When the feature difference is not greater than a preset threshold, generating a second image to be fused based on the adjusted second latent variable includes: The adjusted second latent variable is input into the pre-trained second generative model to obtain the second image to be fused.

5. The method according to claim 1, characterized in that, When the feature difference is not greater than a preset threshold, generating a second image to be fused based on the adjusted second latent variable includes: The adjusted second latent variable is input into the pre-trained second generative model to obtain the predicted image; The predicted image is input into a pre-trained discriminant network to obtain the probability values ​​of the target features in the predicted image; If the probability value of the target feature is greater than a preset probability value threshold, the estimated image is used as the second image to be fused.

6. An image generation apparatus, characterized in that, include: The image acquisition module is used to acquire the first image to be fused and the target fusion image; The candidate image to be fused generation module is used to generate candidate images to be fused based on the first latent variable of the first image to be fused and the randomly generated second latent variable; The second latent variable adjustment module is used to adjust the second latent variable based on the feature differences between the target fused image and the candidate images to be fused. The second image to be fused generation module is used to generate a second image to be fused based on the adjusted second latent variable when the feature difference is not greater than a preset threshold. The candidate image to be fused generation module includes: The weighted summation unit is used to perform a weighted summation of the first latent variable and the second latent variable based on the random fusion coefficients to obtain the fused latent variable; The candidate image to be fused generation unit is used to input the fusion latent variables into a pre-trained first generation model to obtain the candidate image to be fused.

7. The apparatus according to claim 6, characterized in that, Also includes: First latent variable generation module and second latent variable generation module; The first latent variable generation module is used to input the first image to be fused into a pre-trained encoder to obtain the first latent variable corresponding to the first image to be fused. The second latent variable generation module is used to generate a second latent variable containing the feature dimension based on the feature dimension corresponding to the first latent variable.

8. The apparatus according to claim 6, characterized in that, The candidate image to be fused generation module includes: The weighted summation unit is used to perform a weighted summation of the first latent variable and the second latent variable based on the random fusion coefficients to obtain the fused latent variable; The candidate image to be fused generation unit is used to input the fusion latent variables into a pre-trained first generation model to obtain the candidate image to be fused.

9. The apparatus according to claim 6, characterized in that, The second image-to-be-fused generation module is specifically used for: The adjusted second latent variable is input into the pre-trained second generative model to obtain the second image to be fused.

10. The apparatus according to claim 6, characterized in that, The second image generation module to be fused includes: The image prediction generation unit is used to input the adjusted second latent variable into the pre-trained second generation model to obtain the image prediction. The probability value prediction unit is used to input the estimated image into a pre-trained discriminant network to obtain the probability value of the target feature in the estimated image; The second image generation unit is used to generate the estimated image as the second image to be fused if the probability value of the target feature is greater than a preset probability value threshold.

11. An image generation device, characterized in that, include: processor; Memory, used to store executable instructions; The processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the image generation method according to any one of claims 1-5.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, The storage medium stores a computer program, which, when executed by a processor, causes the processor to implement the image generation method according to any one of claims 1-5.

Images