Image generation method, image generation model training method and electronic device

Abstract

The present invention discloses an image generation method, a training method of an image generation model and an electronic device. The method comprises: obtaining a first image of a target object and a second image of a target element; extracting features of the second image using a first branch of the image generation model to obtain image features of the second image, and performing denoising processing on the first image using a second branch of the image generation model to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample object, a second sample image of a sample element and a target constraint, the target constraint is constructed based on the first sample image and a sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; fusing the second image and the processed first image based on the image features to obtain a target image. The present invention solves the technical problem of poor image generation effect in the related art.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to an image generation method, an image generation model training method, and an electronic device. Background Art

[0002] At present, virtual try-on technology refers to the use of digital means to allow consumers to experience the effects of wearing clothing products through mobile phones, computers or other smart devices. The current virtual try-on method uses a generative adversarial network (GAN) to align clothing features with human body images, but there are challenges such as generating images with artifacts, retaining detailed textures, and long inference time, which leads to poor images generated by virtual try-on.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide an image generation method, an image generation model training method, and an electronic device to at least solve the technical problem of poor image generation effect in the related art.

[0005] According to one aspect of an embodiment of the present application, there is provided an image generation method, comprising: acquiring a first image of a target object and a second image of a target element, wherein the target element is used to characterize the element to be displayed on the target object; performing feature extraction on the second image using a first branch of an image generation model to obtain image features of the second image, and performing denoising on the first image using a second branch of the image generation model to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample object, a second sample image of a sample element, and a target constraint, the target constraint is constructed based on the first sample image and a sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; and fusing the second image and the processed first image based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image.

[0006] According to one aspect of an embodiment of the present application, a method for training an image generation model is provided, comprising: acquiring a first sample image of a sample object and a second sample image of a sample element; performing feature extraction on the first sample image to obtain a posture image of the sample object; constructing a target constraint based on the posture image of the sample object and the first sample image; and training an initial image generation model based on the first sample image, the second sample image and the target constraint to obtain an image generation model.

[0007] According to one aspect of an embodiment of the present application, there is provided an image generation method, comprising: obtaining a first image of a model image and a second image of clothing elements, wherein the clothing elements are used to characterize elements to be displayed on the model image; using a first branch of an image generation model to perform feature extraction on the second image to obtain image features of the second image, and using a second branch of the image generation model to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample model image, a second sample image of a sample clothing element, and a target constraint, the target constraint is constructed based on the first sample image and a sample model posture image, and the sample model posture image is used to characterize the posture of the model image in the first sample image; and fusing the second image and the processed first image based on the image features to obtain a target image, wherein clothing elements are displayed on the model image in the target image.

[0008] According to another aspect of the embodiments of the present application, a computer terminal is further provided, including: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present application is executed when the program is running.

[0009] According to another aspect of an embodiment of the present application, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in each embodiment of the present application.

[0010] According to another aspect of the embodiments of the present application, a computer program product is also provided, including a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0011] According to another aspect of an embodiment of the present application, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present application is implemented.

[0012] According to another aspect of the embodiments of the present application, a computer program is also provided, and when the computer program is executed by a processor, the methods in the various embodiments of the present application are implemented.

[0013] In an embodiment of the present application, a first image of a target object and a second image of a target element are obtained, wherein the target element is used to characterize the element to be displayed on the target object; a first branch of an image generation model is used to extract features of the second image to obtain image features of the second image, and a second branch of the image generation model is used to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample object, a second sample image of a sample element, and a target constraint, the target constraint is constructed based on the first sample image and a sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; the second image and the processed first image are fused based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image, thereby achieving the purpose of improving the image generation effect. It is easy to notice that, during the training process of the image generation model, the target constraint constructed by the first sample image of the sample object and the posture image of the sample object can improve the efficiency of the image generation model in processing posture information, so that the image generation model can pay more attention to the processing of sample elements in the second sample image, and the subsequent fusion effect of the first sample image and the second sample image, thereby improving the image processing effect of the image generation model. The first branch of the image generation model can be used to extract features from the second image to capture the complex image features in the first image, and the image features can be fused into the first image obtained after denoising processing, thereby improving the generation effect of the target image, thereby solving the technical problem of poor image generation effect in related technologies.

[0014] It is easy to notice that the above general description and the following detailed description are only for the purpose of exemplifying and explaining the present application, and do not constitute a limitation of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0016] Figure 1 is a schematic diagram of an application scenario of a machine learning model according to an embodiment of the present application;

[0017] Figure 2 is a structural block diagram of a computing environment of an image generation method according to an embodiment of the present application;

[0018] Figure 3 is a flowchart of an image generating method according to an embodiment of the present application;

[0019] Figure 4 is a schematic diagram of a dual-branch diffusion model according to an embodiment of the present application;

[0020] Figure 5 is a flowchart of a training method for an image generation model according to an embodiment of the present application;

[0021] Figure 6 is a flow chart of image generation according to an embodiment of the present application;

[0022] Figure 7 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0023] Figure 8 is a schematic diagram of a training device for an image generation model according to an embodiment of the present application;

[0024] Fig. 9 is a schematic diagram of an image generating device according to an embodiment of the present application;

[0025] Fig.10 It is a structural block diagram of a computer terminal according to an embodiment of the present application. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0027] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0029] Virtual try-on: Given a clothing image and a model image, generate a photo of the model wearing the clothing. The generated clothing needs to be natural, beautiful, and consistent with the clothing. Figure 1 To.

[0030] Diffusion Model: It is a mainstream deep learning model. The core idea of ​​this model is to gradually build a complete image by generating the pixel values ​​of the image through gradual diffusion. In this process, the model gradually generates the pixel values ​​of the image through multi-step reasoning, so that the quality of the generated image gradually improves.

[0031] According to an embodiment of the present application, an image generation method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Considering that the machine learning model consumes a lot of computing resources of the mobile terminal, the above method provided in the embodiment of the present application can be applied to Figure 1 The application scenarios shown are not limited to these. Figure 1 is a schematic diagram of an application scenario of a machine learning model according to an embodiment of the present application. Figure 1 In the application scenario shown, the machine learning model is deployed in the server 10, and the server 10 can be connected to one or more clients 20 through a local area network connection, a wide area network connection, an Internet connection, or other types of data networks. The client 20 here may include but is not limited to: smart phones, tablet computers, laptops, PDAs, personal computers, smart home devices, vehicle-mounted devices, etc. The client 20 can interact with the user through a graphical user interface to call the large model, thereby implementing the method provided in the embodiment of the present application.

[0033] It should be noted that, when the operating resources of the client device can meet the deployment and operating conditions of the large model, the embodiments of the present application can be carried out in the client device.

[0034] Figure 2 is a structural block diagram of a computing environment of an image generation method according to an embodiment of the present application, such as Figure 2 As shown, computing environment 201 includes multiple computing nodes (such as servers) running on a distributed network (shown as 210-1, 210-2, ..., in the figure). Different computing nodes contain local processing and memory resources, and end users 202 can remotely run applications or store data in computing environment 201. Applications can be provided as multiple services 220-1, 220-2, 220-3 and 220-4 in computing environment 201, representing services "A", "D", "E" and "H" respectively.

[0035] The end user 202 can provide and access services through a web browser or other software application on the client, and in some embodiments, the end user 202's provision and / or request can be provided to the entry gateway 230. The entry gateway 230 can include a corresponding agent to handle the provision and / or request for the service (one or more services provided in the computing environment 201).

[0036] Services are provided or deployed based on various virtualization technologies supported by the computing environment 201. In some embodiments, services can be provided based on virtual machine (VM)-based virtualization, container-based virtualization, and / or similar methods. Virtual machine-based virtualization can be to simulate a real computer by initializing a virtual machine, and execute programs and applications without directly contacting any actual hardware resources. While the virtual machine virtualizes the machine, according to container-based virtualization, a container can be started to virtualize the entire operating system (OS) so that multiple workloads can run on a single operating system instance.

[0037] In an embodiment based on container virtualization, several containers of a service can be assembled into a Pod (e.g., a Kubernetes Pod). Figure 2 As shown, service 220-2 can be equipped with one or more Pods 240-1, 240-2, ..., 240-N (collectively referred to as Pods). Pods can include a proxy 245 and one or more containers 242-1, 242-2, ..., 242-M (collectively referred to as containers). One or more containers in a Pod process requests related to one or more corresponding functions of the service, and the proxy 245 generally controls network functions related to the service, such as routing, load balancing, etc. Other services can also be equipped with similar Pods.

[0038] During operation, executing a user request from end user 202 may require invoking one or more services in computing environment 201, and executing one or more functions of one service may require invoking one or more functions of another service. Figure 2 As shown, service "A" 220-1 receives a user request from end user 202 from ingress gateway 230, service "A" 220-1 may call service "D" 220-2, and service "D" 220-2 may request service "E" 220-3 to perform one or more functions.

[0039] The computing environment described above can be a cloud computing environment, where the allocation of resources is managed by the cloud service provider, allowing the development of functions without considering the implementation, adjustment or expansion of servers. The computing environment allows developers to execute code in response to events without building or maintaining complex infrastructure. Services can be divided into a set of functions that can be automatically and independently scaled, rather than expanding a single hardware device to handle potential loads.

[0040] Under the above operating environment, this application provides Figure 3 It should be noted that the image generation method of this embodiment can be Figure 1 The server implementation of the illustrated embodiment. Figure 3 is a flow chart of the image generation method according to Embodiment 1 of the present application. Figure 3 As shown, the method may include the following steps:

[0041] Step S302, acquiring a first image of the target object and a second image of the target element;

[0042] The target element is used to represent the element to be displayed on the target object.

[0043] The above target elements are used to represent the elements to be displayed. The target elements include but are not limited to clothing elements (such as clothes, pants, shoes, etc.), makeup elements (such as lipstick, eye shadow, etc.), accessories elements (such as earrings, necklaces, etc.), and hairstyle elements. The types of target elements are not limited here.

[0044] The target object mentioned above may be an object for displaying the target element. The target object may be a user, a model, an object, etc., which displays the target element. The type of the target object is not limited here.

[0045] The first image of the target object is an image used to display the target object. The first image can be used to display the overall outline, body shape and other features of the target object. The features displayed in the first image can be used to match the target elements during the virtual try-on process to ensure the image fusion effect.

[0046] The second image of the target element mentioned above is an image containing the element to be displayed on the target object. The second image can be used to display the style, color, texture and other features of the target element so that it can be accurately applied to the target object during the virtual try-on process to ensure the image fusion effect.

[0047] In an optional embodiment, the user can upload the first image and the second image by operating the interactive interface. After the upload is successful, the first image and the second image can be displayed on the interactive interface so that the user can confirm whether the first image and the second image are the images previously uploaded by the user. If the user confirms that they are correct, the subsequent image generation process can be performed based on the first image and the second image to ensure that the generated image is the image required by the user.

[0048] In the field of virtual try-on, obtaining the first image of the target object and the second image of the target element is a key step in the virtual try-on system, which can provide a basis for achieving a real and personalized try-on effect. By obtaining the first image and the second image, users can preview the effects of different elements on the user without actually wearing them, so as to make more informed shopping decisions.

[0049] Step S304, using the first branch of the image generation model to perform feature extraction on the second image to obtain image features of the second image, and using the second branch of the image generation model to perform denoising on the first image to obtain a processed first image;

[0050] Among them, the image generation model is trained based on the first sample image of the sample object, the second sample image of the sample element and the target constraint, the target constraint is constructed based on the first sample image and the sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image.

[0051] The above-mentioned image generation model can be a diffusion model, a convolutional neural network model, etc., and the type of the image generation model is not limited here. The diffusion model can be a conditional diffusion model, a dual-branch diffusion model, etc., and is not specifically limited here. This application takes the image generation model as a dual-branch diffusion model as an example for explanation.

[0052] The dual-branch diffusion model described above can be a variant model based on the diffusion model, and its main feature is that two branches are used to handle different tasks or features during the generation process. One branch is used to generate the main image content, and the other branch is used to capture details or specific features. The two branches interact or merge in some way to improve the quality and realism of the generated image.

[0053] Different branches in the above-mentioned image generation model can be used to represent different processing paths, and the first branch and the second branch of the image generation model correspond to different processing paths of the image generation model.

[0054] In the dual-branch diffusion model, the two branches can process different levels or types of information respectively. For example, one branch is responsible for generating the overall structure and main content, and the other branch is responsible for enhancing the detail texture or specific features. Through this division of labor and cooperation, the global consistency and local details of the generated image can be better balanced, thereby improving the quality and realism of the generated image.

[0055] The first branch mentioned above may be a branch in the dual-branch diffusion model responsible for enhancing detail texture or specific features, and the first branch may be called a feature branch. The first branch may be used to extract features of the second image so as to identify image features of the second image, such as edges, textures, colors, etc.

[0056] The second branch mentioned above can be a branch that ensures the overall structure and main content of the image in the dual-branch diffusion model, and the second branch can be called a denoising branch. The second branch can be used to denoise the first image, and can remove noise in the image to ensure a realistic and natural image display effect.

[0057] The above target constraint is constructed based on the first sample image and the sample object posture image. The target constraint is mainly used to improve the efficiency of the image generation model when processing the first sample image.

[0058] The sample object posture image is used to characterize the posture of the sample object in the first sample image. Building target constraints through the first sample image and the sample object posture image helps to better understand and utilize the structural information of the sample object during the training process.

[0059] By constructing target constraints, the image generation model can be given more reference efficiency in the posture learning process during the image generation model training process, so that the image generation model can quickly determine the posture of the sample object in the first sample image, and give more time to the feature processing of the sample elements, thereby improving the image generation model's processing ability for the sample elements in the second sample image, so that the image generation model can process the first image and the second image more accurately during the actual application process, thereby ensuring the effect of the target image obtained by subsequent fusion.

[0060] In an optional embodiment, the first branch of the image generation model is first used to extract features of the second image to obtain image features of the second image, which can capture key information and features of the target element. At the same time, the second branch of the image generation model is used to perform denoising on the first image to obtain the processed first image, which can remove noise and interference in the image, making it clearer and more realistic. The dual-branch structure can effectively extract the features of the target element and remove noise from the target object image, providing better input for subsequent image synthesis and processing. It helps to improve the realism and accuracy of the virtual try-on effect, allowing users to better feel the effect of the target element on the target object, thereby improving user experience and product recognition.

[0061] Step S306: fusing the second image and the processed first image based on the image features to obtain a target image.

[0062] Wherein, a target element is displayed on the target object in the target image.

[0063] In an optional embodiment, the second image and the processed first image are fused based on image features, so that the target image can show the composite effect of the target element and the target object. This fusion effect allows users to intuitively see the display effect of the target element on the target object, helping users to better evaluate the appearance and matching effect of the product, and improving users' recognition of the product.

[0064] The above-mentioned image features may be image features of different dimensions obtained by performing feature extraction on the second image through the first branch. The above-mentioned image features may also be all image features obtained by performing feature extraction on the second image through the first branch. The above-mentioned image features may also be image features of different dimensions and all image features obtained by performing feature extraction on the second image through the first branch. The specific form of the image features is not limited here.

[0065] In an optional embodiment, when the image features are image features of different dimensions, the image features of different dimensions can be input into the second branch so as to be fused with the first image processed by different network layers in the second branch to obtain the target image. When the image features are global image features, the global image features can be fused with the first image processed by the second branch to obtain the target image. It is also possible to fuse image features of different dimensions during the processing of the first image by the second branch, and when the second branch obtains the processed first image, fuse the processed first image with all image features to obtain the target image.

[0066] The above-mentioned scheme of the present application can be applied to image display scenarios of different products, for example, it can be applied to clothing display scenarios, makeup display scenarios, accessories display scenarios, hairstyle display scenarios, etc. The image display scenarios are not specifically limited here.

[0067] In a clothing display scenario, a first image of a model image and a second image of clothing elements can be obtained, and features of the second image can be extracted using the first branch of the image generation model to obtain clothing element features of the second image. The first image can be denoised using the second branch of the image generation model to obtain a processed first image. The second image and the processed first image can be fused based on the clothing element features in the second image to obtain a target image, in which clothing elements are displayed on the model image.

[0068] In a makeup display scenario, a first image of a model's image and a second image of makeup elements can be obtained, and features of the second image can be extracted using the first branch of the image generation model to obtain makeup element features of the second image. The first image can be denoised using the second branch of the image generation model to obtain a processed first image. The second image and the processed first image can be fused based on the makeup element features in the second image to obtain a target image, in which makeup elements are displayed on the model image.

[0069] It should be noted that the above-mentioned model image can be a pre-set standard image, can be an image selected by the user, or can be the user's own image. The type of model image is not limited here.

[0070] Through the above steps, a first image of the target object and a second image of the target element are obtained, wherein the target element is used to characterize the element to be displayed on the target object; the first branch of the image generation model is used to extract features of the second image to obtain image features of the second image, and the second branch of the image generation model is used to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of the sample object, a second sample image of the sample element, and a target constraint, the target constraint is constructed based on the first sample image and the sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; the second image and the processed first image are fused based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image, thereby achieving the purpose of improving the image generation effect. It is easy to notice that, during the training process of the image generation model, the target constraint constructed by the first sample image of the sample object and the posture image of the sample object can improve the efficiency of the image generation model in processing posture information, so that the image generation model can pay more attention to the processing of sample elements in the second sample image, and the subsequent fusion effect of the first sample image and the second sample image, thereby improving the image processing effect of the image generation model. The first branch of the image generation model can be used to extract features from the second image to capture the complex image features in the first image, and the image features can be fused into the first image obtained after denoising processing, thereby improving the generation effect of the target image, thereby solving the technical problem of poor image generation effect in related technologies.

[0071] In the above embodiment of the present application, the first image is denoised using the second branch of the image generation model to obtain a processed first image, including: extracting features from the first image to obtain an object posture image and a mask area image, wherein the object posture image is used to characterize the posture of the target object in the first image, and the mask area image is used to represent the area image of the target element to be displayed in the first image; and the first image is denoised based on the object posture image and the mask area image using the second branch to obtain the processed first image.

[0072] The above-mentioned object posture image contains the structural information of the target object, which can guide the image generation model to more accurately generate an image in which the target element is displayed on the target object.

[0073] The mask area image mentioned above can be used to identify the specific display area and specific display position of the target element in the first image.

[0074] In an optional embodiment, the second branch can remove noise and interference in the first image according to the object posture image and the mask area image, so that the image is clearer and more realistic. It should be noted that when the second branch processes the first image, it can fuse the image features output by the first branch to obtain the processed first image. The second branch can remove interference and noise in the first image, fuse the image features of the target element, and process the image according to the object posture and mask area information, so that the generated image can more clearly and accurately show the posture of the target object and the display effect of the target element. Such processing helps to improve the realism and accuracy of the virtual try-on effect, and provide users with a better experience and decision-making basis.

[0075] Through denoising, the noise and unclear areas in the image can be reduced and the overall quality of the image can be improved. In the denoising process, by using the object posture image and the mask area image to denoise the first image, important information in the image can be more accurately identified and retained, avoiding the loss of important details in the denoising process.

[0076] In the above embodiments of the present application, the first branch includes multiple first network layers, the second branch includes multiple second network layers, different first network layers are used to output image features of different dimensions, the outputs of the multiple first network layers are correspondingly connected to the inputs of the multiple cross-attention modules, and the inputs of the multiple second network layers are correspondingly connected to the outputs of the multiple cross-attention modules.

[0077] The first branch and the second branch may each include multiple network layers. These network layers may be convolutional layers, pooling layers, or other types of deep learning layers. Such a design helps to gradually extract and abstract image features.

[0078] The input and output of the above crisscross attention module correspond to the network layers of the first branch and the second branch respectively. The crisscross attention mechanism helps the network focus on important feature areas when processing images and ignore irrelevant information. This mechanism can improve the generalization ability of the model, reduce overfitting and improve accuracy.

[0079] The outputs of multiple network layers of the first branch are connected to the input of the cross-attention module, while the inputs of multiple network layers of the second branch are connected to the output of the cross-attention module. This corresponding connection ensures the coherence of the information flow, so that the model can effectively guide the learning of another branch based on the features learned by one branch. The outputs of multiple first network layers are correspondingly connected to the inputs of multiple cross-attention modules, and the inputs of multiple second network layers are correspondingly connected to the outputs of multiple cross-attention modules, which realizes information interaction and attention focus between different network layers, helps the model better understand and utilize feature information at different levels, and improves the expression and fusion effect of image features.

[0080] Through the above network structure design, different modules and layers in the network can be effectively organized and utilized to realize multi-dimensional feature extraction and cross-attention mechanism, thereby improving the performance and effect of the model and enabling the model to better adapt to complex task requirements. Multiple first network layers can improve the model's ability to learn and express image features, input image features of different dimensions into multiple second network layers, improve the feature extraction and fusion process in multiple second network layers, and thus improve the effect and performance of the model in image generation tasks.

[0081] In the above embodiment of the present application, the image generation model also includes a feature encoder, and the second branch of the image generation model is used to perform denoising on the first image to obtain a processed first image, including: using the feature encoder to encode the object posture image and the mask area image into the latent space to obtain a first posture feature, a mask area feature and a latent space noise, wherein the first posture feature is used to represent other posture features in the target object except the posture feature associated with the preset element, and the preset element and the target element belong to the same category; using the second branch to perform denoising on the first image based on the object posture image, the mask area image and the latent space noise to obtain the processed first image.

[0082] The above-mentioned feature encoder can be an autoencoder, a variational autoencoder (VAE for short), etc., and the type of the feature encoder is not limited here.

[0083] The above-mentioned preset elements may be elements belonging to the same category as the target element. For example, if the target element is a clothing element, the preset elements may be clothing elements in the target object. This is only an example and is not specifically limited.

[0084] In a variational autoencoder, the encoder part is responsible for encoding the input data into the mean and variance parameters in the latent space, thereby learning the potential distribution of the data. The decoder part samples from the latent space based on these parameters to generate reconstructed data similar to the input data. This structure enables the variational autoencoder to learn the potential structure of the data and generate new data samples.

[0085] The first posture feature mentioned above represents other posture features in the target object except the posture features associated with the preset element (i.e., the element belonging to the same category as the target object). The mask area feature mentioned above represents the area feature in the target object used to display the target element. For example, if it is necessary to display clothing elements on the model image, other posture features in the model image except the area where the clothing elements are located can be extracted to obtain the first posture feature, and the feature of the area where the clothing elements are located can be the mask area feature.

[0086] The latent space noise mentioned above can be latent space random noise, and the latent space noise can be randomly generated. During the denoising process, the second branch can improve the generalization ability of the model by adding latent space random noise, so that the model can better adapt to different input data.

[0087] The latent space mentioned above refers to a low-dimensional or high-dimensional feature space, in which the features can encode relevant information of the original data, but it is a more abstract and compressed form of the original data. The size of the mask area image can be adjusted by resizing so that the mask area image adapts to the needs of the latent space and obtains the mask area features. By encoding the object posture image and the mask area image into the latent space, the purpose of reducing the amount of calculation is achieved and retaining the key features.

[0088] In an optional embodiment, the object posture image and the mask area image are encoded into a latent space through a feature encoder, which can better capture the structural information of the target object and the area information of the elements to be displayed. This helps the model to more accurately generate images related to the target object posture and the elements to be displayed, and improves the accuracy and realism of the generated images. By encoding into a latent space, the amount of calculation of the image generation model can be reduced, thereby improving the operational efficiency of the image generation model. Using the second branch to perform denoising based on the encoded features can eliminate noise and interference in the image, making the generated image clearer and more realistic.

[0089] In the above embodiment of the present application, the second image and the processed first image are fused based on image features to obtain a target image, including: using the second branch to fuse image features of different dimensions with the processed first image to obtain the target image.

[0090] In an optional embodiment, after the first image is denoised by the second branch to obtain the processed first image, the image features of different dimensions output by the first branch are input into the second branch through a plurality of cross-attention modules connected between the first branch and the second branch, so that the second branch can fuse the image features of different dimensions with the processed first image to obtain the target image. The process of fusing the processed first image with the image features of different dimensions can ensure that the target image remains visually coherent and real, making the generated image more realistic and enhancing the user experience.

[0091] The outputs of the cross-attention module are connected in different second network layers in the above-mentioned second branch, and the inputs of the cross-attention module are connected in different first network layers in the first branch. The cross-attention module can be used to inject image features of different dimensions into the second branch, so that the image generation model can learn the image features of the target elements. The role of the cross-attention module is to help the model better pay attention to features of different dimensions and to interact and fuse information between different network layers. By processing the feature representations of different network layers through the cross-attention module, dynamic attention and adjustment of features can be achieved, so that the model can better learn the relationship and importance between features of different dimensions, thereby improving the effect and quality of image generation.

[0092] In the above embodiment of the present application, the second branch is used to fuse image features of different dimensions and the processed first image to obtain a target image, including: extracting features from the first image and / or the second image to obtain enhanced features; and using the second branch to fuse the enhanced features, the processed first image and image features of different dimensions to obtain the target image.

[0093] The above-mentioned enhanced features are used to ensure that the image generation model can focus on more fine-grained features during the image processing process, so as to improve the generation effect of the target image.

[0094] The above-mentioned enhanced features may be global posture features of the target object in the first image. The global posture features may be used to ensure that the posture or form of the target image is consistent with that of the target object in the first image, thereby improving the generation effect of the target image.

[0095] The aforementioned enhanced features may also be global image features of the target element in the second image. The global image features may be used to ensure global information of the target element in the target image, thereby improving the realistic effect of the target image.

[0096] The above-mentioned enhanced features may also include the above-mentioned global posture features and global image features, so that the posture of the target object in the target image remains consistent with the posture of the first image while making the target elements in the target image more realistic, thereby achieving a better effect of the target image.

[0097] In an optional embodiment, by performing feature extraction on the first image and / or the second image again, more complete enhanced features in the image can be extracted, and these enhanced features are fused with image features of different dimensions, which helps to comprehensively utilize feature information of different levels and types, and improve the model's ability to understand and express images. By fusing the enhanced features, the processed first image, and image features of different dimensions, the generated target image can more comprehensively and accurately display the features and morphology of the target elements, thereby improving the quality and realism of the generated image.

[0098] In the above embodiments of the present application, feature extraction is performed on the first image and / or the second image to obtain enhanced features, including: feature extraction is performed on the second image to obtain global image features; feature extraction is performed on the first image to obtain second posture features, wherein the second posture features are used to represent the global posture features of the target object; and enhanced features are determined based on the global image features and / or the second posture features.

[0099] The feature extraction of the second image can be performed through an image encoder (Contrastive Language-Image Pre-Training, referred to as CLIP) to obtain global image features, but is not limited to this. The feature extraction of the second image can also be performed through other methods, which are not specifically limited here.

[0100] The feature extraction of the first image can be performed through a multi-layer convolution posture guidance network to obtain the second posture feature, but it is not limited to this. The feature extraction of the first image can also be performed through other methods, which are not specifically limited here.

[0101] The above-mentioned image encoder is a model that is pre-trained using a large amount of image and text data. The CLIP image encoder is able to capture the rich semantic information of an image and convert it into a coded representation that can be used for further processing. For example, in clothing image processing, the CLIP encoder can extract clothing features, which are then integrated into the denoising branch through a cross-attention mechanism to maintain the global information of the clothing. The features extracted by the image encoder can help the system better understand the image content and improve the accuracy and semantic consistency of clothing image processing.

[0102] The multi-layer convolution posture guidance network described above can utilize multi-layer convolutional neural networks to process images, especially to extract and understand the structural information of target objects in images. Multi-layer convolution can effectively capture features of different scales, thereby better understanding the complex structure of the target object.

[0103] In the field of clothing fitting, the posture guidance network can extract features from the model's posture image, which reflect the precise location of various parts of the human body. These features are then added to the input of the denoising branch to ensure the consistency of the fitting image and the model image in terms of human posture and morphology. The posture guidance network with multiple layers of convolution can accurately extract and apply human posture features, which can significantly improve the realism and naturalness of the fitting image.

[0104] By combining the above-mentioned image encoder and multi-layer convolutional posture guidance network, the fusion effect of the first image and the second image can be processed more effectively, ensuring the naturalness and consistency of the target image in vision and posture, and providing users with a higher quality image experience.

[0105] In an optional embodiment, the global image features obtained by feature extraction of the second image can be global features of the target element in the second image, so as to ensure the global information of the target element. For example, the overall color, texture, composition and other information of the target element helps to capture the overall visual effect of the image. The extraction of global image features helps to maintain the global consistency of the image and ensure that the enhanced image is visually coordinated with the original image. Posture features can be specifically extracted to improve the sensitivity and accuracy of the model to changes in human posture, which is crucial for the fidelity of the subsequent display effect of the target element on the target object. By combining global image features and global posture features to determine the enhanced features, more comprehensive image information can be provided, which helps to generate more realistic and target-compliant images.

[0106] In the above embodiment of the present application, the method also includes: extracting features from the first sample image to obtain a sample object posture image; constructing a target constraint based on the sample object posture image and the first sample image; and training an initial image generation model based on the first sample image, the second sample image and the target constraint to obtain an image generation model.

[0107] Since the above-mentioned sample object posture image provides object structure information that is not interfered by color texture, such as human body structure information, by using the sample object posture image and the first sample image to construct the target constraint, it can be ensured that the generated image maintains a high degree of consistency with the first sample image in terms of object structure, which is conducive to improving the display effect of subsequent sample elements.

[0108] In an optional embodiment, a target constraint can be constructed based on the sample object posture image and the first sample image. In the process of training the initial image generation model based on the first sample image and the second sample image, the target constraint can be used to improve the processing efficiency of the initial image generation model for the first sample image. The initial image generation model is mainly used to process the structural information in the first sample image. By adding the target constraint, the structural information of the sample object can be accurately determined in a short time. By compressing the required step size of the structural information of the sample object, more time steps can be allocated to the feature learning of the sample elements in the second sample image, thereby further enhancing the retention of the sample element details by the initial image generation model.

[0109] In another optional embodiment, when the time step of the initial image generation model is large, a target constraint can be added so that the initial image generation model can accurately determine the structural information of the sample object within a larger time step. By compressing the time step required for the structural information of the sample object, more time steps can be reallocated to feature learning of sample elements in the second sample image during model inference, thereby further enhancing the retention of sample element details by the initial image generation model.

[0110] It should be noted that the above-mentioned initial image generation model will cycle multiple times during training, and different cycles correspond to a time step. The value of the time step is generally from 0 to 1000, but is not limited to this. The time step can also be determined according to actual conditions.

[0111] In the above embodiment of the present application, a target constraint is constructed based on a sample object posture image and a first sample image, including: extracting a first attention feature and a second attention feature of the sample object posture image and the first sample image, respectively; performing singular value decomposition on the first attention feature and the second attention feature, respectively, to obtain first structural information of the sample object posture image and second structural information of the first sample image, wherein the singular value decomposition is used to describe the structural information of the feature by means of matrix decomposition; and constructing the target constraint by minimizing the Euclidean distance between the first structural information and the second structural information.

[0112] The above singular value decomposition is a matrix decomposition method that can decompose a matrix into the product of three matrices, namely the left singular vector matrix, the singular value matrix, and the right singular vector matrix. In feature extraction, the structural information of the feature, i.e., the main singular value and the corresponding singular vector, can be obtained by performing singular value decomposition on the feature matrix. By comparing the Euclidean distance between the structural information of different samples, the constraint and consistency learning of the features can be achieved.

[0113] The above-mentioned first attention feature can be a feature related to the sample object posture obtained by feature extraction of the sample object posture image. By performing singular value decomposition on the first attention feature, the main singular value and the corresponding singular vector of the sample object can be obtained, and the structural information representation of the sample object in the sample object posture image can be determined based on the main singular value and the singular vector, that is, the above-mentioned first structural information.

[0114] The above-mentioned second attention feature can be feature extraction of the first sample image to obtain features related to the sample object posture. By performing singular value decomposition on the second attention feature, the main singular value and the corresponding singular vector of the sample object can be obtained, and the structural information representation of the sample object in the first sample image can be determined based on the main singular value and the singular vector, that is, the above-mentioned second structural information.

[0115] The main singular values ​​and singular vectors obtained through singular value decomposition can help reduce the dimension of the data and extract representative features, which helps to improve the training and reasoning efficiency of the model.

[0116] The above-mentioned Euclidean distance is a method for measuring the distance between two vectors by calculating the sum of the squares of the differences between them and then taking the square root. When constructing the target constraint, by minimizing the Euclidean distance between the first structural information and the second structural information, it is possible to make them as close as possible, even if they are not exactly the same, the gap between them can be as small as possible. Since the Euclidean distance can measure the similarity between structures well, the target constraint can be made stricter and more accurate by minimizing the Euclidean distance between the first structural information and the second structural information. By minimizing the Euclidean distance, it can be ensured that the first structural information and the second structural information are similar to some extent, so as to better meet the constraints.

[0117] In an optional implementation, in order to make full use of the posture structure information provided by the sample object posture graph and avoid the interference of color texture, a new branch can be introduced. This branch takes the sample object posture graph as the only input, constrains the original denoising branch self-attention feature in the initial image generation model, and then effectively guides the first sample image during the training process of the initial image generation model.

[0118] In the above embodiment of the present application, the initial image generation model is trained based on the first sample image, the second sample image and the target constraint to obtain the image generation model, including: in response to the first time step of the initial image generation model in the training process being greater than the preset time step, the initial second branch in the initial image generation model is trained based on the target constraint and the first sample image in the first preset time step, and the initial first branch in the initial image generation model is trained based on the second sample in the second preset time step to obtain the image generation model, and the sum of the first preset time step and the second preset time step is the first time step; in response to the second time step of the initial image generation model in the training process being less than or equal to the preset time step, the initial second branch is trained based on the first sample image in the third preset time step, and the initial first branch is trained based on the second sample image in the fourth preset time step to obtain the image generation model, and the sum of the third preset time step and the fourth preset time step is the second time step; wherein, the proportion of the first time step is less than the proportion of the second time step, the proportion of the first time step is the ratio of the first preset time step to the second preset time step, and the proportion of the second time step is the ratio of the third preset time step to the fourth preset time step.

[0119] The above-mentioned preset time step can be a preset time step used to determine whether a target constraint needs to be added. The preset time step can be set manually or flexibly according to the actual needs of training. The setting method of the preset time step is not limited here.

[0120] Generally, when the time step is large, the model will pay more attention to the structural information in the image. Therefore, when the time step is large, it is necessary to compress the time step of the initial image generation model to determine the corresponding structural information of the sample object by adding target constraints, and allocate more time steps to the learning process of the sample elements.

[0121] If the first time step of the initial image generation model during the training process is greater than the preset time step, it means that the training time step of the initial image generation model is large. At this time, the first preset time step can be used to train the initial second branch, and the second preset time step can be used to train the initial first branch to obtain the image generation model.

[0122] Since the first time step is greater than the preset time step, the model will pay more attention to the structural information in the image. Therefore, the training of the initial second branch can be compressed through the target constraint, so that the first preset time step accounts for a smaller proportion of the first time step of the initial image generation model, and more time steps are allocated to the training process of the initial first branch, so that the generated initial image generation model can accurately determine the structural information of the sample object in the first sample image while reducing the time step of the initial second branch, and allocate more time steps to the initial image generation model for learning the sample elements, so that the obtained image generation model can accurately determine the structural information of the sample object while retaining more sample element details.

[0123] If the second time step of the initial image generation model during the training process is less than or equal to the preset time step, it means that the training time step of the initial image generation model is small. At this time, the third preset time step can be used to train the initial second branch, and the fourth preset time step can be used to train the initial first branch to obtain the image generation model.

[0124] Since the first time step is less than or equal to the preset time step, the model pays balanced attention to different information. Therefore, there is no need to compress the training of the initial second branch through target constraint. The proportion of the third preset time step and the fourth preset time step in the second time step does not fluctuate greatly, so that the initial image generation model can normally learn the sample objects in the first sample image and the sample elements in the second sample image, thereby obtaining an image generation model with better image generation effect.

[0125] In an optional embodiment, the first preset time step and the second preset time step are dynamically adjusted according to the time step of the initial image generation model in the training process, so that the training process can be flexibly controlled so that the model can obtain a more balanced training effect at different stages. In the case of a large time step, by increasing the target constraint to reduce the time step of the denoising branch and allocating more time steps to the feature branch, the computing resources of the model can be more effectively utilized and the model's ability to learn features can be improved. By compressing the structural information of the sample object to determine the required time step, more time steps can be reallocated to the feature learning of the sample elements during model reasoning. In this way, the model can focus more on generating high-quality features of the sample elements and improve the realism and detail expression of the generated image.

[0126] According to the different proportions of time steps, the training of the initial second branch and the initial first branch can be reasonably allocated, which can ensure that the model can maintain the accuracy of the corresponding structural information of the sample object and fully learn the characteristics of the sample elements during the generation process, so as to achieve a more natural and realistic image generation effect. By dynamically adjusting the time step and the size of the time step, the training efficiency can be improved, the model training time can be shortened, and the iteration and optimization process of the model can be accelerated while ensuring the model generation effect. By training different branches at different stages, the model can better adapt to various different input images, improve the generalization ability of the model, and enable it to generate high-quality images when facing various complex scenes. By explicitly allocating time steps, the learning dynamics of the model during the training process can be more clearly understood, which helps to analyze and explain the generation process of the model and provide a basis for further optimizing the model.

[0127] By reasonably adjusting the time step and introducing the target constraint as mentioned above, the training process of the image generation model can be effectively updated, the generation effect and performance of the model can be improved, and more natural, realistic and high-quality image generation can be achieved.

[0128] With the development of the e-commerce industry, for advertisers, advertising creativity based on virtual try-on technology can be used to bring personalized experience to users, better meet the needs of different users, increase users' recognition of products, and improve the conversion efficiency of advertising. Virtual try-on technology provides a new way for product shooting and display, without the need to hire professional models and shooting teams for real-life shooting, saving time and economic costs. The image generation method proposed in this application can be applied in the field of virtual try-on, which can improve production effects while ensuring the beauty and realism of the try-on results.

[0129] Virtual try-on technology refers to the use of digital means to allow consumers to experience the effects of wearing clothing products through mobile phones, computers or other smart devices without actually wearing or touching physical products. For advertisers, advertising creativity based on virtual try-on technology can bring personalized experience to users, better meet the needs of different users, increase users' recognition of products, and improve advertising conversion efficiency.

[0130] The above-mentioned scheme of the present application is applied in the field of virtual try-on, which can ensure the details of clothing. A parallel dual-branch structure can be designed. The main branch can be used to ensure a realistic and natural try-on effect, and the other branch is used to capture complex clothing features. These clothing features can be further integrated into the main branch through the attention mechanism to improve the fidelity of clothing in the try-on result image.

[0131] In order to improve the inference efficiency, a clothing-oriented training strategy is designed. During the training process, more emphasis is placed on learning clothing attributes, and by imposing consistency constraints on the self-attention module, the image generation model is encouraged to quickly determine the human pose and shape at an early stage, allowing it to focus more on the synthesis of clothing features.

[0132] Through the above strategies, the method of the present application can generate realistic fitting effects with less inference time. In the field of virtual fitting, the above image generation model can be a dual-branch diffusion model. The first image of the target object in the present application can be a model image of a model, and the second image of the target element can be a clothing image of a clothing element. Figure 4 is a schematic diagram of a dual-branch diffusion model according to an embodiment of the present application, such as Figure 4 As shown, the pose extraction and clothing mask extraction of the model image can be performed respectively to obtain the model pose image and the clothing mask image, and the model pose image, the clothing mask image and the clothing image can be fused based on the clothing features using a dual-branch diffusion model to obtain a try-on image. Furthermore, the try-on image and the model image can be fused to obtain a final try-on image, ensuring that the final try-on image is consistent with the model image except for the clothing area, such as the background, limbs, etc.

[0133] In the above-mentioned virtual try-on field, in order to achieve an efficient and realistic try-on effect, a dual-branch diffusion model structure is provided and a model training strategy dominated by clothing detail information is designed. The combination of the two can ensure the fidelity of clothing in the try-on result image and improve production efficiency by reducing the sampling steps in the reasoning process. This application has conducted a large number of experiments on an open source virtual try-on dataset, and the results show that the solution provided by this application greatly improves the fidelity of clothing texture details and the generation speed.

[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0135] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0137] According to an embodiment of the present application, a training method for an image generation model is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0138] Figure 5 is a flowchart of a training method for an image generation model according to an embodiment of the present application, such as Figure 5 As shown, the method includes:

[0139] Step S502, obtaining a first sample image of the sample object and a second sample image of the sample element;

[0140] Step S504, extracting features from the first sample image to obtain a sample object posture image;

[0141] Step S506, constructing a target constraint based on the sample object posture image and the first sample image;

[0142] Step S508: training the initial image generation model based on the first sample image, the second sample image and the target constraint to obtain an image generation model.

[0143] Through the above steps, a first sample image of a sample object and a second sample image of a sample element are obtained; a feature extraction is performed on the first sample image to obtain a sample object posture image; a target constraint is constructed based on the sample object posture image and the first sample image; an initial image generation model is trained based on the first sample image, the second sample image and the target constraint to obtain an image generation model, thereby achieving the purpose of improving the image generation effect. It is easy to notice that in the training process of the image generation model, the target constraint constructed by the first sample image of the sample object and the sample object posture image can improve the efficiency of the image generation model in processing posture information, so that the image generation model can pay more attention to the processing of sample elements in the second sample image, and the subsequent fusion effect of the first sample image and the second sample image, thereby improving the image processing effect of the image generation model, and the first branch of the image generation model can be used to extract features from the second image to capture the complex image features in the first image, and the image features are fused into the first image obtained after denoising, thereby improving the generation effect of the target image, thereby solving the technical problem of poor image generation effect in the related art.

[0144] According to an embodiment of the present application, an image generating method is also provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0145] Figure 6 is a flowchart of image generation according to an embodiment of the present application, such as Figure 6 As shown, the method includes:

[0146] Step S602, obtaining a first image of a model image and a second image of clothing elements;

[0147] Among them, clothing elements are used to represent elements to be displayed on the model image.

[0148] Step S604, using the first branch of the image generation model to perform feature extraction on the second image to obtain image features of the second image, and using the second branch of the image generation model to perform denoising on the first image to obtain a processed first image;

[0149] Among them, the image generation model is trained based on a first sample image of a sample model image, a second sample image of a sample clothing element and a target constraint, the target constraint is constructed based on the first sample image and a sample model posture image, and the sample model posture image is used to characterize the posture of the sample model image in the first sample image.

[0150] Step S606: fusing the second image and the processed first image based on the image features to obtain a target image.

[0151] Among them, clothing elements are displayed on the model image in the target image.

[0152] Through the above steps, a first image of a model image and a second image of clothing elements are obtained, wherein the clothing elements are used to characterize the elements to be displayed on the model image; the first branch of the image generation model is used to extract features of the second image to obtain image features of the second image, and the second branch of the image generation model is used to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample model image, a second sample image of a sample clothing element, and a target constraint, the target constraint is constructed based on the first sample image and a sample model posture image, and the sample model posture image is used to characterize the posture of the model image in the first sample image; the second image and the processed first image are fused based on the image features to obtain a target image, wherein clothing elements are displayed on the model image in the target image, thereby achieving the purpose of improving the image generation effect. It is easy to notice that, during the training process of the image generation model, the target constraint constructed by the first sample image of the sample model image and the sample object posture image can improve the efficiency of the image generation model in processing posture information, so that the image generation model can pay more attention to the processing of the sample clothing elements in the second sample image, and the subsequent fusion effect of the first sample image and the second sample image, thereby improving the image processing effect of the image generation model. The first branch of the image generation model can be used to extract features from the second image to capture the complex image features in the first image, and the image features can be fused into the first image obtained after denoising, thereby improving the generation effect of the target image, thereby solving the technical problem of poor image generation effect in related technologies.

[0153] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Figure 7 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Figure 7 As shown, the device 700 includes: an acquisition module 702 , an extraction module 704 , and a fusion module 706 .

[0154] Among them, the acquisition module is used to acquire a first image of the target object and a second image of the target element, wherein the target element is used to characterize the element to be displayed on the target object; the extraction module is used to use the first branch of the image generation model to perform feature extraction on the second image to obtain the image features of the second image, and use the second branch of the image generation model to perform denoising on the first image to obtain the processed first image, wherein the image generation model is trained based on the first sample image of the sample object, the second sample image of the sample element and the target constraint, the target constraint is constructed based on the first sample image and the sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; the fusion module is used to fuse the second image and the processed first image based on the image features to obtain the target image, wherein the target element is displayed on the target object in the target image.

[0155] It should be noted that the acquisition module 702, extraction module 704, and fusion module 706 correspond to steps S302 to S306 in the embodiment, and the three modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of the device and run in the AR / VR device provided in embodiment 1.

[0156] According to an embodiment of the present application, there is also provided a training device for an image generation model for implementing the training method for the above-mentioned image generation model. Figure 8 is a schematic diagram of a training device for an image generation model according to an embodiment of the present application, such as Figure 8 As shown, the device 800 includes: an acquisition module 802 , an extraction module 804 , a construction module 806 , and a training module 808 .

[0157] Among them, the acquisition module is used to acquire a first sample image of the sample object and a second sample image of the sample element; the extraction module is used to extract features from the first sample image to obtain a posture image of the sample object; the construction module is used to construct a target constraint based on the posture image of the sample object and the first sample image; the training module is used to train the initial image generation model based on the first sample image, the second sample image and the target constraint to obtain an image generation model.

[0158] It should be noted that the acquisition module 802, extraction module 804, construction module 806, and training module 808 correspond to steps S502 to S508 in the embodiment, and the four modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules or units can be hardware components or software components stored in a memory and processed by one or more processors, and the above modules can also be run in the AR / VR device provided in embodiment 1 as part of the device.

[0159] According to an embodiment of the present application, an image generating device for implementing the above-mentioned image generating method is also provided. Fig. 9 is a schematic diagram of an image generating device according to an embodiment of the present application, such as Fig. 9 As shown, the device 900 includes: an acquisition module 902 , an extraction module 904 , and a fusion module 906 .

[0160] Among them, the acquisition module is used to acquire a first image of a model image and a second image of clothing elements, wherein the clothing elements are used to characterize the elements to be displayed on the model image; the extraction module is used to use the first branch of the image generation model to perform feature extraction on the second image to obtain the image features of the second image, and use the second branch of the image generation model to perform denoising on the first image to obtain the processed first image, wherein the image generation model is trained based on a first sample image of a sample model image, a second sample image of a sample clothing element and a target constraint, the target constraint is constructed based on the first sample image and a sample model posture image, and the sample model posture image is used to characterize the posture of the sample model image in the first sample image; the fusion module is used to fuse the second image and the processed first image based on the image features to obtain a target image, wherein the model image in the target image is displayed with clothing elements.

[0161] It should be noted that the acquisition module 902, extraction module 904, and fusion module 906 correspond to steps S602 to S606 in the embodiment, and the three modules and corresponding steps implement the same instances and application scenarios, but are not limited to the contents disclosed in the above embodiment 1. It should be noted that the above modules or units may be hardware components or software components stored in a memory and processed by one or more processors, and the above modules may also be part of the device and run in the AR / VR device provided in embodiment 1.

[0162] An embodiment of the present application may provide a computer terminal, which may include a server and a client. The server may be any server in a server device group or a cloud server.

[0163] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of the computer network.

[0164] In this embodiment, the computer terminal can execute the program code in the method.

[0165] Optionally, Fig.10 is a structural block diagram of a computer terminal according to an embodiment of the present application. Fig.10 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 102, a memory 104, a storage controller, and a peripheral interface, wherein the peripheral interface is connected to a radio frequency module, an audio module, and a display.

[0166] Among them, the memory can be used to store software programs and modules, such as program instructions / modules corresponding to the methods and devices in the embodiments of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, implementing the methods in the above embodiments. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0167] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a first image of the target object and a second image of the target element, wherein the target element is used to characterize the element to be displayed on the target object; use the first branch of the image generation model to extract features of the second image to obtain image features of the second image, and use the second branch of the image generation model to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of the sample object, a second sample image of the sample element, and a target constraint, the target constraint is constructed based on the first sample image and the sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; the second image and the processed first image are fused based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image.

[0168] The processor can call the information and application stored in the memory through the transmission device to execute the following steps: obtain a first sample image of the sample object and a second sample image of the sample element; perform feature extraction on the first sample image to obtain a posture image of the sample object; construct a target constraint based on the posture image of the sample object and the first sample image; and train an initial image generation model based on the first sample image, the second sample image and the target constraint to obtain an image generation model.

[0169] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain a first image of a model image and a second image of clothing elements, wherein the clothing elements are used to characterize the elements to be displayed on the model image; use the first branch of the image generation model to extract features of the second image to obtain image features of the second image, and use the second branch of the image generation model to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample model image, a second sample image of a sample clothing element, and a target constraint, the target constraint is constructed based on the first sample image and a sample model posture image, and the sample model posture image is used to characterize the posture of the sample model image in the first sample image; the second image and the processed first image are fused based on the image features to obtain a target image, wherein the model image in the target image is displayed with clothing elements.

[0170] By using an embodiment of the present application, a first image of a target object and a second image of a target element are obtained, wherein the target element is used to characterize the element to be displayed on the target object; a first branch of an image generation model is used to extract features of the second image to obtain image features of the second image, and a second branch of the image generation model is used to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample object, a second sample image of a sample element, and a target constraint, the target constraint is constructed based on the first sample image and a sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; the second image and the processed first image are fused based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image, thereby achieving the purpose of improving the image generation effect. It is easy to notice that, during the training process of the image generation model, the target constraint constructed by the first sample image of the sample object and the posture image of the sample object can improve the efficiency of the image generation model in processing posture information, so that the image generation model can pay more attention to the processing of sample elements in the second sample image, and the subsequent fusion effect of the first sample image and the second sample image, thereby improving the image processing effect of the image generation model. The first branch of the image generation model can be used to extract features from the second image to capture the complex image features in the first image, and the image features can be fused into the first image obtained after denoising processing, thereby improving the generation effect of the target image, thereby solving the technical problem of poor image generation effect in related technologies.

[0171] It can be understood by those skilled in the art that Fig.10 The structure shown is for illustration only, and the computer terminal A may also be a smart phone (such as , tablet computers, PDAs, mobile Internet devices (MID), PADs and other terminal devices. Fig.10 The structure of the electronic device is not limited. For example, the computer terminal A may also include Fig.10 The EMBODIMENT may include more or fewer components (such as network interfaces, display devices, etc.) as shown in the figure, or may have a different configuration than that shown in the figure above.

[0172] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0173] The embodiment of the present application further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the method provided in the above embodiment.

[0174] Optionally, in this embodiment, the computer-readable storage medium may be located in any one of the computer terminals in the computer terminal cluster, or in any one of the mobile terminals in the mobile terminal group.

[0175] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining a first image of a target object and a second image of a target element, wherein the target element is used to characterize an element to be displayed on the target object; performing feature extraction on the second image using a first branch of an image generation model to obtain image features of the second image, and performing denoising on the first image using a second branch of the image generation model to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample object, a second sample image of a sample element, and a target constraint, the target constraint is constructed based on the first sample image and a sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; and fusing the second image and the processed first image based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image.

[0176] Optionally, in this embodiment, the computer-readable storage medium is configured to store program code for executing the following steps: obtaining a first sample image of a sample object and a second sample image of a sample element; performing feature extraction on the first sample image to obtain a posture image of the sample object; constructing a target constraint based on the posture image of the sample object and the first sample image; and training an initial image generation model based on the first sample image, the second sample image and the target constraint to obtain an image generation model.

[0177] Optionally, in this embodiment, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining a first image of a model image and a second image of clothing elements, wherein the clothing elements are used to characterize elements to be displayed on the model image; performing feature extraction on the second image using a first branch of an image generation model to obtain image features of the second image, and performing denoising on the first image using a second branch of the image generation model to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample model image, a second sample image of a sample clothing element, and a target constraint, the target constraint is constructed based on the first sample image and a sample model posture image, and the sample model posture image is used to characterize the posture of the sample model image in the first sample image; and the second image and the processed first image are fused based on the image features to obtain a target image, wherein clothing elements are displayed on the model image in the target image.

[0178] By using an embodiment of the present application, a first image of a target object and a second image of a target element are obtained, wherein the target element is used to characterize the element to be displayed on the target object; a first branch of an image generation model is used to extract features of the second image to obtain image features of the second image, and a second branch of the image generation model is used to perform denoising on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample object, a second sample image of a sample element, and a target constraint, the target constraint is constructed based on the first sample image and a sample object posture image, and the sample object posture image is used to characterize the posture of the sample object in the first sample image; the second image and the processed first image are fused based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image, thereby achieving the purpose of improving the image generation effect. It is easy to notice that, during the training process of the image generation model, the target constraint constructed by the first sample image of the sample object and the posture image of the sample object can improve the efficiency of the image generation model in processing posture information, so that the image generation model can pay more attention to the processing of sample elements in the second sample image, and the subsequent fusion effect of the first sample image and the second sample image, thereby improving the image processing effect of the image generation model. The first branch of the image generation model can be used to extract features from the second image to capture the complex image features in the first image, and the image features can be fused into the first image obtained after denoising processing, thereby improving the generation effect of the target image, thereby solving the technical problem of poor image generation effect in related technologies.

[0179] The embodiment of the present application further provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and the computer program implements the method provided in the embodiment when executed by a processor.

[0180] The embodiments of the present application also provide a computer program product. Optionally, the computer program product may include a non-volatile computer-readable storage medium, which may be used to store a computer program, and when the computer program is executed by a processor, the method provided in the embodiments is implemented.

[0181] The embodiment of the present application further provides a computer program. Optionally, in this embodiment, when the computer program is executed by a processor, the method provided in the above embodiment is implemented.

[0182] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0183] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0184] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0185] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0186] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0187] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0188] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. An image generation method, characterized in that: include: Acquire a first image of a target object and a second image of a target element, wherein the target element is used to represent an element to be displayed on the target object; Using a first branch of an image generation model to perform feature extraction on the second image to obtain image features of the second image, and using a second branch of the image generation model to perform denoising processing on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample object, a second sample image of a sample element, and a target constraint, the target constraint is constructed based on the first sample image and a sample object posture image, the sample object posture image is used to characterize the posture of the sample object in the first sample image, the processed first image is obtained by denoising the first image based on an object posture image, a mask region image, and latent space noise through a second branch, the latent space noise is obtained by encoding the object posture image and the mask region image into a latent space using a feature encoder in the image generation model, and the object posture image and the mask region image are obtained by performing feature extraction on the first image; The second image and the processed first image are fused based on the image features to obtain a target image, wherein the target element is displayed on the target object in the target image.

2. The image generation method according to claim 1, characterized in that: Performing denoising processing on the first image by using the second branch of the image generation model to obtain a processed first image, including: Performing feature extraction on the first image to obtain an object posture image and a mask area image, wherein the object posture image is used to represent the posture of the target object in the first image, and the mask area image is used to represent an area image of the target element to be displayed in the first image; The second branch is used to perform denoising processing on the first image based on the object posture image and the mask area image to obtain the processed first image.

3. The image generation method according to claim 2, characterized in that: The first branch includes multiple first network layers, and the second branch includes multiple second network layers. Different first network layers are used to output image features of different dimensions. The outputs of the multiple first network layers are correspondingly connected to the inputs of multiple cross-attention modules, and the inputs of the multiple second network layers are correspondingly connected to the outputs of the multiple cross-attention modules.

4. The image generation method according to claim 2, characterized in that: The image generation model further includes a feature encoder, and uses the second branch of the image generation model to perform denoising on the first image to obtain a processed first image, including: Encoding the object posture image and the mask region image into a latent space using the feature encoder to obtain a first posture feature, a mask region feature, and latent space noise, wherein the first posture feature is used to represent other posture features of the target object except for the posture feature associated with a preset element, and the preset element and the target element belong to the same category; The second branch is used to perform denoising processing on the first image based on the object posture image, the mask area image and the latent space noise to obtain the processed first image.

5. The image generation method according to claim 3, characterized in that: The method further comprises fusing the second image and the processed first image based on the image feature to obtain a target image, including: The second branch is used to fuse the image features of different dimensions with the processed first image to obtain the target image.

6. The image generation method according to claim 4, characterized in that: Using the second branch to fuse the image features of different dimensions with the processed first image to obtain the target image includes: Performing feature extraction on the first image and / or the second image to obtain enhanced features; The enhanced features, the processed first image and the image features of different dimensions are fused using the second branch to obtain the target image.

7. The image generation method according to claim 6, characterized in that: Extracting features from the first image and / or the second image to obtain enhanced features includes: performing feature extraction on the second image to obtain global image features; Extracting features from the first image to obtain a second posture feature, wherein the second posture feature is used to represent a global posture feature of the target object; The enhanced feature is determined based on the global image feature and / or the second posture feature.

8. The image generation method according to claim 1, characterized in that: The method further comprises: Performing feature extraction on the first sample image to obtain the sample object posture image; constructing a target constraint based on the sample object posture image and the first sample image; An initial image generation model is trained based on the first sample image, the second sample image and the target constraint to obtain the image generation model.

9. The image generation method according to claim 8, characterized in that: Constructing a target constraint based on the sample object posture image and the first sample image includes: Extracting a first attention feature and a second attention feature of the sample object posture image and the first sample image respectively; Performing singular value decomposition on the first attention feature and the second attention feature respectively to obtain first structural information of the sample object posture image and second structural information of the first sample image, wherein the singular value decomposition is used to describe the structural information of the feature by means of matrix decomposition; The target constraint is constructed by minimizing the Euclidean distance between the first structural information and the second structural information.

10. The image generation method according to claim 8, characterized in that: Training an initial image generation model based on the first sample image, the second sample image and the target constraint to obtain the image generation model includes: In response to a first time step of the initial image generation model in the training process being greater than a preset time step, an initial second branch in the initial image generation model is trained based on the target constraint and the first sample image in the first preset time step, and an initial first branch in the initial image generation model is trained based on the second sample in the second preset time step to obtain the image generation model, wherein a sum of the first preset time step and the second preset time step is the first time step; In response to the second time step of the initial image generation model in the training process being less than or equal to the preset time step, the initial second branch is trained based on the first sample image in a third preset time step, and the initial first branch is trained based on the second sample image in a fourth preset time step to obtain the image generation model, and the sum of the third preset time step and the fourth preset time step is the second time step; Among them, the proportion of the first time step is less than the proportion of the second time step, the proportion of the first time step is the ratio of the first preset time step to the second preset time step, and the proportion of the second time step is the ratio of the third preset time step to the fourth preset time step.

11. A training method for an image generation model, characterized in that: include: Acquire a first sample image of the sample object and a second sample image of the sample element; Performing feature extraction on the first sample image to obtain a sample object posture image and a sample mask area image; constructing a target constraint based on the sample object posture image and the first sample image; The initial image generation model is trained based on the first sample image, the second sample image and the target constraint to obtain the image generation model, wherein the image generation model includes a first branch and a second branch, the first branch is used to extract features from the second image of the target element to obtain image features of the second image, the second branch is used to denoise the first image of the target object to obtain a processed first image, the processed first image is obtained by denoising the first image based on the object posture image, the mask area image and latent space noise through the second branch, the latent space noise is obtained by encoding the object posture image and the mask area image into a latent space using a feature encoder in the image generation model, and the object posture image and the mask area image are obtained by extracting features from the first image.

12. An image generation method, characterized in that: include: Acquire a first image of a model image and a second image of a clothing element, wherein the clothing element is used to represent an element to be displayed on the model image; The first branch of the image generation model is used to perform feature extraction on the second image to obtain image features of the second image, and the second branch of the image generation model is used to perform denoising processing on the first image to obtain a processed first image, wherein the image generation model is trained based on a first sample image of a sample model image, a second sample image of a sample clothing element, and a target constraint, the target constraint is constructed based on the first sample image and a sample model posture image, the sample model posture image is used to characterize the posture of the sample model image in the first sample image, the processed first image is obtained by denoising the first image based on an object posture image, a mask area image, and latent space noise through the second branch, the latent space noise is obtained by encoding the object posture image and the mask area image into a latent space using a feature encoder in the image generation model, and the object posture image and the mask area image are obtained by performing feature extraction on the first image; The second image and the processed first image are fused based on the image features to obtain a target image, wherein the clothing elements are displayed on the model image in the target image.

13. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 12 when running.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 12.

15. A computer program product, characterized in that It comprises a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images