Image generation method and device, equipment and storage medium

Abstract

The embodiment of the invention provides an image generation method and device, equipment and a storage medium, and relates to the technical field of artificial intelligence, in the method, character features of a target character are extracted from a stylized image of the target character through a de-stylization model; and then predicting target noise information irrelevant to the image features of the realistic image and the character features of the target character in the pure noise image. And removing target noise information from the pure noise image to obtain a target realistic image of the target person. According to the method, the character features of the target character in the stylized image are taken as the reference information, the target realistic image of the target character is generated in combination with the image features of the realistic image, and the whole process is irrelevant to the style of the stylized image, so that the corresponding realistic image can be generated for any style or any character; the accuracy of generating the realistic image and the stylized image of the same character is improved, and the expandability and generalization ability are also improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to an image generation method, apparatus, device and storage medium. Background Art

[0002] Character stylization technology is a technology that can convert real-life images (also called realistic images) into stylized images. It can be used in a variety of fields such as artistic creation, video entertainment, virtual reality, and games. Character stylization technology is mainly achieved by training character stylization models. Before training a character stylization model, a large number of training samples need to be prepared. Each training sample is a realistic image and a stylized image of the same person.

[0003] In order to obtain realistic images and stylized images of the same person, the related technology first obtains a pre-trained latent diffusion model (LDM), which can generate an image with the same meaning as the text information based on the input text information. Then, the latent diffusion model is fine-tuned and trained using multiple stylized images to obtain a first diffusion model, which can generate a stylized image with the same meaning as the text information based on the input text information. The latent diffusion model is fine-tuned and trained using multiple realistic images to obtain a second diffusion model, which can generate a realistic image with the same meaning as the text information based on the input text information. Finally, a character description text is used as input to generate a stylized image of the character through the first diffusion model; similarly, a character description text is used as input to generate a realistic image of the character through the second diffusion model, thereby obtaining a realistic image and a stylized image of the same character.

[0004] Since the first diffusion model and the second diffusion model are two models obtained through independent training, the two models have different degrees of understanding of text information. In this case, for the same person, the character features in the stylized image generated by the first diffusion model may be quite different from the character features in the realistic image generated by the second diffusion model, that is, the realistic image and the stylized image do not correspond to the same person. Therefore, the accuracy of generating realistic images and stylized images of the same person using the above scheme is low. Summary of the invention

[0005] The present application provides an image generation method, apparatus, device and storage medium for improving the accuracy of generating realistic images and stylized images of the same person. On the one hand, an embodiment of the present application provides an image generation method, the method comprising: Get a pure noise image and a stylized image of the target person; Extracting the character features of the target person from the stylized image of the target person; Extracting target noise information that does not satisfy a correlation condition and image features of a realistic image and character features of a target character from the pure noise image, wherein the image features of the realistic image are obtained by learning from a plurality of sample realistic images; The target noise information is removed from the pure noise image to obtain a target realistic image of the target person.

[0006] On the one hand, an embodiment of the present application provides an image generating device, the device comprising: An acquisition module, used to acquire a pure noise image and a stylized image of a target person; An extraction module, used to extract the character features of the target person from the stylized image of the target person; A prediction module, used to extract target noise information that does not meet the correlation condition and has image features of a realistic image and character features of the target character from the pure noise image, wherein the image features of the realistic image are obtained by learning from a plurality of sample realistic images; The denoising module is used to remove the target noise information from the pure noise image to obtain a target realistic image of the target person.

[0007] Optionally, the extraction module is specifically used for: The trained de-stylization model performs the following operations: Noising the stylized image of the target person according to a preset noise adding mode to obtain a target noisy image, wherein the preset noise adding mode is obtained by training the de-stylization model; Extracting the person features of the target person from the target noisy image.

[0008] Optionally, the preset noise adding mode includes: a plurality of noise adding time steps and first noises corresponding to the plurality of noise adding time steps; the first noises corresponding to the plurality of noise adding time steps have different intensities; The extraction module is specifically used for: In the multiple noise adding time steps, the stylized image is iteratively noised to obtain the target noisy image; wherein each round of iterative noise adding comprises the following steps: Obtain the first noise of a noise adding time step corresponding to this round of iterative noise adding; The first noise of the noise adding time step is used to add noise to the stylized image after the previous round of iterative noise adding, so as to obtain the stylized image after the current round of iterative noise adding.

[0009] Optionally, the prediction module is specifically used for: The following operations are performed using the trained de-stylization model: Splicing the image features of the realistic image and the character features of the target character to obtain a target splicing feature; Using the target splicing feature, predicting the probability that each pixel in the pure noise image contains noise information; Based on the probability that each pixel in the pure noise image contains noise information, obtaining the noise distribution in the pure noise image; Based on the noise distribution in the pure noise image, the target noise information in the pure noise image is extracted.

[0010] Optionally, the target noise information includes: second noise corresponding to each of the multiple denoising time steps; the second noise corresponding to each of the multiple denoising time steps has different intensities; The denoising module is specifically used for: The following operations are performed using the trained de-stylization model: In the multiple denoising time steps, the pure noise image is iteratively denoised to obtain a target realistic image of the target person; wherein each round of iterative denoising includes the following steps: Get the second noise of a denoising time step corresponding to this round of iterative denoising; The second noise of the one denoising time step is removed from the pure noise image after the previous round of iterative denoising to obtain the pure noise image after the current round of iterative denoising.

[0011] Optionally, the image features of the realistic type image include: sub-image features corresponding to each of the multiple denoising time steps; The denoising module is specifically used for: Splicing the character features of the target person with the sub-image features corresponding to the one denoising time step to obtain the sub-splicing features corresponding to the one denoising time step; Based on the sub-joint features corresponding to the one denoising time step, predicting the probability that each pixel in the pure noise image after the previous round of iterative denoising contains noise information; Obtaining noise distribution in the pure noise image after the previous round of iterative denoising based on the probability that each pixel in the pure noise image after the previous round of iterative denoising contains noise information; Based on the noise distribution in the pure noise image after the previous round of iterative denoising, the second noise corresponding to the one denoising time step is obtained.

[0012] Optionally, a module training module is also included; The module training module is specifically used for: Obtain a sample set including multiple training samples, each training sample including: a sample style image and a sample realistic image of the same sample person; The sample set is used to perform multiple rounds of iterative training on the de-stylization model to be trained until a training termination condition is met, thereby obtaining the trained de-stylization model, wherein each round of iterative training process includes the following steps: Using the de-stylization model used in this round, each training sample input in this round of iteration is converted to obtain a predicted realistic image of each training sample; According to the obtained predicted realistic images and the corresponding sample realistic images, the model loss value of the de-stylization model used in this round is determined, and after adjusting the parameters of the de-stylization model used in this round according to the model loss value, the de-stylization model with the adjusted parameters is used to enter the next round of iterative training.

[0013] Optionally, the module training module is specifically used for: For each training sample, the following operations are performed respectively: Noising a sample style image in a training sample to obtain a first noisy image; and Noising a sample realistic image in the training sample to obtain a second noisy image; Extracting character features of a sample character from the first noisy image; and extracting image features of a realistic image from the second noisy image; Based on the character features of the sample character and the image features of the realistic type image, a predicted realistic image of the sample character is obtained.

[0014] Optionally, the module training module is specifically used for: Based on the character features of the sample character and the image features of the realistic image, stitching is performed to obtain a sample stitching feature; Based on the sample splicing features, obtaining predicted noise information in the second noisy image; The predicted noise information is removed from the second noisy image to obtain a predicted realistic image of the sample person.

[0015] On the one hand, an embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the above-mentioned image generation method is implemented, including: The processor acquires a pure noise image and a stylized image of the target person; The processor extracts the character features of the target person from the stylized image of the target person; and extracts the image features of the realistic type image and the character features of the target person from the pure noise image, and the target noise information that does not meet the correlation condition, wherein the image features of the realistic type image are learned from multiple sample realistic images and stored in a memory.

[0016] The processor removes the target noise information from the pure noise image to obtain a target realistic image of the target person.

[0017] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by a computer device. When the program is run on the computer device, the computer device executes the steps of the above-mentioned image generation method.

[0018] On the one hand, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned image generation method.

[0019] In an embodiment of the present application, the character features of the target person in the stylized image are used as reference information, combined with the image features of the realistic type images learned from multiple sample realistic images, to generate a target realistic image of the target person. This ensures that the obtained stylized image and the target realistic image correspond to the same person, thereby improving the accuracy of generating realistic images and stylized images of the same person.

[0020] Secondly, the character features of the target person are extracted from the stylized image to guide the generation of the target realistic image of the target person. It is not necessary to extract the style features of the stylized image. That is, the process of generating a realistic image of a person by the de-stylized model is independent of the style of the input stylized image. Therefore, corresponding realistic images can be generated for any style or any person, realizing zero-shot generation of realistic images and stylized images of the same person, improving the scalability and generalization ability of the de-stylized model, and avoiding resource consumption caused by repeated training.

[0021] In addition, the technical solution of the present application ensures that the obtained stylized image and the realistic image correspond to the same person, so when the obtained stylized image and the realistic image are used as training samples for subsequent model training, the accuracy of the training samples is correspondingly improved. Since the de-stylization model of the present application has zero-shot capability, a large number of diverse training samples can be generated through the de-stylization model for subsequent model training without repeatedly training the de-stylization model, thereby improving the performance of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0023] Figure 1 A schematic diagram of a system architecture provided in an embodiment of the present application; Figure 2 An example of an application scenario provided by the present application Figure 1 ; Figure 3 An example of an application scenario provided by the present application Figure 2 ; Figure 4 A schematic diagram of the structure of a de-stylization model provided in an embodiment of the present application; Figure 5 A schematic diagram of a training method for a de-stylization model provided in an embodiment of the present application Figure 1 ; Figure 6 A style schematic diagram of a style image provided in an embodiment of the present application; Figure 7 A schematic diagram of a Cross-Attention layer provided in an embodiment of the present application; Figure 8 A flowchart of a fine-tuning training method for a UNet network provided in an embodiment of the present application; Fig. 9 A schematic diagram of a flow chart of a method for combining features of a denoising network and a reference network provided in an embodiment of the present application; Fig.10 A schematic diagram of a method for generating an image provided in an embodiment of the present application Figure 1 ; Fig.11A A schematic diagram of a training method for a de-stylization model provided in an embodiment of the present application Figure 2 ; Fig. 11B A schematic diagram of a method for generating an image provided in an embodiment of the present application Figure 2 ; Fig. 11C A schematic diagram of the structure of an image generating device provided in an embodiment of the present application; Fig.12 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and beneficial effects of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0025] For ease of understanding, the terms involved in the embodiments of the present invention are explained below.

[0026] The embodiments of the present application relate to artificial intelligence (AI) technology, and are mainly designed based on computer vision (CV) technology in artificial intelligence technology.

[0027] Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0028] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, pre-trained models are also called large models and basic models. After fine-tuning, they can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0029] Computer vision technology is a science that studies how to make machines "see". To put it more specifically, it refers to using cameras and computers to replace human eyes to identify and measure targets, and further perform image processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multidimensional data. Large model technology has brought important changes to the development of computer vision technology. Pre-trained models in various visual fields can be quickly and widely applied to downstream specific tasks after fine-tuning. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0030] Diffusion models refer to models that draw images based on text or images input by users. Diffusion models include two main processes: forward diffusion and backward diffusion. In the forward diffusion stage, the image is contaminated by gradually introduced noise until the image becomes completely random noise. In the backward process, a series of Markov chains are used to gradually remove the prediction noise at each time step to recover the data from Gaussian noise. Due to the ability of diffusion models to preserve the semantic structure of data, they have been shown to be immune to mode collapse.

[0031] The Latent Diffusion Model (LDM) is a deep learning model for generating images. Its core idea is to generate images through a diffusion process in the latent space. LDM decomposes the generation task into a process of converting noise to data, so that the model can efficiently generate high-quality images.

[0032] Zero-shot: A learning method that aims to enable the model to recognize, classify, or understand categories or concepts that have not been seen during training. This learning method is crucial to solving the long-tail distribution problem common in the real world, that is, for some rare or unknown categories of samples, traditional supervised learning methods may be difficult to handle. Specifically, a large amount of training set data is used to train the model, and without further training, the model's generalization is used to transfer the model's capabilities to the test set. There is no intersection between the training set categories and the test set categories, so there is no need to continue training, which significantly reduces the training cost.

[0033] The LoRA model (Low-Rank Adaptation of Large Language Models) is a fine-tuning method for large language models. It reduces parameters and fine-tuning costs by introducing low-rank matrices while maintaining model performance. The core idea of ​​LoRA is to freeze the pre-trained model weight parameters and only train the newly added network layer parameters. This method is achieved by introducing low-rank matrices, which have a small number of parameters, thereby significantly reducing the cost of fine-tuning. Specifically, LoRA injects low-rank matrices into large pre-trained language models to adapt to specific tasks, which not only retains the generalization ability of the original model, but also significantly reduces the computing resources and data required for fine-tuning.

[0034] Unet network: A denoising network that mainly includes two structures: Residual Network (Residual Network, ResNet) and Transformer (a neural network model based on the self-attention mechanism). ResNet is mainly composed of a convolutional network and is responsible for encoding image features. Transformer is mainly composed of multiple attention mechanism (Attention) layers. The calculation formula of the Attention layer is shown in formula (1):

[0035] Among them, Q is the query matrix, K is the key matrix, and V is the value matrix.

[0036] The Attention layer is responsible for encoding image features and introduced conditional features (text, image, audio, etc.). The Attention layer in Transformer mainly includes two forms: Self-Attention and Cross-Attention. When Q, K, and V are the same (for example, they are all image features), the Attention layer is in the Self-Attention form, which is responsible for calculating the relationship between image features. When Q is an image feature and K and V are other conditional features, the Attention layer is in the Cross-Attention form, which is responsible for calculating the relationship between image features and other features.

[0037] Realistic images: images that are closer to the real world, such as images taken by a mobile phone or camera.

[0038] Stylized images: images that tend to be artistic, such as watercolor style images, pixel style images, line style images, cartoon rendering style images, etc.

[0039] Stylized data pairs: realistic images and stylized images of the same person.

[0040] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless cars, autonomous driving, drones, digital twins, virtual humans, robots, Artificial Intelligence Generated Content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0041] The solution provided in the embodiment of the present application mainly relates to the application of artificial intelligence technology in the image generation scenario. In the specific process, the character features of the target person are extracted from the stylized image of the target person. Then, the noise information in the pure noise image that is unrelated to the image features of the realistic image and the character features of the target person is predicted. Then, the noise information is removed from the pure noise image to obtain a target realistic image of the target person.

[0042] Specifically, in the embodiment of the present application, the image generation process can be divided into two parts, including a training part and an application part, wherein the training part involves the technical field of machine learning. In the training part, the embodiment of the present application uses stylized data pairs (i.e., realistic images and stylized images of the same person) as training data, performs iterative training on the de-stylized model to be trained, and continuously adjusts the model parameters through the optimization algorithm until the model converges, so that the trained de-stylized model has the ability to generate a realistic image of any person in zero-shot.

[0043] In the application part, the de-stylization model trained in the training part is used to convert a stylized image of a person input during actual use to obtain a realistic image of the person.

[0044] In addition, it should be noted that the artificial neural network model in the embodiment of the present application can be trained online or offline, which is not specifically limited here. In this article, offline training is used as an example for illustration.

[0045] It is understandable that in the specific implementation of the present application, related data such as stylized images of characters, realistic images of characters, etc., when the embodiments in the present application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0046] In practical applications, in order to implement character stylization technology, it is often necessary to prepare a large amount of stylized data pairs to train character stylization models. However, it is relatively simple to obtain stylized images of multiple characters or real-life images (i.e., realistic images) of multiple characters separately, but it is difficult to obtain a large number of stylized data pairs (realistic images and stylized images of the same character).

[0047] In order to obtain realistic images and stylized images of the same person, the related technology first obtains a pre-trained latent diffusion model, which can generate an image with the same meaning as the text information based on the input text information. Then, the Lora fine-tuning scheme is used to fine-tune the latent diffusion model based on multiple stylized images to obtain a first diffusion model, which can generate a stylized image with the same meaning as the text information based on the input text information. The Lora fine-tuning scheme is used to fine-tune the latent diffusion model based on multiple realistic images to obtain a second diffusion model, which can generate a realistic image with the same meaning as the text information based on the input text information. Finally, a character description text is used as input to generate a stylized image of the character through the first diffusion model; similarly, a character description text is used as input to generate a realistic image of the character through the second diffusion model, thereby obtaining a realistic image and a stylized image of the same character.

[0048] Since the first diffusion model and the second diffusion model are two models obtained through independent training, the two models have different degrees of understanding of text information. In this case, for the same person, the character features in the stylized image generated by the first diffusion model may be quite different from the character features in the realistic image generated by the second diffusion model, that is, the realistic image and the stylized image do not correspond to the same person. Therefore, the accuracy of generating realistic images and stylized images of the same person using the above scheme is low.

[0049] If the first diffusion model and the second diffusion model are fused into the same model, at this time, since the positions of the additional parameter matrices introduced in the LDM model are the same in the process of respectively training the first diffusion model and the second extended model using the Lora fine-tuning scheme, there is a position conflict problem when the first diffusion model and the second extended model are fused. In this case, only the fusion weights of the first diffusion model and the second extended model can be adjusted. Then, the first diffusion model generates a stylized image according to the fusion weights alone, and the second diffusion model generates a realistic image according to the fusion weights alone. In essence, the first diffusion model and the second diffusion model generate stylized images and realistic images independently. Therefore, the accuracy of generating realistic images and stylized images of the same person by this scheme is also low.

[0050] Secondly, the first diffusion model and the second diffusion model have poor scalability; specifically, the first diffusion model and the second diffusion model can only generate images with the same characters or styles as the training samples, and cannot expand styles or characters that have not been trained. In this way, each time a new style or character is added, fine-tuning training needs to be re-performed, and each fine-tuning training requires dozens or even hundreds of training samples and takes dozens of minutes, resulting in high training costs and large training time expenditures.

[0051] In view of this, an embodiment of the present application provides an image generation method, in which a pure noise image and a stylized image of a target person are used as inputs of a de-stylization model, and the character features of the target person are extracted from the stylized image of the target person through the de-stylization model. Then, from the pure noise image, target noise information that does not meet the correlation condition with the image features of the realistic type image and the character features of the target person is extracted. Then, the target noise information is removed from the pure noise image to obtain a target realistic image of the target person.

[0052] That is to say, in the embodiment of the present application, the character features of the target person in the stylized image are used as reference information, combined with the image features of the realistic type images learned from multiple sample realistic images, to generate a target realistic image of the target person. This ensures that the obtained stylized image and the target realistic image correspond to the same person, thereby improving the accuracy of generating realistic images and stylized images of the same person.

[0053] Secondly, the character features of the target person are extracted from the stylized image to guide the generation of the target realistic image of the target person. It is not necessary to extract the style features of the stylized image. That is, the process of generating a realistic image of a person by the de-stylized model is independent of the style of the input stylized image. Therefore, corresponding realistic images can be generated for any style or any person, realizing zero-shot generation of realistic images and stylized images of the same person, improving the scalability and generalization ability of the de-stylized model, and avoiding resource consumption caused by repeated training.

[0054] In addition, the technical solution of the present application ensures that the obtained stylized image and the realistic image correspond to the same person, so when the obtained stylized image and the realistic image are used as training samples for subsequent model training, the accuracy of the training samples is correspondingly improved. Since the de-stylization model of the present application has zero-shot capability, a large number of diverse training samples can be generated through the de-stylization model for subsequent model training without repeatedly training the de-stylization model, thereby improving the performance of model training.

[0055] The following is a brief introduction to the system architecture diagram applicable to the technical solution of the embodiment of the present application. It should be noted that the system architecture diagram introduced below is only used to illustrate the embodiment of the present application and is not a limitation.

[0056] refer to Figure 1 , which is a system architecture diagram applicable to an embodiment of the present application. The system architecture includes at least a terminal device 101 and a server 102. The number of terminal devices 101 can be one or more, and the number of servers 102 can also be one or more. The present application does not specifically limit the number of terminal devices 101 and servers 102.

[0057] The terminal device 101 can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart home appliance, an intelligent voice interaction device, an intelligent vehicle-mounted device, etc., but is not limited thereto.

[0058] Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, but is not limited to these.

[0059] It should be noted that the method in the embodiment of the present application can be executed by the terminal device 101 or the server 102 alone, or can be executed by the terminal device 101 and the server 102 together.

[0060] When the terminal device 101 or the server 102 executes the process independently, the de-stylization model is deployed on the terminal device 101 or the server 102. When the terminal device 101 or the server 102 executes the process independently, the model training and application process can be realized independently by the terminal device 101 or the server 102. For example, the de-stylization model can be trained on the terminal device 101 by the collected training data. Accordingly, after the training, the terminal device 101 can use the trained de-stylization model to generate a realistic image of a person, or the server 102 can execute the process.

[0061] When the server 102 and the terminal device 101 jointly execute, the server 102 can train the de-stylization model and then deploy the trained de-stylization model to the terminal device 101, and the terminal device 101 generates a realistic image of the person, or part of the model training or application process is implemented by the terminal device 101 and part of the process is implemented by the server 102, and the two cooperate to implement the model training or application process. In actual application, specific configurations can be made according to the situation, and this application does not make specific limitations here.

[0062] The server 102 and the terminal device 101 may each include one or more processors, memories, and interactive I / O interfaces, etc. In addition, the server 102 may also be configured with a database, which may be used to store model parameters of a de-stylized model, etc. The memories of the server 102 and the terminal device 101 may also store program instructions required to be executed in the image generation method provided in the embodiment of the present application, and these program instructions, when executed by the processor, may be used to implement the image generation process provided in the embodiment of the present application.

[0063] It should be noted that when the image generation method provided in the embodiment of the present application is executed by the server 102 or the terminal device 101 alone, the system architecture of the present application may also include only a single device of the server 102 or the terminal device 101, or the server 102 and the terminal device 101 may be considered to be the same device. Of course, in actual application, when the image generation method provided in the embodiment of the present application is jointly executed by the server 102 and the terminal device 101, the server 102 and the terminal device 101 may also be the same device, that is, the server 102 and the terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0064] In some embodiments, a user may provide a stylized image of a target person through the terminal device 101 to initiate an image generation process, and the server 102 may receive the stylized image of the target person provided by the user, and then adopt the image generation method of an embodiment of the present application to generate a target realistic image of the target person based on a pure noise image and the stylized image of the target person, and return it to the terminal device 101 for presentation.

[0065] In the embodiment of the present application, the terminal device 101 and the server 102 may be directly or indirectly connected to each other through one or more networks. The network may be a wired network or a wireless network, for example, the wireless network may be a mobile cellular network, or a wireless fidelity (Wireless-Fidelity, WIFI) network, and of course, other possible networks, which are not limited in the embodiment of the present application.

[0066] The following briefly introduces the application scenarios to which the technical solutions of the embodiments of the present application can be applied. It should be noted that the application scenarios introduced below are only used to illustrate the embodiments of the present application and are not limited. In the specific implementation process, the technical solutions provided by the embodiments of the present application can be flexibly applied according to actual needs.

[0067] The solution provided in the embodiments of the present application can be applied to image generation in various application scenarios, such as training data enhancement scenarios, game scenarios, extended reality (XR) scenarios, etc. The solution can also be applied as a basic technology in various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, audio and video, etc.

[0068] The following is an illustrative description of the application scenarios to which the solution provided in this application is applicable.

[0069] Application scenario 1: Training data enhancement scenario.

[0070] See also Figure 2 , set the target application as an application with the function of generating personalized portraits, and the target application realizes the function of generating personalized portraits through the deployed stylized model. Before training the stylized model, it is necessary to prepare a large number of stylized data pairs (i.e., realistic images and stylized images of the same person). The stylized data pairs are obtained in the following way: The terminal device 1 sends a set of stylized images of multiple characters to a server, and the server is deployed with a trained de-stylization model. In the server, for each stylized image in the stylized image set (such as the stylized image of character A), the collected pure noise image that meets the standard normal distribution and the stylized image of character A are input into the de-stylization model. The de-stylization model is used to extract the character features of character A from the stylized image of character A; and the noise information in the pure noise image that is unrelated to the image features of the realistic type image and the character features of character A is predicted. Then, the noise information is removed from the pure noise image to obtain a realistic image of character A. A training sample is obtained based on the realistic image of character A and the stylized image of character A.

[0071] It should be noted that by adjusting the pure noise image input to the de-stylization model, different realistic images of person A can be obtained, and then different training samples can be constructed.

[0072] The server may directly use the obtained multiple training samples to train a stylized model, and then deploy the trained stylized model on the server or terminal device 2. Terminal device 1 and terminal device 2 may be the same device or different devices.

[0073] The server may also send the obtained multiple training samples to the terminal device 2 , and the terminal device 2 may use the obtained multiple training samples to train a stylized model, and then deploy the trained stylized model on the server or the terminal device 2 .

[0074] In the application stage, it is assumed that the trained stylized model is deployed on the server. When the user uses the target application to generate a personalized character portrait, the target application is triggered on the terminal device 2 to display the input interface, and the user's realistic image (such as an ID photo) is submitted on the input interface. The terminal device 2 sends the user's realistic image to the server. The server inputs the user's realistic image into the trained stylized model, obtains the user's personalized character portrait, and returns the user's personalized character portrait to the terminal device 2. The terminal device 2 displays the user's personalized character portrait in the conversion result interface of the target application.

[0075] Application scenario 2: Game scenario.

[0076] See also Figure 3 In order to make the game characters closer to the real world and thus improve the gaming experience, the stylized images of the game characters can be converted into realistic images of the game characters through the de-stylization model of the present application.

[0077] Specifically, in the terminal device or server, the collected pure noise image that meets the standard normal distribution and the stylized image of the game character B are input into the de-stylization model. The de-stylization model extracts the character features of the game character B from the stylized image of the game character B, and predicts the noise information in the pure noise image that is unrelated to the image features of the realistic type image and the character features of the game character B. Then, the noise information is removed from the pure noise image to obtain the realistic image of the game character B.

[0078] It should be noted that the embodiments of the present application are not limited to the several application scenarios exemplified above, but may also be other application scenarios, and the present application does not make any specific limitation on this.

[0079] In the embodiment of the present application, the image generation process can be divided into two parts, including a training part and an application part. The training part and the application part are respectively introduced in detail below.

[0080] In order to more clearly introduce the contents of the training part and the application part of this application, we first introduce the de-stylization model structure involved in both the training part and the application part. Figure 4 , is a schematic diagram of a de-stylization model provided in an embodiment of the present application, the de-stylization model includes: A first encoding module 401 , a first denoising module 402 , a reference network 403 , a second encoding module 404 , a second denoising module 405 , a denoising network 406 , and a decoding module 407 .

[0081] The first encoding module 401 is used to map the stylized image of the character from the pixel space to the latent space. The first noise adding module 402 is used to add noise to the stylized image in the latent space, wherein the added noise is noise that satisfies the standard normal distribution, such as Gaussian noise.

[0082] The second encoding module 404 is used to map the realistic image of the person from the pixel space to the latent space. The second noise adding module 405 is used to add noise to the realistic image in the latent space.

[0083] The reference network 403 is used to extract character features from the stylized image after adding noise, and transfer the character features to the denoising network 406. The reference network 403 and the denoising network 406 have the same network structure.

[0084] The denoising network 406 is used to denoise the input noisy image using the character features as reference information to obtain a denoising result.

[0085] The decoding module 407 is used to map the denoising result from the latent space to the pixel space to obtain a realistic image of the person.

[0086] The following is a detailed introduction to the training process of the de-stylization model combined with the structure of the de-stylization model. Figure 5 , including the following steps: Step 501: Obtain a sample set including a plurality of training samples, each training sample including: a sample style image and a sample realistic image of the same sample person.

[0087] In this application, the characteristics of the style are abstracted into two aspects: structure and texture. Based on this, representative styles in each aspect are selected to achieve the purpose of expanding to other different styles. The structure mainly focuses on 2D, 3D and facial structure, etc.; the texture mainly focuses on light, color and material, etc.

[0088] For example, see Figure 6 , which are 8 different styles and corresponding features and example pictures provided in the embodiments of this application, wherein the 8 different styles are 3D real-life cartoon style, ink style, 3D cartoon style, ice sculpture style, 2D animation style, single-line drawing style, 2D comic style, and magic style.

[0089] In practical applications, we use Wensheng graph models (such as Flux1 dev model) to generate specific resolutions (such as ) stylized image of a person. Specifically, the input text template includes three adjustable placeholders, namely placeholder Token1, placeholder Token2 and placeholder Token3. Gender can be selected at placeholder Token1, for example, {"female", "male"}; age can be selected at placeholder Token2, for example, {5,10,20,30,40,50,60,70}; different styles can be selected at placeholder Token3, for example, 8 different styles. Under this text template, the noise random number of the model initialization input is changed to generate stylized images of multiple age groups, different genders, and different styles.

[0090] After generating stylized images of multiple age groups, different genders, and different styles, select images with consistent style and high quality. 80 stylized images can be retained for each style. These 80 stylized images are a combination of 2 genders, 8 age groups, and 5 stylized images. That is, for each of the 2 genders, the corresponding stylized images of the gender in the 8 age groups are retained, and 5 stylized images are retained for each age group.

[0091] Next, we obtain the StableDiffusion XL (SDXL) model trained with publicly available text-image pairs. The SDXL model itself has the ability to generate realistic images of specific people. For example, the training data for training the SDXL model contains a large amount of information about celebrities, so the SDXL model itself has the ability to generate realistic images of celebrities.

[0092] On this basis, the pre-trained SDXL model is fine-tuned based on multiple stylized images obtained from the above-mentioned literary image model to obtain a Lora style model capable of generating stylized images of specific people (such as celebrities).

[0093] Specifically, the SDXL model contains a UNet network for denoising, and during fine-tuning training, additional trainable parameters are introduced in the linear layer of the Cross-Attention in the UNet network. Specifically, additional trainable parameters are introduced in the linear projection part of the QKV of the Cross-Attention layer and in the linear part of the subsequent Feed Forward Networks (FFN).

[0094] In order to reduce the number of parameters required for fine-tuning, the number of trainable parameters can be reduced by matrix decomposition. Figure 7 , x represents the input of the Cross-Attention layer, which includes the matrix Q from the image and the matrix K and matrix V from the text information, and h represents the output of the Cross-Attention layer. Represents the original parameters of the Cross-Attention layer after pre-training, represents the additional trainable parameters (i.e., inserted layers), and are matrices of the same size.

[0095] In order to minimize the number of newly added trainable parameters and improve efficiency, decomposed into the product of two matrices, B and A. For example, if yes The matrix B can be The matrix A is Theoretically, the smaller the rank r of the multiplication between BA matrices, the smaller the number of parameters in the inserted layer.

[0096] At this time, after the additional trainable parameters are introduced in the Cross-Attention layer, the encoding result of the image is calculated under the condition of text information through the Attention operation in the Cross-Attention layer. The specific calculation formula is shown in the following formula (2):

[0097] During fine-tuning training, see Figure 8 , freeze the pre-trained model weight parameters in the SDXL model (i.e. ), the newly added trainable parameters (i.e., inserted layers) in the UNet network are trained using the multiple stylized images obtained above to obtain the Lora style model. Since Cross-Attention in the UNet network plays an important role in the SDXL model, even if the parameters of the entire model are not fine-tuned, only fine-tuning the newly added parameter matrix is ​​sufficient to achieve good performance.

[0098] Since the fine-tuning training only adjusts the parameter information of the insertion layer, the weight parameters of the pre-trained model in the SDXL model have not changed. Therefore, the Lora style model obtained through training is essentially a pre-trained SDXL model with a fine-tuned insertion layer (i.e., a Lora module) attached.

[0099] In this way, after the fine-tuning training is completed, the text template and the noise random number are used as the input of the Lora style model to generate stylized images of specific people (such as celebrities). For example, the input text template includes 2 placeholders, namely placeholder Token1 and placeholder Token2. Among them, placeholder Token1 is a required option and is filled with the name of the celebrity; placeholder Token2 is an optional option, and 8 different styles can be selected. When no style is specified, a realistic image of the corresponding celebrity is generated. Under this text template, the noise random number input for model initialization is changed to generate stylized images of different styles of specified celebrities.

[0100] In addition, the text template and the noise random number are used as the input of the pre-trained SDXL model (i.e. the Lora style model without the Lora module) to obtain a realistic image of a specific person (e.g., a celebrity). Finally, the Lora style model and the pre-trained SDXL model are combined to obtain a stylized image and a realistic image of the same specific person.

[0101] In the embodiment of the present application, when the SDXL model itself has the ability to generate realistic images of specific characters, the SDXL model is fine-tuned and trained to obtain a Lora style model that can generate stylized images of specific characters, thereby avoiding the fusion of multiple models and solving the problem of conflict between character features and style fusion. Secondly, when the Lora style model is obtained by fine-tuning training, only a small amount of training data (more than a dozen images) is required to stably generate a fixed style, which significantly reduces the cost of image generation and improves style consistency compared to the direct use of the basic model of the Vincent map.

[0102] In some embodiments, in order to further improve the consistency between the generated stylized image and the character features of the corresponding specific person (eg, a celebrity), the present application screens the generated stylized image and the realistic image of the specific person.

[0103] Specifically, the stylized image is encoded as a multi-dimensional (e.g., 512-dimensional) first feature vector, and the realistic image is encoded as a multi-dimensional (e.g., 512-dimensional) second feature vector. Then, the cosine similarity between the first feature vector and the second feature vector is calculated. The cosine similarity is the facial similarity between the face in the stylized image and the face in the realistic image. The cosine value range is [-1, 1]. After linear transformation, the cosine value range is [0, 1]. The larger the cosine similarity, the greater the probability that the stylized image and the realistic image correspond to the same person. The specific calculation process is shown in formula (3):

[0104] in, Represents the face similarity between the stylized image X and the realistic image Y.

[0105] After obtaining the face similarity of each group of stylized images and realistic images using the above formula (3), the stylized images and realistic images with face similarity greater than a preset threshold (such as 0.8) are retained as training samples.

[0106] Step 502: Use the sample set to perform multiple rounds of iterative training on the de-stylization model to be trained until the training termination condition is met, thereby obtaining a trained de-stylization model.

[0107] Specifically, by continuously optimizing the model parameters, the model can meet the accuracy requirements. When this round of iteration is the first iteration, the model used in this round is the initial model; when this round of iteration is not the first iteration, the model used in this round is the model after the previous round of parameter adjustment. In each round of iterative training, some or all of the training samples can be extracted from the sample set and input into the model used in this round. For example, a random selection method can be used, or the sample set can be divided into batches in advance, and a batch of training samples is input each time. Each training is based on these input training samples, that is, the model used in this round can be used to perform model forward inference on each training sample.

[0108] During the training of the de-stylization model, since each round of iterative training process is similar, the following takes one iterative process as an example. In each round of iterative training process, the following steps are performed: Step 5021: transform each training sample input in this round of iteration through the de-stylization model used in this round, and obtain a predicted realistic image of each training sample.

[0109] In some embodiments, for each training sample, the following operations are performed: A sample style image in a training sample is denoised to obtain a first noisy image; and a sample realistic image in the training sample is denoised to obtain a second noisy image. Character features of the sample person are extracted from the first noisy image; and image features of the realistic type image are extracted from the second noisy image. Based on the character features of the sample person and the image features of the realistic type image, a predicted realistic image of the sample person is obtained.

[0110] Specifically, the sample style image is mapped from the pixel space to the latent space through the first encoding module, and then the sample style image is denoised in the latent space through the first denoising module to obtain a first noisy image.

[0111] In practical applications, the first noise adding module iteratively adds noise to the sample style image in multiple noise adding time steps to obtain a first noisy image. The intensities of the noises corresponding to the multiple noise adding time steps may be the same or different (for example, the intensities of the noises decrease successively in multiple noise adding time steps). The noise may be Gaussian noise that satisfies the standard normal distribution, wherein each round of iterative noise adding includes the following steps: The noise of a noise adding time step corresponding to the current round of iterative noise adding is obtained. The noise of the noise adding time step is used to add noise to the sample style image after the previous round of iterative noise adding package, and the sample style image after the current round of iterative noise adding package is obtained.

[0112] It should be noted that the first round of noise addition is performed on the input sample style image, and the sample style image after the last round of noise addition is the first noisy image, which is a pure noise image.

[0113] In addition, the sample realistic image is mapped from the pixel space to the latent space through the second encoding module, and then the sample realistic image is denoised in the latent space through the second denoising module to obtain a second noisy image.

[0114] In practical applications, the second noise adding module iteratively adds noise to the sample realistic image in multiple noise adding time steps to obtain a second noisy image. The intensities of the noises corresponding to the multiple noise adding time steps may be the same or different (for example, the intensities of the noises decrease successively in multiple noise adding time steps). The noise may be Gaussian noise that satisfies the standard normal distribution, wherein each round of iterative noise adding includes the following steps: The noise of a noise adding time step corresponding to the current round of iterative noise adding is obtained. The noise of the noise adding time step is used to add noise to the sample realistic image after the previous round of iterative noise adding package, and the sample realistic image after the current round of iterative noise adding package is obtained.

[0115] It should be noted that the first round of noise addition is to add noise to the input sample realistic image, and the sample realistic image after the last round of noise addition is the second noisy image, which is a pure noise image. The number of noise addition time steps corresponding to the first noise addition module and the second noise addition module is the same. The noise intensity of each noise addition time step used by the first noise addition module and the second noise addition module can be the same or different.

[0116] The reference network extracts the character features of the sample person from the first noisy image and passes the character features of the sample person to the denoising network. The denoising network extracts the image features of the realistic type image from the second noisy image, and denoises the second noisy image based on the character features of the sample person and the image features of the realistic type image to obtain a sample denoising result. The sample denoising result is mapped from the latent space to the pixel space through the decoding module to obtain a predicted realistic image of the sample person.

[0117] In an embodiment of the present application, a sample style image and a sample realistic image of the same sample person are used as training samples, and the de-stylization model is trained for multiple rounds of iterations, so that the de-stylization model learns to generate a realistic image of the corresponding person based on the person features in the sample style image, thereby obtaining the ability to generate a realistic image of the same person by zero-shot, thereby improving the scalability and generalization ability of the model.

[0118] In some embodiments, the sample character features and the image features of the realistic image are spliced ​​to obtain sample splicing features. Based on the sample splicing features, the predicted noise information in the second noisy image is obtained. The predicted noise information is removed from the second noisy image to obtain a predicted realistic image of the sample character.

[0119] Specifically, the denoising network is used to iteratively denoise the second noisy image in multiple denoising time steps to obtain a sample denoising result, and the noise intensity corresponding to each of the multiple denoising time steps may be the same or different (for example, the noise intensity increases in sequence in multiple denoising time steps). The noise may be Gaussian noise that satisfies a standard normal distribution, wherein each round of iterative denoising includes the following steps: The noise of a denoising time step corresponding to the current round of iterative denoising is obtained, and the noise of the denoising time step is removed from the second denoised image after the previous round of iterative denoising to obtain the second denoised image after the current round of iterative denoising.

[0120] In practical applications, in order to generate realistic images that are consistent with the character features of the stylized image, in addition to using the denoising network for denoising, a reference network (ReferenceNet) is also introduced. The reference network has the same network structure as the denoising network.

[0121] In each denoising time step, the character features of the sample person are extracted from the first denoised image through the reference network; the sub-image features of the realistic type image are extracted from the second denoised image after the previous round of iterative denoising through the denoising network. The character features of the sample person generated by the reference network are then passed to the denoising network and spliced ​​with the sub-image features of the realistic type image generated at the same position of the denoising network to obtain the sub-spliced ​​features. The sub-spliced ​​features are used to estimate the noise of the denoising time step.

[0122] In the specific implementation, when the denoising network is a Unet network, the reference network is also a Unet network, that is, the denoising network and the reference network both contain multiple Attention layers. Considering that the Attention layer in the form of Cross-Attention involves not only picture features but also text features, and the role of the reference network is to encode character features and help the denoising network restore the realistic image of the character, which only involves picture features, the Attention layers in the denoising network and the reference network of this application are both in the form of Self-Attention.

[0123] The input features of the Attention layer of the reference network coding (including the character features of the sample character) are transferred to the Attention layer at the same position in the denoising network in the form of vector splicing, and spliced ​​with the input features of the Attention layer at the same position in the denoising network (including the sub-image features of the realistic type image) to obtain the splicing result. Then, the noise of the denoising time step is estimated based on the splicing result.

[0124] For example, see Fig. 9 , the denoising network includes Attention layer 1, Attention layer 2 and Attention layer 3. The denoising network has the same network structure as the reference network, so the reference network includes Attention layer 1, Attention layer 2 and Attention layer 3.

[0125] Taking Attention layer 1 as an example, the input features of Attention layer 1 in the reference network include: matrix ,matrix and matrix ; The input features of the Attention layer 1 in the denoising network include: matrix ,matrix and matrix The reference network will matrix Attention layer 1 and matrix passed to the denoising network splicing; the reference network will matrix Attention layer 1 and matrix passed to the denoising network In this case, the calculation formula of the Attention layer 1 of the denoising network is as shown in formula (4):

[0126] Among them, y represents the output of the Attention layer.

[0127] Similarly, the input features of other Attention layers in the reference network can also be passed to the Attention layer at the same position in the denoising network and concatenated with the input of the Attention layer at the same position, which will not be repeated here.

[0128] In an embodiment of the present application, the character features of the sample person generated by the reference network are passed to the denoising network and spliced ​​with the sub-image features of the realistic type image generated at the same position of the denoising network to obtain sub-spliced ​​features, and then the sub-spliced ​​features are used to estimate the noise of the denoising time step. In this way, the character features of the sample person can play a full role in the denoising process, so that the de-stylization model learns to generate a realistic image of the sample person through gradual denoising, thereby improving the training effect of the de-stylization model.

[0129] Step 5022: Determine the model loss value of the de-stylization model used in this round based on the obtained predicted realistic images and the corresponding sample realistic images.

[0130] Specifically, when calculating the model loss value, the model loss value of the de-stylization model is obtained based on the difference between each predicted realistic image and the corresponding sample realistic image.

[0131] In an embodiment of the present application, the model loss value can adopt a loss function such as a cross entropy loss function (Cross Entropy Loss, celoss), a mean squared error (MSE) loss function, a squared absolute error loss function, a maximum likelihood loss (Likelihood Loss, LHL) function, etc. to obtain the model loss value of the de-stylization model. Of course, it can also be other possible loss functions, and the embodiment of the present application does not limit this.

[0132] Step 5023, determine whether the de-stylization model used in this round meets the iteration termination condition, if so, end, otherwise execute step 5024.

[0133] In the embodiment of the present application, the iteration termination condition may include at least one of the following conditions: (1) The number of iterations reaches the set threshold.

[0134] (2) The model loss value is less than the set loss threshold, or the model loss value reaches the minimum value.

[0135] Step 5024, adjust the parameters of the de-stylization model used in this round according to the model loss value, and use the de-stylization model after the adjustment to enter the next round of iterative training.

[0136] In the embodiment of the present application, when the number of iterations does not exceed the preset number threshold, and the model loss value is not less than the set loss threshold, the determination process of step 5023 is no, that is, it is considered that the current model does not meet the iteration termination condition, and the model parameters need to be adjusted to continue training. After the parameter adjustment, the next round of iterative training process is entered, that is, jumping to step 5021.

[0137] In one possible implementation, when the model still does not meet the convergence conditions, the model weight parameters can be updated by optimization algorithms such as gradient descent method and stochastic gradient descent algorithm to minimize the above loss function, and continue training with the updated model weight parameters.

[0138] When the number of iterations has exceeded the preset threshold, or the model loss value is less than the set loss threshold or reaches the minimum value, the determination process of step 5023 is yes, that is, it is considered that the current model has met the convergence conditions, the model training is completed, and the trained de-stylized model is obtained.

[0139] In an embodiment of the present application, a sample style image and a sample realistic image of the same sample person are used as training samples, and the de-stylization model is trained for multiple rounds of iterations, so that the de-stylization model fully understands the person features in the sample style image and the image features of the realistic type image in the sample realistic image, so as to obtain the ability to zero-shot generate realistic images that are consistent with the person features of the stylized image. Therefore, when the de-stylization model is used to generate realistic images that are consistent with the person features of the stylized image in the application stage, not only the scalability and generalization capabilities are improved, but also the accuracy of the obtained realistic images and stylized images of the same person is improved.

[0140] After introducing the training process of the de-stylization model, the following is based on the trained de-stylization model and Figure 1 The system architecture diagram shown in FIG. 1 introduces the process of an image generation method provided by an embodiment of the present application, such as Fig.10 As shown, the process of the method is executed by a computer device, which may be Figure 1 The terminal device 101 and / or server 102 shown includes the following steps: Step 1001, obtaining a pure noise image and a stylized image of a target person.

[0141] Specifically, the pure noise image satisfies the standard normal distribution, and the size of the pure noise image is the same as the stylized image of the target person. It should be noted that in the application stage of the de-stylization model, it is no longer necessary to input the realistic image of the person into the de-stylization model, and accordingly, it is no longer necessary for the second encoding module and the second denoising module in the de-stylization model to denoise the realistic image of the person to obtain the pure noise image; instead, the pure noise image that satisfies the standard normal distribution is directly collected as the input of the de-stylization model.

[0142] The style corresponding to the stylized image of the target person can be the style included in the training sample when training the de-stylization model, or a new style not included in the training sample. The person corresponding to the stylized image of the target person can be the person included in the training sample when training the de-stylization model, or a new person not included in the training sample.

[0143] Step 1002: extracting the target person's features from the stylized image of the target person.

[0144] Specifically, the character features of the target person are related to the characteristics of the target person itself, and have nothing to do with the style corresponding to the stylized image. The character features of the target person may be the characteristics of the person himself, such as the gender, age, eye size, height of the nose bridge, etc.

[0145] In some embodiments, the de-stylization model performs the following operations to extract the character features of the target character: The stylized image of the target person is denoised according to a preset denoising mode to obtain a target noisy image, wherein the preset denoising mode is obtained by training a de-stylization model. The person features of the target person are extracted from the target noisy image.

[0146] In practical applications, the stylized image of the target person is input into the first encoding module in the de-stylization model, and the stylized image of the target person is mapped from the pixel space to the latent space through the first encoding module. The stylized image of the target person is denoised in the latent space through the first denoising module to obtain a target noisy image, wherein the target noisy image is a pure noise image that satisfies the standard normal distribution.

[0147] In the embodiment of the present application, the character features of the target person are extracted from the stylized image to guide the generation of a target realistic image of the target person. In this process, the style features of the stylized image are not of concern. Therefore, corresponding realistic images can be generated for any style or any person, thereby achieving zero-shot generation of realistic images consistent with the character features of the stylized image, significantly reducing training costs and improving scalability.

[0148] In practical applications, the preset noise adding mode can be used to perform noise adding once on the stylized image of the target person to directly obtain the target noisy image; or the preset noise adding mode can be used to perform multiple rounds of iterative noise adding on the stylized image of the target person to obtain the target noisy image.

[0149] In some embodiments, when a preset noise adding mode is used to perform multiple rounds of iterative noise adding on the stylized image of the target person, the preset noise adding mode includes: multiple noise adding time steps and first noises corresponding to each of the multiple noise adding time steps. The first noise may be Gaussian noise satisfying a standard normal distribution.

[0150] The intensity of the first noise corresponding to each of the multiple noise adding time steps may be the same or different. For example, the intensity of the first noise corresponding to each of the multiple noise adding time steps decreases in sequence within the multiple noise adding time steps. For another example, the intensity of the first noise corresponding to each of the multiple noise adding time steps increases in sequence within the multiple noise adding time steps. Of course, the intensity of the first noise corresponding to each of the multiple noise adding time steps may also be in other forms, which are not specifically limited here.

[0151] In multiple noisy time steps, the stylized image is iteratively denoised to obtain a target noisy image; wherein each round of iterative denoising includes the following steps: The first noise of a noise adding time step corresponding to the current round of iterative noise adding is obtained. The stylized image after the previous round of iterative noise adding is denoised using the first noise of the noise adding time step to obtain the stylized image after the current round of iterative noise adding.

[0152] Specifically, in the first round of noisy addition (corresponding to the first noisy addition time step), the information of the first noisy addition time step and the stylized image of the target person are input into the first noisy addition module, the first noise of the first noisy addition time step is predicted, and the first noise of the first noisy addition time step is used to noisy the input stylized image of the target person to obtain the stylized image after the first round of noisy addition (i.e., the noisy image in the intermediate state).

[0153] For any other round of noisy, the information of the noisy time step corresponding to the current round of noisy and the stylized image after the previous round of noisy are input into the first noisy module, the first noise of the noisy time step corresponding to the current round of noisy is predicted, and the stylized image after the previous round of noisy is denoised using the obtained first noise to obtain the stylized image after the current round of noisy (i.e., the noisy image in the intermediate state).

[0154] The stylized image after the last round of noise addition (corresponding to the last noise addition time step) is the target noisy image obtained.

[0155] In an embodiment of the present application, multiple rounds of iterative noise addition are performed on the stylized image of the target person within multiple noise addition time steps, and the obtained stylized image of the target person is gradually converted into a target noisy image with pure noise, so that the realistic image of the target person can be gradually restored from the pure noise image in reverse, thereby improving the quality of the restored realistic image of the target person.

[0156] Step 1003, extracting the image features of the realistic type image and the character features of the target person from the pure noise image, and the target noise information that does not meet the correlation condition. The image features of the realistic type image are learned from multiple sample realistic images.

[0157] Step 1004 , removing target noise information from the pure noise image to obtain a target realistic image of the target person.

[0158] Specifically, failure to satisfy the correlation condition may be irrelevant to the image features of the realistic image and the character features of the target person, or may be that the correlation between the image features of the realistic image and the character features of the target person is less than a preset threshold.

[0159] The target person's features are extracted from the target noisy image through the reference network, and the target person's features are passed to the denoising network. The image features of the realistic type image are extracted from the pure noise image through the denoising network, and then the target noise information in the pure noise image that is unrelated to the image features of the realistic type image and the target person's features is predicted in combination with the target person's features passed by the reference network, and the target noise information is removed from the pure noise image to obtain the target denoising result. The target denoising result is mapped from the latent space to the pixel space through the decoding module to obtain the target realistic image of the target person.

[0160] In an embodiment of the present application, the character features of the target person in the stylized image are used as reference information, combined with the image features of the realistic type images learned from multiple sample realistic images, to generate a target realistic image of the target person. This ensures that the obtained stylized image and the target realistic image correspond to the same person, thereby improving the accuracy of generating realistic images and stylized images of the same person.

[0161] Secondly, the character features of the target person are extracted from the stylized image to guide the generation of the target realistic image of the target person. It is not necessary to extract the style features of the stylized image. That is, the process of generating a realistic image of a person by the de-stylized model is independent of the style of the input stylized image. Therefore, corresponding realistic images can be generated for any style or any person, realizing zero-shot generation of realistic images and stylized images of the same person, improving the scalability and generalization ability of the de-stylized model, and avoiding resource consumption caused by repeated training.

[0162] In addition, the technical solution of the present application ensures that the obtained stylized image and the realistic image correspond to the same person, so when the obtained stylized image and the realistic image are used as training samples for subsequent model training, the accuracy of the training samples is correspondingly improved. Since the de-stylization model of the present application has zero-shot capability, a large number of diverse training samples can be generated through the de-stylization model for subsequent model training without repeatedly training the de-stylization model, thereby improving the performance of model training.

[0163] In some embodiments, a denoising network may be used to denoise the input pure noise image once to directly obtain a target realistic image of the target person. Specifically, the following operations are performed through the de-stylization model: The image features of the realistic image and the character features of the target character are spliced ​​to obtain the target splicing features. Then, the target splicing features are used to predict the probability that each pixel in the pure noise image contains noise information. Based on the probability that each pixel in the pure noise image contains noise information, the noise distribution in the pure noise image is obtained. Based on the noise distribution in the pure noise image, the target noise information in the pure noise image is extracted.

[0164] Specifically, the reference network passes the generated character features of the target person to the denoising network, and splices them with the image features of the realistic type image generated at the same position of the denoising network to obtain the target splicing features. When the target splicing features are used to predict the target noise information in the pure noise image, the obtained target noise information not only has a low correlation with the character features of the target person, but also has a low correlation with the image features of the realistic type image. Then, after removing the target noise information from the pure noise image, useful information with a high correlation with the character features of the target person and the image features of the realistic type image is retained. Therefore, restoring the target realistic image of the target person based on these useful information can effectively improve the quality of the restored realistic image.

[0165] In some embodiments, a denoising network may be used to perform multiple rounds of iterative denoising on the pure noise image to obtain a target realistic image of the target person. In this case, the target noise information includes: second noise corresponding to each of the multiple denoising time steps. In the denoising process, the following operations are performed by the de-stylization model: In multiple denoising time steps, the pure noise image is iteratively denoised to obtain a target realistic image of the target person, wherein each round of iterative denoising includes the following steps: The second noise of a denoising time step corresponding to the current iteration is obtained; the second noise of the denoising time step is removed from the pure noise image after the previous iteration denoising to obtain the pure noise image after the current iteration denoising.

[0166] Specifically, the second noises corresponding to each of the multiple denoising time steps are all obtained through denoising network prediction. The intensities of the second noises corresponding to each of the multiple denoising time steps may be the same or different. For example, the intensities of the second noises corresponding to each of the multiple denoising time steps decrease successively within the multiple denoising time steps. For another example, the intensities of the second noises corresponding to each of the multiple denoising time steps increase successively within the multiple denoising time steps. Of course, the intensities of the second noises corresponding to each of the multiple denoising time steps may also be in other forms, which are not specifically limited here.

[0167] In some cases, the number of denoising time steps used by the first denoising module is the same as the number of denoising time steps used by the denoising network, and the intensity change trend of the first noise corresponding to each of the multiple denoising time steps is symmetrical with the intensity change trend of the second noise corresponding to each of the multiple denoising time steps.

[0168] For example, the first denoising module includes 4 denoising time steps, and the intensity of the first noise corresponding to each of the 4 denoising time steps decreases successively within the 4 denoising time steps; the denoising network includes 4 denoising time steps, and the intensity of the second noise corresponding to each of the 4 denoising time steps increases successively within the 4 denoising time steps.

[0169] In an embodiment of the present application, multiple rounds of iterative denoising are performed on the pure noise image within multiple denoising time steps, and in each round of denoising, noise information with low correlation with the character features of the target person and the image features of the realistic type image is removed, and the input pure noise image is gradually converted into a target realistic image of the target person, thereby improving the quality of the obtained target realistic image of the target person.

[0170] In some embodiments, the image features of the realistic image include: sub-image features corresponding to each of a plurality of denoising time steps. The sub-image features corresponding to each denoising time step are extracted from the pure noise image after the previous round of iterative denoising.

[0171] In each round of denoising, the denoising network predicts the second noise required for this round of iterative denoising based on the character features of the target character transmitted by the reference network and the sub-image features of the corresponding denoising time step. The following is a specific description of the second noise acquisition process corresponding to a denoising time step in this round of iteration: The character features of the target person are spliced ​​with the sub-image features corresponding to the denoising time step to obtain the sub-splicing features corresponding to the denoising time step. Based on the sub-splicing features corresponding to the denoising time step, the probability of each pixel in the pure noise image after the previous round of iterative denoising containing noise information is predicted.

[0172] Based on the probability that each pixel in the pure noise image after the previous round of iterative denoising contains noise information, the noise distribution in the pure noise image after the previous round of iterative denoising is obtained. Based on the noise distribution in the pure noise image after the previous round of iterative denoising, the second noise corresponding to the denoising time step is obtained.

[0173] Specifically, in the first round of denoising (corresponding to the first denoising time step), the information of the first denoising time step and the original pure noise image are input into the denoising network to obtain the sub-image features of the realistic type image corresponding to the first denoising time step. The reference network passes the generated character features of the target person to the denoising network and splices them with the sub-image features of the realistic type image generated by the denoising network to obtain the sub-splicing features. The denoising network uses the sub-splicing features to predict the second noise of the first denoising time step. Furthermore, the denoising network uses the obtained second noise to denoise the original pure noise image to obtain the pure noise image after the first round of denoising (i.e., the denoised image in the intermediate state).

[0174] For any other round of denoising, the information of the current denoising time step and the pure noise image after the previous denoising are input into the denoising network to obtain the sub-image features of the realistic type image corresponding to the current denoising time step. The reference network passes the generated character features of the target person to the denoising network and splices them with the sub-image features of the realistic type image generated by the denoising network to obtain the sub-splicing features. The denoising network uses the sub-splicing features to predict the second noise of the current denoising time step. Furthermore, the denoising network uses the obtained second noise to denoise the pure noise image after the previous denoising to obtain the pure noise image after the current denoising (i.e., the denoised image in the intermediate state).

[0175] The pure noise image after the last round of denoising (corresponding to the last denoising time step) is the target denoising result. The target denoising result is mapped from the latent space to the pixel space through the decoding module to obtain the target realistic image of the target person.

[0176] In the embodiment of the present application, in each round of denoising, sub-image features of the realistic type image used in the current round of iterative denoising are extracted from the pure noise image after the previous round of iterative denoising. The character features of the target person are spliced ​​with the sub-image features of the realistic type image used in the current round of iterative denoising to obtain the sub-splicing features used in the current round of iterative denoising, and then based on the obtained sub-splicing features, the second noise used in the current round of iterative denoising is predicted, so that the second noise is more matched with the noise image to be denoised in the current round, thereby improving the effect of multiple rounds of iterative denoising, and then improving the quality of the generated target realistic image.

[0177] In some embodiments, a de-stylization model of an embodiment of the present application may be used to obtain a large amount of stylized data pairs (i.e., realistic images and stylized images of the same person). The obtained large amount of stylized data pairs may then be used to train a stylized model.

[0178] Specifically, in order to obtain multi-style image data, the LAION-5B dataset is selected as pre-training data to pre-train the stylized model, wherein the LAION-5B dataset is a large-scale cultural image dataset, and the LAION-5B dataset contains multi-domain data. In order to improve the quality of the pre-training data, this application screens the obtained LAION-5B dataset, and the specific screening methods include: aesthetic quality scoring and face ratio screening.

[0179] Specifically, the aesthetic quality score is an image quality assessment indicator used in the Laion dataset, ranging from [0, 10]. The higher the score, the better the quality of the image. This application uses the aesthetic quality score to screen the LAION-5B dataset and retains images with a score greater than 6.5.

[0180] Face ratio screening refers to screening images according to image size and face ratio. Specifically, the shortest side of the image is greater than 768. Since the smaller the face ratio, the greater the possibility of face distortion and blur. Therefore, this application proposes to only retain images with a face ratio of more than 1 / 16. For example, the size of a square face that accounts for 1 / 16 is , this size basically guarantees the generation of high-quality faces.

[0181] After screening the massive LAION-5B dataset using aesthetic quality scores and face ratio screening, the stylized model is pre-trained using the retained dataset. After the pre-training is completed, the pre-trained stylized model is fine-tuned using the massive stylized data pairs obtained in this application (i.e., realistic images and stylized images of the same person) to obtain the final stylized model with zero-shot capability.

[0182] In the embodiment of the present application, a de-stylization model is used to obtain a large amount of stylized data pairs (i.e., realistic images and stylized images of the same person) as training samples to train the stylized model, which greatly reduces the difficulty of obtaining training samples.

[0183] In order to better explain the embodiment of the present application, an image generation method provided by the embodiment of the present application is introduced below in combination with a specific implementation scenario. The process of the method can be as follows: Figure 1The terminal device 101 shown is executed, and may also be executed by the server 102, or may be executed interactively between the terminal device 101 and the server 102. The process of the method mainly includes a training phase and an application phase.

[0184] First, we introduce the training phase, see Fig.11A , it is assumed that the first encoding module and the second encoding module in the de-stylization model are both variational auto-encoders (VAE), the reference network and the denoising network are both UNet networks, and the decoding module is the decoder corresponding to VAE.

[0185] A sample set containing multiple training samples is obtained, each training sample includes: a realistic image and a stylized image of the same person. The sample set is used to perform multiple rounds of iterative training on the de-stylized model to be trained until the training termination condition is met, and a trained de-stylized model is obtained. The following is a specific introduction using a training sample in an iterative process as an example, including the following steps: A training sample is assumed to include: a sample realistic image of person C and a sample style image of person C, and a training sample is input into the de-stylization model to be trained.

[0186] In the de-stylization model to be trained, the first encoding module maps the sample style image of person C from the pixel space to the latent space. In the latent space, the first noise adding module gradually adds Gaussian noise to the sample style image of person C through multiple noise adding time steps to obtain a pure noise image 1.

[0187] The second encoding module maps the sample realistic image of person C from the pixel space to the latent space. The second noise adding module gradually adds Gaussian noise to the sample realistic image of person C in the latent space through multiple noise adding time steps to obtain a pure noise image 2.

[0188] At the first denoising time step, the reference network extracts the character features of person C from the pure noise image 1 and passes the character features of person C to the denoising network. The denoising network extracts the image features of the realistic type image based on the first denoising time step and the pure noise image 2, and splices them with the received character features of person C to obtain the first splicing feature. The denoising network uses the first splicing feature to predict the Gaussian noise of the first denoising time step, and removes the Gaussian noise of the first denoising time step from the pure noise image 2 to obtain the intermediate state image of the first time step.

[0189] At the second denoising time step, the reference network extracts the character features of character C from the pure noise image 1, and passes the character features of character C to the denoising network. The denoising network extracts the image features of the realistic type image based on the second denoising time step and the intermediate state image of the first time step, and splices them with the received character features of character C to obtain the second splicing feature. The denoising network uses the second splicing feature to predict the Gaussian noise of the second denoising time step, and removes the Gaussian noise of the second denoising time step from the intermediate state image of the first time step to obtain the intermediate state image of the second time step.

[0190] And so on, until the intermediate state image of the last denoising time step is obtained.

[0191] The decoding module maps the intermediate state image of the last denoising time step from the latent space to the pixel space to obtain the predicted realistic image of character C.

[0192] The model loss value of the de-stylization model is obtained by using the difference between each predicted realistic image and the corresponding sample realistic image (including the difference between the sample realistic image of character C and the predicted realistic image of character C), and the parameters of the de-stylization model used in this round are adjusted according to the model loss value.

[0193] After introducing the training phase, the application phase is introduced next. Fig. 11B , the collected stylized image of person D and pure noise image 3 are input into the trained de-stylization model.

[0194] In the trained de-stylization model, the first encoding module maps the stylized image of the character D from the pixel space to the latent space. In the latent space, the first noise adding module gradually adds Gaussian noise to the stylized image of the character D through multiple noise adding time steps to obtain a noisy image.

[0195] At the first denoising time step, the reference network extracts the character features of person D from the noisy image and passes the character features of person D to the denoising network. The denoising network extracts the image features of the realistic type image based on the first denoising time step and the input pure noise image 3, and splices them with the received character features of person D to obtain the first splicing feature. The denoising network uses the first splicing feature to predict the Gaussian noise of the first denoising time step, and removes the Gaussian noise of the first denoising time step from the pure noise image 3 to obtain the intermediate state image of the first time step.

[0196] At the second denoising time step, the reference network extracts the character features of person D from the noisy image and passes the character features of person D to the denoising network. The denoising network extracts the image features of the realistic type image based on the second denoising time step and the intermediate state image of the first time step, and splices them with the received character features of person D to obtain a second spliced ​​feature. The denoising network uses the second spliced ​​feature to predict the Gaussian noise of the second denoising time step, and removes the Gaussian noise of the second denoising time step from the intermediate state image of the first time step to obtain the intermediate state image of the second time step.

[0197] And so on, until the intermediate state image of the last denoising time step is obtained.

[0198] The decoding module maps the intermediate state image of the last denoising time step from the latent space to the pixel space to obtain a realistic image of person D.

[0199] In the embodiment of the present application, the sample style image and the sample realistic image of the same sample person are used as training samples, and the de-stylization model is trained for multiple rounds of iterations, so that the de-stylization model fully understands the character features in the sample style image and the image features of the realistic type image in the sample realistic image, so as to obtain the ability to generate realistic images consistent with the character features of the stylized image by zero-shot. Therefore, when the de-stylization model is used to generate realistic images consistent with the character features of the stylized image in the application stage, not only the scalability and generalization ability are improved, but also the accuracy of the obtained realistic images and stylized images of the same person is improved. Secondly, the method can construct a large number of stylized data pairs for the training of the stylized model, which greatly reduces the difficulty of obtaining training samples.

[0200] Based on the same technical concept, the present application embodiment provides a structural schematic diagram of an image generating device, such as Fig. 11C As shown, the image generating device 1100 includes: An acquisition module 1101 is used to acquire a pure noise image and a stylized image of a target person; An extraction module 1102 is used to extract the character features of the target person from the stylized image of the target person; A prediction module 1103 is used to extract target noise information that does not meet the correlation condition and has image features of a realistic image and character features of the target character from the pure noise image, wherein the image features of the realistic image are obtained by learning from a plurality of sample realistic images; The denoising module 1104 is used to remove the target noise information from the pure noise image to obtain a target realistic image of the target person.

[0201] Optionally, the extraction module 1102 is specifically used for: The trained de-stylization model performs the following operations: Noising the stylized image of the target person according to a preset noise adding mode to obtain a target noisy image, wherein the preset noise adding mode is obtained by training the de-stylization model; Extracting the character features of the target person from the target noisy image.

[0202] Optionally, the preset noise adding mode includes: a plurality of noise adding time steps and first noises corresponding to the plurality of noise adding time steps; the first noises corresponding to the plurality of noise adding time steps have different intensities; The extraction module 1102 is specifically used for: In the multiple noise adding time steps, the stylized image is iteratively noised to obtain the target noisy image; wherein each round of iterative noise adding comprises the following steps: Obtain the first noise of a noise adding time step corresponding to this round of iterative noise adding; The first noise of the noise adding time step is used to add noise to the stylized image after the previous round of iterative noise adding, so as to obtain the stylized image after the current round of iterative noise adding.

[0203] Optionally, the prediction module 1103 is specifically used for: The following operations are performed using the trained de-stylization model: Splicing the image features of the realistic image and the character features of the target character to obtain a target splicing feature; Using the target splicing feature, predicting the probability that each pixel in the pure noise image contains noise information; Based on the probability that each pixel in the pure noise image contains noise information, obtaining the noise distribution in the pure noise image; Based on the noise distribution in the pure noise image, the target noise information in the pure noise image is extracted.

[0204] Optionally, the target noise information includes: second noise corresponding to each of the multiple denoising time steps; the second noise corresponding to each of the multiple denoising time steps has different intensities; The denoising module 1104 is specifically used for: The following operations are performed using the trained de-stylization model: In the multiple denoising time steps, the pure noise image is iteratively denoised to obtain a target realistic image of the target person; wherein each round of iterative denoising includes the following steps: Get the second noise of a denoising time step corresponding to this round of iterative denoising; The second noise of the one denoising time step is removed from the pure noise image after the previous round of iterative denoising to obtain the pure noise image after the current round of iterative denoising.

[0205] Optionally, the image features of the realistic type image include: sub-image features corresponding to each of the multiple denoising time steps; The denoising module 1104 is specifically used for: Splicing the character features of the target person with the sub-image features corresponding to the one denoising time step to obtain the sub-splicing features corresponding to the one denoising time step; Based on the sub-joint features corresponding to the one denoising time step, predicting the probability that each pixel in the pure noise image after the previous round of iterative denoising contains noise information; Obtaining noise distribution in the pure noise image after the previous round of iterative denoising based on the probability that each pixel in the pure noise image after the previous round of iterative denoising contains noise information; Based on the noise distribution in the pure noise image after the previous round of iterative denoising, the second noise corresponding to the one denoising time step is obtained.

[0206] Optionally, a module training module 1105 is also included; The module training module 1105 is specifically used for: Obtain a sample set including multiple training samples, each training sample including: a sample style image and a sample realistic image of the same sample person; The sample set is used to perform multiple rounds of iterative training on the de-stylization model to be trained until a training termination condition is met, thereby obtaining the trained de-stylization model, wherein each round of iterative training process includes the following steps: Using the de-stylization model used in this round, each training sample input in this round of iteration is converted to obtain a predicted realistic image of each training sample; According to the obtained predicted realistic images and the corresponding sample realistic images, the model loss value of the de-stylization model used in this round is determined, and after adjusting the parameters of the de-stylization model used in this round according to the model loss value, the de-stylization model with the adjusted parameters is used to enter the next round of iterative training.

[0207] Optionally, the module training module 1105 is specifically used for: For each training sample, the following operations are performed respectively: Noising a sample style image in a training sample to obtain a first noisy image; and Noising a sample realistic image in the training sample to obtain a second noisy image; Extracting character features of a sample character from the first noisy image; and extracting image features of a realistic image from the second noisy image; Based on the character features of the sample character and the image features of the realistic type image, a predicted realistic image of the sample character is obtained.

[0208] Optionally, the module training module 1105 is specifically used for: Based on the character features of the sample character and the image features of the realistic image, stitching is performed to obtain a sample stitching feature; Based on the sample splicing features, obtaining predicted noise information in the second noisy image; The predicted noise information is removed from the second noisy image to obtain a predicted realistic image of the sample person.

[0209] In an embodiment of the present application, the character features of the target person in the stylized image are used as reference information, combined with the image features of the realistic type images learned from multiple sample realistic images, to generate a target realistic image of the target person. This ensures that the obtained stylized image and the target realistic image correspond to the same person, thereby improving the accuracy of generating realistic images and stylized images of the same person.

[0210] Secondly, the character features of the target person are extracted from the stylized image to guide the generation of the target realistic image of the target person. It is not necessary to extract the style features of the stylized image. That is, the process of generating a realistic image of the person by the de-stylized model is independent of the style of the input stylized image. Therefore, corresponding realistic images can be generated for any style or any person, realizing zero-shot generation of realistic images and stylized images of the same person, improving scalability and generalization capabilities, and avoiding resource consumption caused by repeated training.

[0211] In addition, the technical solution of the present application ensures that the obtained stylized image and the realistic image correspond to the same person, so when the obtained stylized image and the realistic image are used as training samples, the accuracy of the training samples is correspondingly improved. Moreover, the de-stylization model of the present application has zero-shot capability, so a large number of training samples can be generated through the de-stylization model for model training, thereby improving the performance of model training.

[0212] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories), or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0213] Based on the same technical concept, the embodiment of the present application provides a computer device, which can be Figure 1 The terminal device and / or server shown, such as Fig.12 As shown, it includes at least one processor 1201 and a memory 1202 connected to the at least one processor. The specific connection medium between the processor 1201 and the memory 1202 is not limited in the embodiment of the present application. Fig.12 For example, the processor 1201 and the memory 1202 are connected via a bus. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0214] In the embodiment of the present application, the memory 1202 stores instructions that can be executed by at least one processor 1201, and the at least one processor 1201 can perform the steps of the above-mentioned image generation method by executing the instructions stored in the memory 1202.

[0215] As an embodiment, the processor 1201 obtains a pure noise image and a stylized image of a target person; the processor 1201 extracts the character features of the target person from the stylized image of the target person; and extracts the target noise information that does not meet the correlation condition with the image features of the realistic type image and the character features of the target person from the pure noise image, wherein the image features of the realistic type image are learned from multiple sample realistic images and stored in the memory 1202. The processor 1201 removes the target noise information from the pure noise image to obtain a target realistic image of the target person.

[0216] Among them, the processor 1201 is the control center of the computer device, and various interfaces and lines can be used to connect various parts of the computer device, and by running or executing instructions stored in the memory 1202 and calling data stored in the memory 1202, a realistic image of the character is formed. Optionally, the processor 1201 may include one or more processing units, and the processor 1201 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1201. In some embodiments, the processor 1201 and the memory 1202 may be implemented on the same chip, and in some embodiments, they may also be implemented separately on independent chips.

[0217] Processor 1201 may be a general-purpose processor, such as a central processing unit (CPU), a digital signal processor, an application-specific integrated circuit (ASIC), a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0218] The memory 1202 is a non-volatile computer-readable storage medium that can be used to store non-volatile software programs, non-volatile computer executable programs and modules. The memory 1202 may include at least one type of storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory, a random access memory (Random Access Memory, RAM), a static random access memory (Static Random Access Memory, SRAM), a programmable read-only memory (Programmable Read Only Memory, PROM), a read-only memory (Read Only Memory, ROM), an electrically erasable programmable read-only memory (Electrically Erasable Programmable Read-Only Memory, EEPROM), a magnetic memory, a disk, an optical disk, and the like. The memory 1202 is any other medium that can be used to carry or store the desired program code in the form of an instruction or data structure and can be accessed by a computer device, but is not limited thereto. The memory 1202 in the embodiment of the present application can also be a circuit or any other device that can realize a storage function, for storing program instructions and / or data.

[0219] Based on the same inventive concept, an embodiment of the present application provides a computer-readable storage medium storing a computer program executable by a computer device. When the computer program runs on the computer device, the computer device executes the steps of the above-mentioned image generation method.

[0220] Based on the same inventive concept, an embodiment of the present application provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer device, the computer device executes the steps of the above-mentioned image generation method.

[0221] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0222] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer device or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0223] These computer program instructions may also be stored in a computer readable memory capable of directing a computer device or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0224] These computer program instructions can also be loaded onto a computer device or other programmable data processing device so that a series of operation steps are executed on the computer device or other programmable device to produce a process implemented by the computer device, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0225] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0226] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. An image generation method, characterized in that: include: Get a pure noise image and a stylized image of the target person; Extracting the character features of the target person from the stylized image of the target person; Extracting target noise information that does not satisfy a correlation condition and image features of a realistic image and character features of a target character from the pure noise image, wherein the image features of the realistic image are obtained by learning from a plurality of sample realistic images; The target noise information is removed from the pure noise image to obtain a target realistic image of the target person.

2. The method according to claim 1, characterized in that The step of extracting the character features of the target person from the stylized image of the target person includes: The trained de-stylization model performs the following operations: Noising the stylized image of the target person according to a preset noise adding mode to obtain a target noisy image, wherein the preset noise adding mode is obtained by training the de-stylization model; Extracting the character features of the target person from the target noisy image.

3. The method according to claim 2, characterized in that The preset noise adding mode includes: a plurality of noise adding time steps and first noises corresponding to the plurality of noise adding time steps; the first noises corresponding to the plurality of noise adding time steps have different intensities; The step of adding noise to the stylized image of the target person according to a preset noise adding mode to obtain a target noisy image includes: In the multiple noise adding time steps, the stylized image of the target person is iteratively noised to obtain the target noisy image; wherein each round of iterative noise adding comprises the following steps: Obtain the first noise of a noise adding time step corresponding to this round of iterative noise adding; The first noise of the noise adding time step is used to add noise to the stylized image after the previous round of iterative noise adding, so as to obtain the stylized image after the current round of iterative noise adding.

4. The method according to claim 2, characterized in that The step of extracting target noise information that does not satisfy the correlation condition with the image features of the realistic image and the character features of the target character from the pure noise image includes: The following operations are performed using the trained de-stylization model: Splicing the image features of the realistic image and the character features of the target character to obtain a target splicing feature; Using the target splicing feature, predicting the probability that each pixel in the pure noise image contains noise information; Based on the probability that each pixel in the pure noise image contains noise information, obtaining the noise distribution in the pure noise image; Based on the noise distribution in the pure noise image, the target noise information in the pure noise image is extracted.

5. The method according to claim 2, characterized in that The target noise information includes: second noises corresponding to each of the multiple denoising time steps; the second noises corresponding to each of the multiple denoising time steps have different intensities; The step of removing the target noise information from the pure noise image to obtain a target realistic image of the target person includes: The following operations are performed using the trained de-stylization model: In the multiple denoising time steps, the pure noise image is iteratively denoised to obtain a target realistic image of the target person; wherein each round of iterative denoising includes the following steps: Get the second noise of a denoising time step corresponding to this round of iterative denoising; The second noise of the one denoising time step is removed from the pure noise image after the previous round of iterative denoising to obtain the pure noise image after the current round of iterative denoising.

6. The method according to claim 5, characterized in that The image features of the realistic type image include: sub-image features corresponding to each of the multiple denoising time steps; The obtaining of the second noise of a denoising time step corresponding to the current round of iterative denoising comprises: Splicing the character features of the target person with the sub-image features corresponding to the one denoising time step to obtain the sub-splicing features corresponding to the one denoising time step; Based on the sub-joint features corresponding to the one denoising time step, predict the probability that each pixel in the pure noise image after the previous round of iterative denoising contains noise information; Obtaining noise distribution in the pure noise image after the previous round of iterative denoising based on the probability that each pixel in the pure noise image after the previous round of iterative denoising contains noise information; Based on the noise distribution in the pure noise image after the previous round of iterative denoising, the second noise corresponding to the one denoising time step is obtained.

7. The method according to any one of claims 2 to 6, characterized in that: The trained de-stylization model is trained in the following manner: Obtain a sample set including multiple training samples, each training sample including: a sample style image and a sample realistic image of the same sample person; The sample set is used to perform multiple rounds of iterative training on the de-stylization model to be trained until a training termination condition is met, thereby obtaining the trained de-stylization model, wherein each round of iterative training process includes the following steps: Using the de-stylization model used in this round, each training sample input in this round of iteration is converted to obtain a predicted realistic image of each training sample; According to the obtained predicted realistic images and the corresponding sample realistic images, the model loss value of the de-stylization model used in this round is determined, and after adjusting the parameters of the de-stylization model used in this round according to the model loss value, the de-stylization model with the adjusted parameters is used to enter the next round of iterative training.

8. The method according to claim 7, characterized in that The de-stylization model used in this round is used to transform each training sample input in this round of iteration to obtain a predicted realistic image of each training sample, including: For each training sample, the following operations are performed respectively: Noising a sample style image in a training sample to obtain a first noisy image; and Noising a sample realistic image in the training sample to obtain a second noisy image; Extracting character features of a sample character from the first noisy image; and extracting image features of a realistic image from the second noisy image; Based on the character features of the sample character and the image features of the realistic type image, a predicted realistic image of the sample character is obtained.

9. The method according to claim 8, characterized in that The step of obtaining a predicted realistic image of the sample person based on the person features of the sample person and the image features of the realistic type image includes: Based on the character features of the sample character and the image features of the realistic image, stitching is performed to obtain a sample stitching feature; Based on the sample splicing features, obtaining predicted noise information in the second noisy image; The predicted noise information is removed from the second noisy image to obtain a predicted realistic image of the sample person.

10. An image generating device, characterized in that: include: An acquisition module, used to acquire a pure noise image and a stylized image of a target person; An extraction module, used to extract the character features of the target person from the stylized image of the target person; A prediction module, used to extract target noise information that does not meet the correlation condition and has image features of a realistic image and character features of the target character from the pure noise image, wherein the image features of the realistic image are obtained by learning from a plurality of sample realistic images; The denoising module is used to remove the target noise information from the pure noise image to obtain a target realistic image of the target person.

11. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of any one of the methods of claims 1 to 9 are implemented.

12. A computer-readable storage medium, characterized in that: It stores a computer program executable by a computer device, and when the computer program is run on the computer device, the computer device executes the steps of any method described in claims 1 to 9.

13. A computer program product, characterized in that The computer program product comprises a computer program stored on a computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer device, the computer device is caused to execute the steps of the method according to any one of claims 1 to 9.

Images