Method and device for training image generation model, equipment and storage medium

Abstract

The embodiment of the invention provides a method and device for training an image generation model, equipment and a storage medium. The method comprises the following steps: acquiring a reference image and a target image; the reference image and attitude information corresponding to the target image are provided for an image generation model to generate an intermediate image, and the attitude information is used for describing the attitude of a target object in the target image; determining a first area corresponding to a preset part of the target object in the intermediate image; and training an image generation model at least based on the difference between the first region and a second region corresponding to the preset part in the target image.

Description

Technical Field

[0001] The exemplary embodiments disclosed herein relate generally to the field of computers, and more particularly to methods, apparatus, devices, and computer-readable storage media for training image generation models. Background Technology

[0002] With the development of computer technology, animation generation has become a key research area, combining multiple subfields such as computer vision, deep learning, image processing, and pattern recognition. The rapid development of video diffusion models has made it possible to generate highly realistic and controllable dynamic images. These technologies show broad application prospects in various fields, including the entertainment industry, film production, virtual reality, and augmented reality. Summary of the Invention

[0003] In a first aspect of this disclosure, a method for training an image generation model is provided. The method includes: acquiring a reference image and a target image; providing the reference image and pose information corresponding to the target image to the image generation model to generate an intermediate image, the pose information being used to describe the pose of a target object in the target image; determining a first region in the intermediate image corresponding to a preset part of the target object; and training the image generation model based at least on the difference between the first region and a second region in the target image corresponding to the preset part.

[0004] In a second aspect of this disclosure, a method for generating an image is provided. The method includes: providing an input image and target pose information to an image generation model; and acquiring an output image generated by the image generation model, wherein the pose of a preset object in the output image corresponds to the target pose information, wherein the image generation model is trained based on region difference information indicating a difference between a first region in an intermediate image and a second region in a target image, the first and second regions corresponding to preset parts of the target object, and the intermediate image is generated by the image generation model based on a reference image and pose information corresponding to the target image, the pose information describing the pose of the target object in the target image.

[0005] In a third aspect of this disclosure, an apparatus for training an image generation model is provided. The apparatus includes: an acquisition module configured to acquire a reference image and a target image; a providing module configured to provide the reference image and pose information corresponding to the target image to the image generation model to generate an intermediate image, the pose information describing the pose of a target object in the target image; a determining module configured to determine a first region in the intermediate image corresponding to a preset part of the target object; and a training module configured to train the image generation model based at least on the difference between the first region and a second region in the target image corresponding to the preset part.

[0006] In a fourth aspect of this disclosure, an apparatus for generating an image is provided. The apparatus includes an input module configured to provide an input image and target pose information to an image generation model; and a generation module configured to acquire an output image generated by the image generation model, wherein the pose of a preset object in the output image corresponds to the target pose information, wherein the image generation model is trained based on region difference information indicating a difference between a first region in an intermediate image and a second region in the target image, the first and second regions corresponding to preset parts of the target object, and the intermediate image is generated by the image generation model based on a reference image and pose information corresponding to the target image, the pose information describing the pose of the target object in the target image.

[0007] In a fifth aspect of this disclosure, an electronic device is provided. The device includes at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit. When executed by the at least one processing unit, the instructions cause the device to perform the methods of the first or second aspect.

[0008] In a sixth aspect of this disclosure, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program that can be executed by a processor to implement the methods of the first or second aspect.

[0009] It should be understood that the content described in this content section is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0011] Figure 1 Example image generation models according to some embodiments of this disclosure are shown;

[0012] Figure 2 A flowchart illustrating an example process for training an image generation model according to some embodiments of the present disclosure is shown;

[0013] Figure 3 Example training architectures according to some embodiments of this disclosure are shown;

[0014] Figure 4 A block diagram of an apparatus for training an image generation model according to some embodiments of the present disclosure is shown; and

[0015] Figure 5 A block diagram of an electronic device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0016] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0017] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0018] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, which can display the prompt message in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0019] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0020] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0021] The term "in response to" as used herein refers to a state in which a corresponding event occurs or a condition is satisfied. It will be understood that the timing of subsequent actions performed in response to such event or condition is not necessarily strongly correlated with the time when the event occurs or the condition is met. For example, in some cases, subsequent actions may be performed immediately upon the occurrence of the event or the fulfillment of the condition; while in others, they may be performed some time after the occurrence of the event or the fulfillment of the condition.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] It should be noted that the headings of any section / subsection provided herein are not limiting. Various embodiments are described throughout this document, and embodiments of any type may be included under any section / subsection. Furthermore, embodiments described in any section / subsection may be combined in any way with any other embodiments described in the same section / subsection and / or different sections / subsections.

[0024] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The term "some embodiments" should be understood as "at least some embodiments". Other explicit and implicit definitions may also be included below. The terms "first", "second", etc., may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0025] Although existing research has made some progress in image animation generation through generative adversarial networks (GANs) and diffusion-based methods, these methods still have limitations in ensuring the quality of local details and the realism of motion blur in the animation results.

[0026] Specifically, traditional methods typically use the mean squared error (MSE) of the whole-body image as the learning target. While effective, this approach is insufficient to guarantee the appearance quality of smaller areas such as the face and hands. Furthermore, motion blur is common in human-centered videos due to rapid motion and limitations of capture equipment, but existing work does not explicitly consider this factor, resulting in unconditionally synthesized motion blur that affects the realism of the animation.

[0027] Therefore, embodiments of this disclosure propose a scheme for training an image generation model. According to various embodiments of this disclosure, a reference image and a target image can be obtained. Further, the image generation model can be provided with the reference image and pose information corresponding to the target image to generate an intermediate image, wherein the pose information describes the pose of the target object in the target image.

[0028] Accordingly, a first region in the intermediate image corresponding to a predetermined part of the target object can be determined. Furthermore, an image generation model can be trained based at least on the difference between the first region and a second region in the target image corresponding to the predetermined part.

[0029] Therefore, by applying additional loss functions to these specific regions (e.g., preset parts of the target object), embodiments of this disclosure can focus more on optimizing the features of these regions, thereby improving the accuracy and clarity of the generated image. Furthermore, embodiments of this disclosure can also maintain the consistency of the target object in the generated image and improve the realism of the generated image.

[0030] Example embodiments of this disclosure are described below with reference to the accompanying drawings.

[0031] Example System

[0032] Figure 1 An example structure of an example image generation model according to some embodiments of the present disclosure is shown.

[0033] like Figure 1 As shown, the image generation model 135 may include a combination of multiple models or units. For example, the model generation model 135 may include an appearance encoder 140, UNet 145, and ControlNet 150.

[0034] like Figure 1 As shown, input image 105 can be provided to appearance encoder 140 to generate corresponding visual features. Furthermore, input image 105 can also be provided to CLIP (Contrastive Language-Image Pre-training) unit 130 to generate a text description corresponding to input image 105. As shown, such a text description can also be provided to UNet 145.

[0035] Furthermore, the image generation model 135 can also acquire initial noise 110 to perform a denoising process. Further, the image generation model 135 can also acquire one or more control signals.

[0036] As an example, such a control signal may include attitude information 115, motion information 120, and sharpness information 125. Specific details of the control signal will be referenced below. Figure 2 and Figure 3 describe.

[0037] As shown in the figure, control signals can be provided to ControlNet 150 as control signals for the generation process.

[0038] Accordingly, the image generation model 135 can generate a decoded image code 155 based on the acquired input information. By decoding the image code 155 using the decoder, an image corresponding to the pose information 115 can be obtained.

[0039] As an example, such an image generation model 135 can be used to generate a set of images with continuous motion, thereby generating image animation, such as dance animation, etc.

[0040] The following will be further referenced Figure 2 This describes the specific training process of image generation model 135.

[0041] Example process

[0042] Figure 2 A flowchart illustrating an example process 200 for training an image generation model according to some embodiments of the present disclosure is shown. Process 200 can be implemented at the training system. Reference is made below. Figure 1 To describe process 200.

[0043] like Figure 2 As shown in box 210, the training system acquires the reference image and the target image.

[0044] In some embodiments, the training system may acquire video content associated with a target object (e.g., a dancer).

[0045] Furthermore, the training system can extract two video frames from the video content, serving as a reference image and a target image, respectively. For example, the image frame corresponding to the dancer's starting movement can be used as the reference image, and the image frame corresponding to the dance movement can be used as the target image.

[0046] Figure 3 An example process for training a graph generative model according to some embodiments of this disclosure is shown. For example... Figure 3 As shown, the training system can acquire reference image 305 and target image 340 for training image generation model 135.

[0047] In box 220, the training system provides the image generation model with a reference image and pose information corresponding to the target image to generate an intermediate image. The pose information is used to describe the pose of the target object in the target image.

[0048] Continue to refer to Figure 3 The training system can provide the image generation model 135 with a reference image 305, noise data 310, and control signals. In some embodiments, the control signals may include pose information associated with the target image 340.

[0049] As an example, this pose information can describe the pose of a target object (e.g., a dancer) in the target image 340. In some embodiments, such pose information can be characterized by multiple key points of the target object (e.g., a dancer).

[0050] In some examples, taking dance scenes as an example, the dancer's hand movements are often fast, which may lead to motion blur problems. To improve the quality of the trained image generation model, the control signal may also include motion information 320, which may indicate motion vectors associated with preset parts of the target object (e.g., a dancer).

[0051] As an example, considering that the hand region is more prone to motion blur, motion information 320 may include motion vectors associated with a set of key points in the hand region:

[0052]

[0053] Where v represents the motion vector, p h This represents a set of key points for the hand, where i represents the time of the corresponding video frame.

[0054] In some embodiments, the control signal may further include sharpness information 325. This sharpness information 325 may be used, for example, to indicate the sharpness information of a preset part (e.g., hand) of a target object (e.g., a dancer) in the target image 340.

[0055] As an example, we can first compute the Laplace operator:

[0056]

[0057] Among them, I h Let x represent the hand image in target image 340, where x and y represent the row and column of the image pixels, respectively. Further, a sharpness score (i.e., sharpness information) can be obtained by calculating the variance of the Laplacian operator result. A higher sharpness score indicates a clearer hand region in the image, with more distinct edges and details; a lower sharpness score indicates a more blurred hand region.

[0058] Continue to refer to Figure 3 The image generation model 135 can decode the noise based on the received input information to generate the corresponding image code. Furthermore, the decoder 330 can generate an intermediate image 335 by decoding the generated image code.

[0059] Continue to refer to Figure 2 In box 230, the training system determines the first region in the intermediate image that corresponds to a preset part of the target object.

[0060] In some embodiments, in order to improve the stability of the animation content generated by the image generation model, the training system can extract regions corresponding to preset parts of the target object (e.g., a dancer) from the intermediate image 335.

[0061] In some embodiments, such a preset region may include a face, and the training system 335 may determine the facial region in the intermediate image 335. Alternatively or additionally, such a preset region may include a hand, and the training system 335 may determine the hand region in the intermediate image 335.

[0062] In box 240, the training system trains the image generation model based at least on the difference between the first region and the second region in the target image corresponding to the preset part.

[0063] like Figure 3 As shown, the training system can determine the corresponding training loss based on regional differences. For example, when the preset regions include the face, the training system can determine a first loss associated with the face.

[0064] In some embodiments, the training system may determine the first loss based on the following formula:

[0065]

[0066] Among them, I tgt Represents target image 340, I pre Indicates intermediate image 335, M face This represents a mask for the face. Therefore, the training system can determine a first loss based on the difference between a set of pixels in the facial region of the target image 340 and a set of pixels in the facial region of the intermediate image.

[0067] Similarly, when the preset body part includes the hand, the training system can determine a second loss associated with the hand.

[0068] In some embodiments, the training system may determine the second loss based on the following formula:

[0069]

[0070] Among them, I tgt Represents target image 340, I pre Indicates intermediate image 335, M hand A mask representing the hand. Therefore, the training system can determine a second loss based on the difference between a set of pixels in the hand region of the target image 340 and a set of pixels in the hand region of the intermediate image.

[0071] Therefore, the target loss associated with the predefined body part (face and / or hand) can include the first loss. and / or second loss

[0072] In some embodiments, in order to ensure the continuity between the generated intermediate image 335 and the reference image 305, the training system may also determine a third region in the reference image 305 that corresponds to a preset part (e.g., face).

[0073] Furthermore, the training system can determine the similarity between the first region and the third region based on the first feature representation of the first region and the second feature representation of the third region, and determine the loss based on this similarity.

[0074] As an example, the training system can determine the loss based on the following formula.

[0075]

[0076] Where, ψ ref ψ represents the feature representation of the facial region in reference image 305. pre This represents the feature representation of the facial region in the intermediate image 335.

[0077] Based on the process described above, by applying additional loss functions to these specific regions (e.g., preset parts of the target object), embodiments of this disclosure can focus more on optimizing the features of these regions, thereby improving the accuracy and clarity of the generated image. Furthermore, embodiments of this disclosure can also maintain the consistency of the target object in the generated image and improve the realism of the generated image.

[0078] In some embodiments, when training the image generation model 135 based on region supervision, the training system can fix the model parameters of UNet 145 and ControlNet 150 and adjust only the model parameters of the appearance encoder 140.

[0079] Additionally, the training system can perform a multi-stage training process. Specifically, the training system can train the image generation model 135 based on a conventional diffusion loss and tune the model parameters of the appearance encoder 140, UNet 145, and ControlNet 150.

[0080] Furthermore, the training system can perform a region-supervised fine-tuning process. During this fine-tuning process, the training system can adjust the parameters of the appearance encoder 140.

[0081] like Figure 1As described, the image generation model 135 may be based on a diffusion model architecture. In some embodiments, the training system 100 may also perform the training process of the image generation model based on shift signal-to-noise ratio (shift SNR).

[0082] Traditionally, during the training of diffusion models, a signal-to-noise ratio (SNR) linearly correlated with the time step is used to control the generation of noisy data. However, experiments have shown that such a linearly correlated SNR is unsuitable for high-resolution image generation tasks.

[0083] In high-resolution training, the original noise scheduler may fail to effectively destroy and reconstruct images, resulting in poor image quality. By adjusting the SNR, the model's balance between noise and signal during generation can be improved, thereby enhancing image quality.

[0084] Therefore, at the target time step in the training process of image generation model 135, the training system can determine the corresponding target signal-to-noise ratio based on the target time step, so that the target signal-to-noise ratio has a non-linear relationship with the target time step. Furthermore, the training system can train the image generation model based on the target signal-to-noise ratio at the target time step.

[0085] Specifically, the process of determining the noise control factor β can be referred to the following formulas (6) to (10):

[0086]

[0087] α t =1-β t (7)

[0088]

[0089]

[0090]

[0091] Formula (6) is used to calculate the original β value; Formula (7) is used to calculate the original α value; Formula (8) is used to calculate the adjusted SNR for each time step t; Formulas (9) and (10) are used to recalculate the β value based on the adjusted SNR.

[0092] In some embodiments, the training system may also employ a progressive training strategy, that is, first adapting to low-resolution sample data for training, and then using higher-resolution sample data for training in subsequent iterations.

[0093] In some embodiments, after training, such an image generation model can be provided for use in the image generation process. Specifically, during the inference phase of the image generation model, the model can receive an input image, noise information, and control parameters (e.g., pose information, motion vectors, sharpness information) to generate a corresponding output image. Such pose information can indicate the pose of a predefined object (e.g., a dancer) in the desired generated image.

[0094] In some embodiments, noise information can also be determined by performing a predetermined number of diffusion processes on the encoded representation of the input image, thereby improving the quality of the generated result.

[0095] Furthermore, the generated output image may contain a preset object (e.g., a dancer) that corresponds to the pose indicated in the pose information.

[0096] Example devices and equipment

[0097] Embodiments of this disclosure also provide corresponding apparatus for implementing the above methods or processes. Figure 4 A schematic structural block diagram of an apparatus 400 for training an image generation model according to certain embodiments of the present disclosure is shown. The apparatus 400 may be implemented as or included in a training system. The various modules / components in the apparatus 400 may be implemented by hardware, software, firmware, or any combination thereof.

[0098] like Figure 4 As shown, the apparatus 400 includes: an acquisition module 410 configured to acquire a reference image and a target image; a providing module 420 configured to provide the reference image and pose information corresponding to the target image to an image generation model to generate an intermediate image, wherein the pose information is used to describe the pose of a target object in the target image; a determining module 430 configured to determine a first region in the intermediate image corresponding to a preset part of the target object; and a training module 440 configured to train the image generation model based at least on the difference between the first region and a second region in the target image corresponding to the preset part.

[0099] In some embodiments, the acquisition module 410 is further configured to: acquire video content associated with the target object; and acquire two video frames from the video content as a reference image and a target image, respectively.

[0100] In some embodiments, the training module 440 is further configured to: determine a target loss based on a first set of pixel values ​​in a first region and a second set of pixel values ​​in a second region; and train an image generation model based at least on the target loss.

[0101] In some embodiments, the training module 440 is further configured to: determine a third region in a reference image corresponding to a preset location; determine the similarity between the first region and the third region based on a first feature representation of the first region and a second feature representation of the third region; and train an image generation model based on the difference and similarity.

[0102] In some embodiments, the preset location includes the face and / or hands.

[0103] In some embodiments, the preset part is a first preset part, and the providing module 420 is further configured to: provide motion blur information to the image generation model for generating an intermediate image, wherein the motion blur information is associated with a second preset part of the target object, and the first preset part and the second preset part may be the same or different.

[0104] In some embodiments, motion blur information indicates: the sharpness information of the second preset region in the target image; and the motion vector associated with the second preset region.

[0105] In some embodiments, the image generation model is based on a diffusion model, and the training module 440 is further configured to: determine a target signal-to-noise ratio at a target time step, the target signal-to-noise ratio having a non-linear relationship with the target time step; and train the image generation model based on the target signal-to-noise ratio at the target time step.

[0106] In some embodiments, the apparatus 400 further includes an inference module configured to process the input image using a trained image generation model to generate a corresponding output image.

[0107] In some embodiments, the image generation model also generates an output image based on noise information determined by encoding the input image to represent a predetermined number of rounds of diffusion process.

[0108] The units included in device 400 can be implemented in various ways, including software, hardware, firmware, or any combination thereof. In some embodiments, one or more units may be implemented using software and / or firmware, such as machine-executable instructions stored on a storage medium. In addition to or as an alternative to machine-executable instructions, some or all of the units in device 400 may be implemented at least partially by one or more hardware logic components. By way of example and not limitation, exemplary types of hardware logic components that may be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-chips (SoCs), complex programmable logic devices (CPLDs), and so on.

[0109] Figure 5 A block diagram of an electronic device 500 in which one or more embodiments of the present disclosure may be implemented is shown. It should be understood that... Figure 5 The electronic device 500 shown is merely exemplary and should not be construed as limiting the functionality and scope of the embodiments described herein. Figure 5 The electronic device 500 shown can be used to implement the training system described above.

[0110] like Figure 5 As shown, electronic device 500 is in the form of a general-purpose electronic device. Components of electronic device 500 may include, but are not limited to, one or more processors or processing units 510, memory 520, storage device 530, one or more communication units 540, one or more input devices 550, and one or more output devices 560. Processing unit 510 may be a physical or virtual processor and is capable of performing various processes according to programs stored in memory 520. In a multiprocessor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capability of electronic device 500.

[0111] Electronic device 500 typically includes multiple computer storage media. Such media can be any accessible media that is accessible to electronic device 500, including but not limited to volatile and non-volatile media, removable and non-removable media. Memory 520 can be volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory), or some combination thereof. Storage device 530 can be removable or non-removable media and can include machine-readable media, such as flash drives, disks, or any other media that can be used to store information and / or data and can be accessed within electronic device 500.

[0112] Electronic device 500 may further include additional removable / non-removable, volatile / non-volatile storage media. Although not explicitly stated... Figure 5 As shown, disk drives for reading from or writing to removable, non-volatile disks (e.g., "floppy disks") and optical disk drives for reading from or writing to removable, non-volatile optical disks can be provided. In these cases, each drive can be connected to a bus (not shown) via one or more data media interfaces. Memory 520 may include computer program product 525 having one or more program modules configured to perform various methods or actions of various embodiments of this disclosure.

[0113] Communication unit 540 enables communication with other electronic devices via a communication medium. Additionally, the functionality of components of electronic device 500 can be implemented using a single computing cluster or multiple computing machines capable of communicating via communication connections. Therefore, electronic device 500 can operate in a networked environment using logical connections to one or more other servers, network personal computers (PCs), or another network node.

[0114] Input device 550 can be one or more input devices, such as a mouse, keyboard, trackball, etc. Output device 560 can be one or more output devices, such as a monitor, speaker, printer, etc. Electronic device 500 can also communicate with one or more external devices (not shown) via communication unit 540 as needed. These external devices include storage devices, display devices, etc., and can communicate with one or more devices that enable user interaction with electronic device 500, or with any device that enables electronic device 500 to communicate with one or more other electronic devices (e.g., network card, modem, etc.). Such communication can be performed via input / output (I / O) interface (not shown).

[0115] According to an exemplary implementation of this disclosure, a computer-readable storage medium is provided that stores computer-executable instructions thereon, wherein the computer-executable instructions are executed by a processor to implement the methods described above. According to an exemplary implementation of this disclosure, a computer program product is also provided, which is tangibly stored on a non-transitory computer-readable medium and includes computer-executable instructions, which are executed by a processor to implement the methods described above.

[0116] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, devices, and computer program products implemented according to this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0117] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0118] Computer-readable program instructions can be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions that execute on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0119] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0120] Various implementations of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed implementations. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations. The terminology used herein is chosen to best explain the principles, practical applications, or improvements to technology in the market, or to enable others skilled in the art to understand the various implementations disclosed herein.

Claims

1. A method for training an image generation model, comprising: obtaining a reference image and a target image; providing the reference image and pose information corresponding to the target image to the image generation model to generate an intermediate image, the pose information being used to describe a pose of a target object in the target image; determining a first region in the intermediate image corresponding to a preset part of the target object; and training the image generation model based at least on a difference between the first region and a second region in the target image corresponding to the preset part.

2. The method of claim 1, wherein obtaining a reference image and a target image comprises: obtaining video content associated with the target object; and obtaining two video frames from the video content as the reference image and the target image, respectively.

3. The method of claim 1, wherein training the image generation model based at least on a difference between the first region and a second region in the target image corresponding to the preset part comprises: determining a target loss based on a first set of pixel values of the first region and a second set of pixel values of the second region; and training the image generation model based at least on the target loss.

4. The method of claim 1, wherein training the image generation model based at least on a difference between the first region and a second region in the target image corresponding to the preset part comprises: determining a third region in the reference image corresponding to the preset part; determining a similarity between the first region and the third region based on a first feature representation of the first region and a second feature representation of the third region; and training the image generation model based on the difference and the similarity.

5. The method of claim 1, wherein the preset part comprises a face and / or a hand.

6. The method of claim 1, wherein the preset part is a first preset part, further comprising: providing motion blur information to the image generation model for generating the intermediate image, the motion blur information being associated with a second preset part of the target object, the first preset part being the same as or different from the second preset part.

7. The method of claim 6, wherein the motion blur information indicates: sharpness information of the second preset part in the target image; a motion vector associated with the second preset part.

8. The method of claim 1, wherein the image generation model is based on a diffusion model, the method further comprising: determining a target signal-to-noise ratio at a target time step, the target signal-to-noise ratio having a non-linear relationship with the target time step; and training the image generation model based on the target signal-to-noise ratio at the target time step.

9. The method of claim 1, further comprising: processing an input image using the trained image generation model to generate a corresponding output image. ​ ​ ​ ​ ​ 10.The method of claim 9, wherein the image generation model further generates the output image based on noise information determined by performing a predetermined number of diffusion processes on the encoded representation of the input image. 11.A method of generating an image, comprising: providing an input image and target pose information to an image generation model; and obtaining an output image generated by the image generation model, a pose of a preset object in the output image corresponding to the target pose information, wherein the image generation model is trained based on region difference information indicating a difference between a first region of an intermediate image and a second region in a target image, the first region and the second region corresponding to a preset part of a target object, the intermediate image being generated by the image generation model based on a reference image and pose information corresponding to the target image, the pose information being used to describe a pose of a target object in the target image. 12.An apparatus for training an image generation model, comprising: an obtaining module configured to obtain a reference image and a target image; a providing module configured to provide the reference image and pose information corresponding to the target image to an image generation model to generate an intermediate image, the pose information being used to describe a pose of a target object in the target image; a determining module configured to determine a first region in the intermediate image corresponding to a preset part of the target object; and a training module configured to train the image generation model based at least on a difference between the first region and a second region in the target image corresponding to the preset part. 13.An apparatus for generating an image, comprising: an inputting module configured to provide an input image and target pose information to an image generation model; and an outputting module configured to obtain an output image generated by the image generation model, a pose of a preset object in the output image corresponding to the target pose information, wherein the image generation model is trained based on region difference information indicating a difference between a first region of an intermediate image and a second region in a target image, the first region and the second region corresponding to a preset part of a target object, the intermediate image being generated by the image generation model based on a reference image and pose information corresponding to the target image, the pose information being used to describe a pose of a target object in the target image. 14.An electronic device, comprising: at least one processing unit; and at least one memory coupled to the at least one processing unit and storing instructions for execution by the at least one processing unit, the instructions when executed by the at least one processing unit causing the electronic device to perform the method according to any one of claims 1 to 10 or 11. 15.A computer-readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the method according to any one of claims 1 to 10 or 11. ​

Images