Training method of generative model and image generation method

Abstract

The application discloses a training method of a generation model and an image generation method. Relates to the field of artificial intelligence technology, the method comprises the following steps: obtaining a first training sample set; performing first stage training on a first generation model through the first training sample set to obtain a second generation model; obtaining a second training sample set; performing second stage training on the second generation model through the second training sample set to obtain a target generation model. The application solves the technical problem that the generation module in the related art has information loss, thereby causing the accuracy of the generated image to be relatively low.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and more specifically, to a method for training a generative model and an image generation method. Background Technology

[0002] In digital content creation, particularly in virtual try-on, character animation, and e-commerce applications, pose-guided image generation technology plays a crucial role. However, current mainstream methods face inherent limitations in handling this task, primarily in terms of detail fidelity and consistency during image generation. Existing generation models often employ a multi-module, multi-stage architecture, combined through complex intermediate mechanisms. This design leads to bottlenecks in information transmission at each stage, causing a gradual loss of facial and clothing details, severely impacting the fidelity and consistency of the generated image.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a method for training a generative model and an image generation method to at least solve the technical problem in related technologies where information loss in the generation module leads to low accuracy of generated images.

[0005] According to one aspect of the embodiments of this application, a method for training a generative model is provided, comprising: acquiring a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; performing a first stage of training on a first generative model using the first training sample set to obtain a second generative model; acquiring a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image and a second pose image corresponding to the first target image; and performing a second stage of training on the second generative model using the second training sample set to obtain a target generative model.

[0006] Furthermore, before performing a first-stage training on the first generative model using the first training sample set to obtain a second generative model, the method further includes: segmenting the first sample reference image to obtain multiple image blocks; randomly shuffling the order of the multiple image blocks to obtain a processed sample reference image; performing a first-stage training on the first generative model using the first training sample set to obtain a second generative model includes: performing a first-stage training on the first generative model using the processed sample reference image, the first pose image, and the first sample reference image to obtain the second generative model.

[0007] Further, the second generation model is obtained by performing a first-stage training on the first generation model using the processed sample reference image, the first pose image, and the first sample reference image. This includes: processing the first sample reference image, the processed sample reference image, and the first pose image using the encoder in the first generation model to obtain a first target token sequence, wherein the target token sequence consists of a first token sequence corresponding to the first pose image, a second token sequence corresponding to the first sample reference image and random noise, and a third token sequence corresponding to the processed sample reference image; processing the first target token sequence using the diffusion transformer model in the first generation model to obtain a first predicted velocity vector converted from the first noise amount to a predicted sample reference image, wherein the predicted sample reference image is obtained by processing the first target token sequence using the diffusion transformer model; and performing a first-stage training on the diffusion transformer model in the first generation model based on the first predicted velocity vector and the first true velocity vector to obtain the second generation model, wherein the first true velocity vector is calculated based on the first sample reference image and the first noise amount.

[0008] Furthermore, the method further includes: processing the processed sample reference image through the encoder to obtain a first initial token sequence; randomly discarding the first initial token sequence to obtain the third token sequence.

[0009] Further, the second stage of training the second generative model using the second training sample set to obtain the target generation model includes: processing the second training sample set using the encoder in the second generative model to obtain a second target token sequence; processing the second target token sequence using the diffusion transformer model in the second generative model to obtain a second predicted velocity vector converted from a second noise quantity into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence using the diffusion transformer model; and performing a second stage of training the diffusion transformer model in the second generative model based on the second predicted velocity vector and the second true velocity vector to obtain the target generation model, wherein the second true velocity vector is calculated based on the first target image and the second noise quantity.

[0010] According to one aspect of the embodiments of this application, an image generation method is provided, comprising: acquiring a reference image, a target pose image corresponding to a second target image to be generated, and a token sequence corresponding to random noise, wherein the token sequence corresponding to the random noise is randomly sampled from a target noise distribution; processing the reference image and the target pose image through an encoder in a target generation model to obtain a processed token sequence, and obtaining a third target token sequence based on the processed token sequence and the token sequence corresponding to the random noise; and processing the third target token sequence by calibrating it through a diffusion transformer model in the target generation model to obtain a second target image.

[0011] Further, the process of calibrating the third target token sequence through the diffusion transformer model in the target generation model to obtain the second target image includes: calculating the first token and the second token in the third target token sequence to obtain the target attention score corresponding to the first token, wherein the first token is any token in the third target token sequence, and the second token is any token in the third target token sequence; and obtaining the second target image based on the target attention score.

[0012] Further, calculating the target attention score corresponding to the first token in the third target token sequence by calculating the first token and the second token includes: if the first token is a token in the token sequence corresponding to the random noise and the second token is a token in the token sequence corresponding to the reference image, then calculating an initial attention value based on the first token and the second token; calculating a positional bias value based on the first positional information of the first token in the random noise and the second positional information of the second token in the reference image; and calculating the target attention score based on the initial attention value and the positional bias value.

[0013] According to one aspect of the embodiments of this application, an image generation method is provided, comprising: acquiring a reference image uploaded by a client, a target pose image corresponding to a second target image to be generated, and a token sequence corresponding to random noise, wherein the token sequence corresponding to the random noise is randomly sampled from a target noise distribution; processing the reference image and the target pose image in a target generation model in a cloud server using an encoder to obtain a processed token sequence, and obtaining a third target token sequence based on the processed token sequence and the token sequence corresponding to the random noise; processing the third target token sequence by calibrating it using a diffusion transformer model in the target generation model to obtain a second target image; and returning the second target image to the client.

[0014] According to one aspect of the embodiments of this application, a training apparatus for a generative model is provided, comprising: a first acquisition unit, configured to acquire a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; a first training unit, configured to perform a first stage training on a first generative model using the first training sample set to obtain a second generative model; a second acquisition unit, configured to acquire a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image; and a second training unit, configured to perform a second stage training on the second generative model using the second training sample set to obtain a target generative model.

[0015] Furthermore, the apparatus further includes: a segmentation unit, configured to segment the first sample reference image to obtain multiple image blocks before performing a first-stage training on the first generative model using the first training sample set to obtain a second generative model; a first processing unit, configured to randomly shuffle the order of the multiple image blocks to obtain a processed sample reference image; the first training unit is further configured to perform a first-stage training on the first generative model using the processed sample reference image, the first pose image, and the first sample reference image to obtain the second generative model.

[0016] Further, the first training unit includes: a first processing module, configured to process the first sample reference image, the processed sample reference image, and the first pose image through the encoder in the first generative model to obtain a first target token sequence, wherein the target token sequence is composed of a first token sequence corresponding to the first pose image, a second token sequence corresponding to the first sample reference image and random noise, and a third token sequence corresponding to the processed sample reference image; a second processing module, configured to process the first target token sequence through the diffusion transformer model in the first generative model to obtain a first predicted velocity vector converted from the first noise amount into a predicted sample reference image, wherein the predicted sample reference image is obtained by processing the first target token sequence through the diffusion transformer model; and a first training module, configured to perform a first-stage training on the diffusion transformer model in the first generative model based on the first predicted velocity vector and the first true velocity vector to obtain a second generative model, wherein the first true velocity vector is calculated based on the first sample reference image and the first noise amount.

[0017] Furthermore, the device further includes: a second processing unit, configured to process the processed sample reference image through the encoder to obtain a first initial token sequence; and a third processing unit, configured to randomly discard the first initial token sequence to obtain the third token sequence.

[0018] Further, the second training unit includes: a third processing module, used to process the second training sample set through the encoder in the second generation model to obtain a second target token sequence; a fourth processing module, used to process the second target token sequence through the diffusion transformer model in the second generation model to obtain a second predicted velocity vector converted from the second noise amount into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence through the diffusion transformer model; and a second training module, used to perform a second stage of training on the diffusion transformer model in the second generation model based on the second predicted velocity vector and the second true velocity vector to obtain the target generation model, wherein the second true velocity vector is calculated based on the first target image and the second noise amount.

[0019] According to one aspect of the embodiments of this application, an image generation apparatus is provided, comprising: a third acquisition unit, configured to acquire a reference image, a target pose image corresponding to a second target image to be generated, and a token sequence corresponding to random noise, wherein the token sequence corresponding to the random noise is randomly sampled from a target noise distribution; a fourth processing unit, configured to process the reference image and the target pose image through an encoder in a target generation model to obtain a processed token sequence, and to obtain a third target token sequence based on the processed token sequence and the token sequence corresponding to the random noise; and a fifth processing unit, configured to process the third target token sequence by calibrating it through a diffusion transformer model in the target generation model to obtain a second target image.

[0020] Furthermore, the fifth processing unit includes: a calculation module, used to calculate the first token and the second token in the third target token sequence to obtain the target attention score corresponding to the first token, wherein the first token is any token in the third target token sequence and the second token is any token in the third target token sequence; and a determination module, used to obtain the second target image based on the target attention score.

[0021] Further, the calculation module includes: a first calculation submodule, configured to calculate an initial attention value based on the first token and the second token if the first token is a token in the token sequence corresponding to the random noise and the second token is a token in the token sequence corresponding to the reference image; a second calculation submodule, configured to calculate a positional bias value based on the first positional information of the first token in the random noise and the second positional information of the second token in the reference image; and a third calculation submodule, configured to calculate the target attention score based on the initial attention value and the positional bias value.

[0022] According to another aspect of the present invention, an electronic device is also provided, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes a training method for a generative model or an image generation method of any of the above-mentioned methods during runtime.

[0023] According to another aspect of the present invention, a computer-readable storage medium is also provided, the storage medium storing a program, wherein, when the program is running, the device where the storage medium is located executes a training method for the generative model of any of the above-mentioned methods, or an image generation method.

[0024] According to another aspect of the present invention, a computer program product is also provided, including a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement a training method for a generative model of any of the above-mentioned methods, or an image generation method.

[0025] In this embodiment, the following steps are employed: obtaining a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; performing a first stage of training on a first generation model using the first training sample set to obtain a second generation model; obtaining a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image; performing a second stage of training on the second generation model using the second training sample set to obtain a target generation model, thereby solving the technical problem in related technologies where the generation module suffers from information loss, leading to relatively low accuracy of the generated images.

[0026] In this scheme, the generative model is self-supervised trained using a first training sample set. In this stage, the model learns the ability to recover a complete and clear image from damaged contextual information through self-supervised learning, thereby establishing an intrinsic connection between appearance and structure and strong prior knowledge. Building upon the first stage of pre-training, the second stage of training focuses more on specific generation tasks. Using a second training sample set, the model learns how to recover from a noisy state to a state that matches the first target image (i.e., the image to be ultimately generated). This two-stage training strategy fundamentally solves the problem of information loss in complex multi-module pipelines and overcomes the problem of insufficient model generalization ability caused by data scarcity. The first stage of self-supervised pre-training enables the model to learn the skill of recovering image structure and appearance from a single image. The second stage of fine-tuning further optimizes the model's generation ability under specific control conditions, ensuring high accuracy and efficiency on specific generation tasks, thereby achieving the technical effect of improving the quality of generated images. Attached Figure Description

[0027] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0028] Figure 1 This is a hardware structure block diagram of a computer terminal provided according to Embodiment 1 of this application;

[0029] Figure 2 This is a flowchart of the training method for the generative model provided in Embodiment 1 of this application;

[0030] Figure 3 This is a schematic diagram of the posture according to Embodiment 1 of this application;

[0031] Figure 4 This is a schematic diagram of the training method for the generative model provided in Embodiment 1 of this application. Figure 1 ;

[0032] Figure 5 This is a schematic diagram of the training method for the generative model provided in Embodiment 1 of this application. Figure 2 ;

[0033] Figure 6 This is a flowchart of the image generation method provided according to Embodiment 2 of this application;

[0034] Figure 7 This is a flowchart of the image generation method provided according to Embodiment 3 of this application;

[0035] Figure 8 This is a schematic diagram of a training apparatus for a generative model provided according to Embodiment 4 of this application;

[0036] Figure 9 This is a schematic diagram of an image generation apparatus according to Embodiment 5 of this application;

[0037] Figure 10 This is a structural block diagram of an electronic device provided according to Embodiment Six of this application. Detailed Implementation

[0038] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0039] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0040] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0041] Example 1

[0042] According to an embodiment of this application, a method for training a generative model is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0043] The method embodiment provided in Embodiment 1 of this application can be executed on a mobile terminal, computer terminal, or similar computing device. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing a training method for generative models is shown. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA, and the processor set 102 may include a processor set, Figure 1 The model is illustrated using 102a, 102b, ..., 102n, and includes a memory 104 for storing data and a transmission model 106 for communication functions. In addition, it may include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0044] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be implemented wholly or partially as software, hardware, firmware, or any other combination. Furthermore, the data processing circuits may be a single, independent processing model, or may be wholly or partially integrated into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuit serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0045] The memory 104 can be used to store software programs and models of application software, such as the program instructions / data storage device corresponding to the training method of the generative model in the embodiments of this application. The processor 102 executes various functional applications and data processing by running the software programs and models stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0046] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a radio frequency (RF) model, used for wireless communication with the Internet.

[0047] The display may be a touchscreen LCD display that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0048] Under the aforementioned operating environment, this application provides the following: Figure 2 The training method for the generative model is shown. Figure 2 This is a flowchart of the training method for the generative model according to Embodiment 1 of this application. The training method for the generative model includes:

[0049] Step S201: Obtain a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image.

[0050] Optionally, a first training sample set is obtained, which includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image.

[0051] It should be noted that the first sample reference image contains appearance information of a specific person. The first pose image is associated with the first sample reference image; the first pose image provides the person's pose information in the image, such as limb position and body posture. The first pose image serves as a constraint, ensuring that the model can generate an image based on the specified pose. For example, as... Figure 3 The sample reference image and its corresponding pose image are shown. The first noisy image latent variable is generated based on the first sample reference image and random noise. In the diffusion transformer model, the image generation process is regarded as a reverse diffusion process from noise to a clear image.

[0052] In an alternative embodiment, the objective of the model is to extract latent variables from standard Gaussian noise. ( (As the identity matrix), learn a latent variable from the noise to the clean image. The straight-line trajectory was obtained, and noise was constructed. and clean latent variables A linear interpolation path between [0,1]. A point on this path where time (t) belongs to the interval [0,1] is... Provided. This is the reference image for the first sample.

[0053] It should be noted that the purpose of the first training sample set is to reconstruct the original, clear sample reference image using only a single image (i.e., the first sample reference image mentioned above) under damaged context conditions.

[0054] Step S202: The first generative model is trained in the first stage using the first training sample set to obtain the second generative model.

[0055] Optionally, the first generative model is trained using a first training sample set so that it can gradually master the skill of recovering images from disrupted context. It should be noted that during this training phase, the first generative model needs to recover the first sample reference image.

[0056] At this stage, the model's learning does not rely on paired image data. Instead, it utilizes multiple variations of a single image, namely its pose image and the first noisy image latent variable generated at different time steps t by mixing standard Gaussian noise and image content, to construct and understand the complex relationship between image content and structure.

[0057] Related technologies heavily rely on paired supervised data (i.e., images of the same person and clothing in different poses) for training. However, paired supervised data is difficult to obtain on a large scale, resulting in insufficient diversity in the training set. Therefore, when faced with clothing styles, character features, or complex backgrounds not seen in the training set, the model performance drops sharply, and its generalization ability is poor.

[0058] By employing self-supervised learning on a single image, the problem of insufficient diversity in the training dataset is addressed, enhancing the model's generalization ability and enabling it to handle out-of-distribution data. For example, the model can successfully process complex clothing styles, diverse facial features, and backgrounds without experiencing a significant performance degradation, unlike models that rely on paired supervised data. Through this stage of training, the second generative model demonstrates stronger adaptability and higher-quality generation results in subsequent fine-tuning and applications.

[0059] Step S203: Obtain a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image.

[0060] Optionally, a second training sample set is obtained, which includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image.

[0061] It should be noted that the second sample reference image refers to an image containing the appearance information of the target person, the first target image refers to the image expected to be the final output of the second generative model, and the second pose image corresponds to the pose information in this target image. It details the person's limb positions, body angles, etc., serving as guidance and constraints when the model generates new images. It also represents the model's expectation of retrieving latent variables from standard Gaussian noise. ( (As the identity matrix), learn a latent variable from the noise to the clean image. The straight-line trajectory was obtained, and noise was constructed. and clean latent variables A linear interpolation path between [0,1]. A point on this path where time (t) belongs to the interval [0,1] is... Given. In the second phase of training, This is the first target image.

[0062] Step S204: The second generation model is trained in the second stage using the second training sample set to obtain the target generation model.

[0063] Alternatively, unlike the various variations that process a single image during the pre-training phase, the fine-tuning phase (i.e., the second-stage training) uses specific pairs of data (i.e., a second sample reference image and a first target image).

[0064] In an optional embodiment, the model begins the second phase of training by receiving a second sample reference image, a second pose image, and a second noisy image latent variable at the target time step t. The goal of this phase is to learn how to recover the first target image from the noise given the appearance and pose conditions.

[0065] Similar to the first stage, the model also builds upon the noise in the second stage. to clean image latent variables The linear interpolation path. It refers to the first target image under a specific task, that is, the expected output image.

[0066] Through the first stage of self-supervised pre-training, the model learns general image restoration skills. However, to achieve better performance in real-world applications, the model needs to be optimized for specific tasks through a second stage of fine-tuning to ensure that it can generate images that have the same appearance features as the second sample reference image and meet the pose requirements in the second pose image.

[0067] In summary, the generative model is self-supervised trained using the first training sample set. In this stage, the model learns the ability to recover a complete and clear image from damaged contextual information through self-supervised learning, thereby establishing an intrinsic connection between appearance and structure and strong prior knowledge. Building upon the first stage of pre-training, the second stage of training focuses more on specific generation tasks. Using the second training sample set, the model learns how to recover from a noisy state to a state that matches the first target image (i.e., the image to be ultimately generated). This two-stage training strategy fundamentally solves the problem of information loss in complex multi-module pipelines and overcomes the problem of insufficient model generalization ability caused by data scarcity. The first stage of self-supervised pre-training enables the model to learn the skill of recovering image structure and appearance from a single image. The second stage of fine-tuning further optimizes the model's generation ability under specific control conditions, ensuring high accuracy and efficiency on specific generation tasks, thereby achieving the technical effect of improving the quality of generated images.

[0068] In order to construct the contextual information of the destruction, in the training method of the generative model provided in Embodiment 1 of this application, before performing a first-stage training on the first generative model using a first training sample set to obtain a second generative model, the method further includes: segmenting the first sample reference image to obtain multiple image blocks; randomly shuffling the order of the multiple image blocks to obtain a processed sample reference image; and performing a first-stage training on the first generative model using the first training sample set to obtain a second generative model includes: performing a first-stage training on the first generative model using the processed sample reference image, a first pose image, and the first sample reference image to obtain a second generative model.

[0069] Optionally, the first sample reference image is decomposed into multiple image blocks. These image blocks can be fixed-size regions or content-based segmented regions (such as the head, limbs, and torso of a person). In this embodiment, the first sample reference image is decomposed into multiple image blocks of fixed size. Then, the order of the image blocks is randomly shuffled, disrupting the original relative positional relationships between the image blocks, thereby generating the processed sample reference image. This operation forces the model to learn to recognize the content of the image blocks rather than relying solely on their spatial location.

[0070] Finally, in the first stage of training, the first generative model is trained using the processed sample reference image, the first pose image, and the first sample reference image.

[0071] The above steps construct a "destruction-reconstruction" mechanism in the self-supervised pre-training stage of the model, which helps the model learn the ability to recover details from a single image and enhances the model's understanding of the correspondence between local appearance and structure without relying on a large amount of paired training data.

[0072] To improve the training effect in the pre-training stage, the training method of the generative model provided in Embodiment 1 of this application includes the following steps: First, the first generative model is trained in the first stage using a processed sample reference image, a first pose image, and a first sample reference image to obtain a second generative model. This includes: processing the first sample reference image, the processed sample reference image, and the first pose image using an encoder in the first generative model to obtain a first target token sequence, wherein the target token sequence consists of a first token sequence corresponding to the first pose image, a second token sequence corresponding to the first sample reference image and random noise, and a third token sequence corresponding to the processed sample reference image; processing the first target token sequence using a diffusion transformer model in the first generative model to obtain a first predicted velocity vector converted from a first noise quantity to a predicted sample reference image, wherein the predicted sample reference image is obtained by processing the first target token sequence using the diffusion transformer model; and performing a first stage of training on the diffusion transformer model in the first generative model based on the first predicted velocity vector and the first true velocity vector to obtain the second generative model, wherein the first true velocity vector is calculated based on the first sample reference image and the first noise quantity.

[0073] Optionally, the processed sample reference image, the first pose image, and the first sample reference image are converted into token sequences by initializing the encoder in the generative model. The first token sequence corresponding to the first pose image can be denoted as... It contains latent variable representations of attitude information.

[0074] It should be noted that the first noisy image latent variable corresponding to the target time step t ( The first token sequence () is a mixture of the first sample reference image and random noise. In the diffusion transformer model, it represents the noise state at time step t, which is the second token sequence corresponding to the first sample reference image and random noise mentioned above. The third token sequence corresponding to the processed sample reference image can be denoted as... .

[0075] In an optional embodiment, the first sample reference image can be converted into a token sequence by first initializing the encoder in the generative model, and then based on... Add random noise to obtain the second token sequence.

[0076] The goal of the pre-training phase is to enable the model to learn the process of recovering from a noisy state to a reference image state. Specifically, the learning objective of the diffusion transformer model is to predict a velocity vector that guides the noisy latent variables to gradually approximate the latent variables of the original image along a specific path.

[0077] Therefore, after obtaining the first target token sequence, the first target token sequence is processed by the diffusion transformer model in the first generation model to obtain the first predicted velocity vector converted from the first noise amount into the predicted sample reference image.

[0078] Finally, the first predicted velocity vector and the first true velocity vector are used to train the diffusion transformer model in the first generative model in the first stage to obtain the second generative model. The first true velocity vector is calculated based on the first sample reference image and the first noise level.

[0079] In an optional embodiment, the first true velocity vector can be simplified to The loss function is constructed using the first predicted velocity vector and the first true velocity vector, as shown below:

[0080]

[0081] in, This is the diffusion converter model in the first generative model.

[0082] By employing a pre-training strategy, the model establishes a correspondence between appearance and structure even without clear spatial cues. This ensures that the model can accurately match appearance features to specified structural locations during the generation process, thereby improving the quality and consistency of the generated images.

[0083] In order to construct the context information of the destruction, the training method of the generative model provided in Embodiment 1 of this application further includes: processing the processed sample reference image through an encoder to obtain a first initial token sequence; and randomly discarding the first initial token sequence to obtain a third token sequence.

[0084] Optionally, the processed sample reference image is encoded by an encoder to obtain a first initial token sequence. This sequence consists of local feature representations of the image, but since the image patches have been shuffled, it no longer carries the original structural information. The first initial token sequence is then subjected to random discarding, i.e., a portion of the tokens are randomly selected from the sequence for occlusion (e.g., occlusion at a certain occlusion rate, such as 70%) and the occluded portion is discarded to generate a third token sequence.

[0085] It should be noted that the first token sequence corresponding to the first pose image can be obtained in the following way: first, the first pose image is processed by the encoder to obtain the token sequence, and then the token sequence is randomly occluded and discarded to obtain the first token sequence corresponding to the first pose image.

[0086] The processed sample reference image (whose order has been shuffled) is encoded by an encoder to obtain a first initial token sequence. The first initial token sequence is then randomly discarded to obtain a third token sequence. This allows the model to recover the image even when some image information is missing. The model learns how to recover details from a limited context, thereby enhancing its detail recovery and completion capabilities.

[0087] In an alternative embodiment, it can be achieved through, as follows: Figure 4 The diagram illustrates the first stage of training: acquiring a pose image (the pose corresponding to the reference image), a shuffled reference image, and a reference image; processing the pose image, the shuffled reference image, and the reference image using an encoder to obtain a token sequence; then performing random occlusion processing on the token sequences of the pose image and the shuffled reference image to obtain... and And add random noise to the token sequence of the reference image to obtain the first noisy image latent variable corresponding to the target time step t. ,right , and The sequence dimensions are concatenated to form a unified input sequence (i.e., the first target token sequence mentioned above). Then, the diffusion transformer model processes all the information of the first target token sequence simultaneously, enabling the model to learn and generate priors from massive amounts of unlabeled single images. This breaks the heavy dependence on scarce paired data and greatly improves the model's generalization ability in unfamiliar people, clothing, and scenes.

[0088] To improve the training effect during the fine-tuning stage, the training method for the generative model provided in Embodiment 1 of this application includes the following steps: processing the second training sample set using the encoder in the second generative model to obtain a second target token sequence; processing the second target token sequence using the diffusion transformer model in the second generative model to obtain a second predicted velocity vector converted from a second noise quantity into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence using the diffusion transformer model; and performing a second stage of training on the diffusion transformer model in the second generative model based on the second predicted velocity vector and the second true velocity vector to obtain the target generative model, wherein the second true velocity vector is calculated based on the first target image and the second noise quantity.

[0089] Optionally, the first target image, the second reference image, and the second pose image are encoded by the encoder in the second generative model to obtain the token sequence corresponding to the first target image, the token sequence corresponding to the second reference image, and the token sequence corresponding to the second pose image, based on... Adding noise to the token sequence corresponding to the first target image yields the latent variable of the second noisy image at time step t. Then, the token sequence corresponding to the target image, the token sequence corresponding to the second reference image, and the latent variables of the second noisy image are concatenated to obtain the second target token sequence.

[0090] Then, the second target token sequence is processed by the diffusion transformer model in the second generation model to obtain the second predicted velocity vector converted from the second noise amount into the predicted target image.

[0091] Finally, the diffusion converter model in the second generation model is trained in the second stage based on the second predicted velocity vector and the second true velocity vector to obtain the target generation model.

[0092] In an optional embodiment, the second true velocity vector can be simplified to The loss function is constructed using the second predicted velocity vector and the second true velocity vector, as shown below:

[0093]

[0094] in, The token sequence corresponding to the second pose image. This is the token sequence corresponding to the second reference image.

[0095] In an alternative embodiment, it can be achieved through, as follows: Figure 5 The schematic diagram illustrates the second stage of training: acquiring a pose image (the pose corresponding to the target image), a reference image, and a target image; processing the pose image, reference image, and target image using an encoder to obtain a token sequence; and adding random noise to the token sequence of the reference image to obtain the second noise image latent variable corresponding to the target time step t. ,right , and The sequence dimensions are concatenated to form a unified input sequence (i.e., the second target token sequence mentioned above). Then, the diffusion transformer model processes all the information of the second target token sequence simultaneously, thereby learning the complex dependencies between appearance, structure and content to be generated as a whole, avoiding information loss when it is transferred between different modules.

[0096] Through the second stage of training, the model is able to more effectively utilize control conditions (i.e., pose and reference image) to generate images that are highly similar to the target image. Since the model has already learned strong appearance-structure associations during the pre-training stage, the fine-tuning in the second stage further optimizes the application of these associations under control conditions, ensuring that the model can accurately match appearance features to the correct structural locations during the generation process, thereby improving the detail fidelity of the generated images.

[0097] In the training method of the generative model provided in Embodiment 1 of this application, a first training sample set is obtained, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; a first generation model is trained in a first stage using the first training sample set to obtain a second generation model; a second training sample set is obtained, wherein the second training sample set includes at least: a second sample reference image, a first target image and a second pose image corresponding to the first target image; a second generation model is trained in a second stage using the second training sample set to obtain a target generation model. This solves the technical problem in related technologies where the generation module suffers from information loss, resulting in relatively low accuracy of the generated image.

[0098] In this scheme, the generative model is self-supervised trained using a first training sample set. In this stage, the model learns the ability to recover a complete and clear image from damaged contextual information through self-supervised learning, thereby establishing an intrinsic connection between appearance and structure and strong prior knowledge. Building upon the first stage of pre-training, the second stage of training focuses more on specific generation tasks. Using a second training sample set, the model learns how to recover from a noisy state to a state that matches the first target image (i.e., the image to be ultimately generated). This two-stage training strategy fundamentally solves the problem of information loss in complex multi-module pipelines and overcomes the problem of insufficient model generalization ability caused by data scarcity. The first stage of self-supervised pre-training enables the model to learn the skill of recovering image structure and appearance from a single image. The second stage of fine-tuning further optimizes the model's generation ability under specific control conditions, ensuring high accuracy and efficiency on specific generation tasks, thereby achieving the technical effect of improving the quality of generated images.

[0099] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and models involved are not necessarily essential to this application.

[0100] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0101] Example 2

[0102] This application provides, as follows: Figure 6 The image generation method shown. Figure 6 This is a flowchart of an image generation method according to Embodiment 2 of this application. The image generation method includes:

[0103] Step S601: Obtain the target pose image corresponding to the reference image and the second target image to be generated, and the token sequence corresponding to the random noise, wherein the token sequence corresponding to the random noise is randomly sampled from the target noise distribution.

[0104] Optionally, a reference image is obtained. The reference image (also known as the source image) is an important condition in the model generation process. It provides the appearance features that need to be retained when generating the target image, such as clothing patterns, colors, facial features, etc.

[0105] The target pose image corresponding to the second target image to be generated is obtained. The target pose image specifies the pose that the second target image should have. In portrait image generation tasks, this is usually an image or pose map depicting a person in a specific pose, used to guide the model to generate images with the same pose. The target pose image is used in conjunction with the reference image, enabling the model to learn how to change the person's pose while preserving appearance features.

[0106] A sequence of random noise tokens is obtained, which is randomly sampled from a predefined target noise distribution, which can be a multidimensional Gaussian distribution. In the diffusion model, the random noise token sequence is the starting point of the generation process. The model needs to learn how to gradually recover from this noisy state to a clear target image state. The randomness of the noise ensures that the model can learn to generate images under different initial conditions, enhancing the model's generalization ability.

[0107] Step S602: The reference image and the target pose image are processed by the encoder in the target generation model to obtain the processed token sequence, and the third target token sequence is obtained based on the processed token sequence and the token sequence corresponding to the random noise.

[0108] Optionally, the encoder converts the reference image into a token sequence, which captures the image's appearance features, such as color, texture, and facial features. The target pose image is also converted into a token sequence by the encoder. This sequence focuses on conveying the person's pose information, such as limb posture and body contours, aiming to guide the model to follow this pose during generation. Finally, the obtained token sequence and a random noise token sequence are concatenated to obtain the third target token sequence.

[0109] The construction of the third target token sequence means that the model treats the reference image, target pose image, and random noise as a unified input sequence. In the diffusion model, this unified processing approach can better capture and utilize the complex relationships between appearance, structure, and noise information through the self-attention mechanism, thereby enhancing the model's generative and control capabilities.

[0110] Step S603: The third target token sequence is processed by the diffusion transformer model in the target generation model to obtain the second target image.

[0111] Alternatively, the workflow of the diffusion transformer model can be viewed as a reverse diffusion process, starting with random noise and progressively denoising until a clear image is recovered. During the generation phase, the model receives a third target token sequence containing information from the reference image, the target pose image, and random noise. The model's task is to convert these token sequences into a latent variable representation of the second target image through iterative denoising steps, ultimately decoding them into the image, thus obtaining the second target image.

[0112] By controlling reference and target pose images, the model can generate human images that conform to specific poses while maintaining consistency in clothing and facial features, providing a powerful tool for virtual try-on and the creation of animated characters. In human-computer interaction systems, the model can also generate human images in specific poses based on user commands, enhancing the intuitiveness and controllability of the interaction. Furthermore, the model's efficient generation capabilities make it suitable for large-scale real-time content generation scenarios, such as online games and social media.

[0113] To improve the quality of the second target image, the image generation method provided in Embodiment 2 of this application processes the third target token sequence by using the diffusion transformer model in the target generation model to obtain the second target image. This includes: calculating the first token and the second token in the third target token sequence to obtain the target attention score corresponding to the first token, wherein the first token is any token in the third target token sequence and the second token is any token in the third target token sequence; and obtaining the second target image based on the target attention score.

[0114] Optionally, the diffusion transformer model in this application includes a self-attention layer. This layer performs attention calculations on the tokens in the sequence. The first and second tokens refer to any token in the third target token sequence, which can originate from a reference image, a target pose image, or a sequence corresponding to random noise. For the first token in the third target token sequence, an attention score is calculated between it and all second tokens. It should be noted that the first and second tokens can be the same token. This score reflects the degree to which the first token depends on the second tokens during generation, helping the model understand the relationship between local regions and other parts.

[0115] During the generation process at each time step, the model updates the tokens in the third target token sequence based on the target attention score to gradually denoise and restore the latent variable state of the clear image, thereby obtaining the second target image.

[0116] By using attention scores to guide the generation process, the diffusion transformer model can more precisely control every generation detail, thereby improving the overall quality and control accuracy of the second target image.

[0117] To further improve the accuracy of the attention score, in the image generation method provided in Embodiment 2 of this application, the calculation of the first token and the second token in the third target token sequence to obtain the target attention score corresponding to the first token includes: if the first token is a token in the token sequence corresponding to random noise and the second token is a token in the token sequence corresponding to the reference image, then an initial attention value is obtained based on the first token and the second token; a positional bias value is obtained based on the first position information of the first token in the random noise and the second position information of the second token in the reference image; and a target attention score is obtained based on the initial attention value and the positional bias value.

[0118] Optionally, for the first token in the random noise and the second token in the reference image, embodiments of this application further consider the token's positional information within the image. Positional information is crucial for image generation tasks, helping the model better understand the layout and structure of the image, especially when dealing with spatially relevant features (such as the position of clothing and limbs).

[0119] First, the initial attention value is calculated using a self-attention mechanism. For the tokens in the third target token sequence, there are two parameters: the key token (i.e., the...) ) and query token (i.e. The initial attention value is calculated using the query token of the first token and the key token of the second token.

[0120] For example, the initial attention value is ,in, The i-th image patch corresponding to random noise, i.e., the first token. The j-th image patch corresponding to the reference image, i.e., the second token, and d is the vector dimension of the self-attention layer.

[0121] Then, the model calculates the position bias value using the first position information corresponding to the first token in random noise and the second position information of the second token in the reference image.

[0122] For example, position offset value Wherein, tokens i and j are the two-dimensional spatial coordinates of the image blocks corresponding to them, i.e., the first position information of the first token in random noise and the second position information of the second token in the reference image. Used to control the strength of the position offset value and The radius of influence can be controlled by setting corresponding values ​​based on model testing.

[0123] Finally, the target attention score is calculated using the initial attention value and the positional bias value. .

[0124] It should be noted that if the first token is not a token in the token sequence corresponding to the random noise, or if the second token is not a token in the token sequence corresponding to the reference image, then the corresponding target attention score is directly calculated using the query token of the first token and the key token of the second token. The target attention score is determined.

[0125] By introducing a positional offset value, the spatial structure of the generated image is ensured to be consistent with that of the reference image. For example, by ensuring that the head features are always located at the top of the generated image, feature misalignment in the generated result can be avoided, thereby enhancing the structural rationality of the generated image.

[0126] The improved attention mechanism based on location information proposed in this embodiment ensures that the model can accurately map features from the reference image (such as the head position and clothing texture) to the corresponding positions in the target pose image when generating images. This precise transfer of location information reduces feature misalignment in the generated image, enabling spatially relevant features such as clothing and limbs to be more accurately aligned with the target pose.

[0127] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and models involved are not necessarily essential to this application.

[0128] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0129] Example 3

[0130] This application provides, as follows: Figure 7 The image generation method shown. Figure 7 This is a flowchart of an image generation method according to Embodiment 3 of this application. The image generation method includes:

[0131] Step S701: Obtain the reference image uploaded by the client, the target pose image corresponding to the second target image to be generated, and the token sequence corresponding to random noise, wherein the token sequence corresponding to random noise is randomly sampled from the target noise distribution;

[0132] In step S702, the reference image and the target pose image are processed by the encoder in the target generation model in the cloud server to obtain the processed token sequence. Based on the processed token sequence and the token sequence corresponding to random noise, a third target token sequence is obtained. The third target token sequence is then processed by the diffusion transformer model in the target generation model to obtain the second target image.

[0133] Step S703: Return the second target image to the client.

[0134] It should be noted that the specific steps for generating the image on the cloud server are the same as those in Embodiment 2 of this application, and will not be repeated here.

[0135] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and models involved are not necessarily essential to this application.

[0136] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0137] Example 4

[0138] According to embodiments of this application, a training apparatus for a generative model used to implement the above-described training method for the generative model is also provided, such as... Figure 8 As shown, the device includes: a first acquisition unit 801, a first training unit 802, a second acquisition unit 803, and a second training unit 804.

[0139] The first acquisition unit 801 is used to acquire a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image;

[0140] The first training unit 802 is used to perform a first-stage training on the first generative model using the first training sample set to obtain the second generative model.

[0141] The second acquisition unit 803 is used to acquire a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image;

[0142] The second training unit 804 is used to perform a second-stage training on the second generative model using the second training sample set to obtain the target generative model.

[0143] In the training apparatus for the generative model provided in Embodiment 4 of this application, a first acquisition unit 801 acquires a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; a first training unit 802 performs a first stage of training on the first generative model using the first training sample set to obtain a second generative model; a second acquisition unit 803 acquires a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image; a second training unit 804 performs a second stage of training on the second generative model using the second training sample set to obtain a target generative model, thereby solving the technical problem in related technologies where the generation module suffers from information loss, resulting in relatively low accuracy of the generated image.

[0144] In this scheme, the generative model is self-supervised trained using a first training sample set. In this stage, the model learns the ability to recover a complete and clear image from damaged contextual information through self-supervised learning, thereby establishing an intrinsic connection between appearance and structure and strong prior knowledge. Building upon the first stage of pre-training, the second stage of training focuses more on specific generation tasks. Using a second training sample set, the model learns how to recover from a noisy state to a state that matches the first target image (i.e., the image to be ultimately generated). This two-stage training strategy fundamentally solves the problem of information loss in complex multi-module pipelines and overcomes the problem of insufficient model generalization ability caused by data scarcity. The first stage of self-supervised pre-training enables the model to learn the skill of recovering image structure and appearance from a single image. The second stage of fine-tuning further optimizes the model's generation ability under specific control conditions, ensuring high accuracy and efficiency on specific generation tasks, thereby achieving the technical effect of improving the quality of generated images.

[0145] Optionally, in the training apparatus for the generative model provided in Embodiment 4 of this application, the apparatus further includes: a segmentation unit, used to segment the first sample reference image to obtain multiple image blocks before performing a first-stage training on the first generative model using a first training sample set to obtain a second generative model; a first processing unit, used to randomly shuffle the order of the multiple image blocks to obtain a processed sample reference image; and a first training unit, used to perform a first-stage training on the first generative model using the processed sample reference image, the first pose image, and the first sample reference image to obtain a second generative model.

[0146] Optionally, in the training apparatus for the generative model provided in Embodiment 4 of this application, the first training unit includes: a first processing module, configured to process the first sample reference image, the processed sample reference image, and the first pose image through the encoder in the first generative model to obtain a first target token sequence, wherein the target token sequence consists of a first token sequence corresponding to the first pose image, a second token sequence corresponding to the first sample reference image and random noise, and a third token sequence corresponding to the processed sample reference image; a second processing module, configured to process the first target token sequence through the diffusion transformer model in the first generative model to obtain a first predicted velocity vector converted from the first noise amount to the predicted sample reference image, wherein the predicted sample reference image is obtained by processing the first target token sequence through the diffusion transformer model; and a first training module, configured to perform a first-stage training on the diffusion transformer model in the first generative model based on the first predicted velocity vector and the first true velocity vector to obtain a second generative model, wherein the first true velocity vector is calculated based on the first sample reference image and the first noise amount.

[0147] Optionally, in the training apparatus for the generative model provided in Embodiment 4 of this application, the apparatus further includes: a second processing unit, used to process the processed sample reference image through an encoder to obtain a first initial token sequence; and a third processing unit, used to randomly discard the first initial token sequence to obtain a third token sequence.

[0148] Optionally, in the training apparatus for the generative model provided in Embodiment 4 of this application, the second training unit includes: a third processing module, used to process the second training sample set through the encoder in the second generative model to obtain a second target token sequence; a fourth processing module, used to process the second target token sequence through the diffusion transformer model in the second generative model to obtain a second predicted velocity vector converted from the second noise quantity into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence through the diffusion transformer model; and a second training module, used to perform a second-stage training on the diffusion transformer model in the second generative model based on the second predicted velocity vector and the second true velocity vector to obtain a target generation model, wherein the second true velocity vector is calculated based on the first target image and the second noise quantity.

[0149] It should be noted that the first acquisition unit 801, the first training unit 802, the second acquisition unit 803, and the second training unit 804 mentioned above correspond to steps S201 to S204 in Embodiment 2. The four units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above units, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0150] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0151] Example 5

[0152] According to embodiments of this application, an image generation apparatus for implementing the above-described image generation method is also provided, such as... Figure 9 As shown, the device includes: a third acquisition unit 901, a fourth processing unit 902, and a fifth processing unit 903.

[0153] The third acquisition unit 901 is used to acquire the reference image, the target pose image corresponding to the second target image to be generated, and the token sequence corresponding to random noise, wherein the token sequence corresponding to random noise is randomly sampled from the target noise distribution;

[0154] The fourth processing unit 902 is used to process the reference image and the target pose image through the encoder in the target generation model to obtain the processed token sequence, and to obtain the third target token sequence based on the processed token sequence and the token sequence corresponding to random noise.

[0155] The fifth processing unit 903 is used to process the third target token sequence by means of the diffusion transformer model in the target generation model to obtain the second target image.

[0156] Optionally, in the image generation apparatus provided in Embodiment 5 of this application, the fifth processing unit includes: a calculation module, used to calculate the first token and the second token in the third target token sequence to obtain the target attention score corresponding to the first token, wherein the first token is any one of the tokens in the third target token sequence, and the second token is any one of the tokens in the third target token sequence; and a determination module, used to obtain the second target image based on the target attention score.

[0157] Optionally, in the image generation apparatus provided in Embodiment 5 of this application, the calculation module includes: a first calculation submodule, configured to calculate an initial attention value based on the first token and the second token if the first token is a token in a token sequence corresponding to random noise and the second token is a token in a token sequence corresponding to a reference image; a second calculation submodule, configured to calculate a positional bias value based on the first positional information of the first token in the random noise and the second positional information of the second token in the reference image; and a third calculation submodule, configured to calculate a target attention score based on the initial attention value and the positional bias value.

[0158] It should be noted that the third acquisition unit 901, the fourth processing unit 902 and the fifth processing unit 903 mentioned above correspond to steps S601 to S603 in Embodiment 2. The three units and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 2.

[0159] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 2, but are not limited to the schemes provided in Embodiment 2.

[0160] Example 6

[0161] Embodiments of this application may provide an electronic device, which may be any one of a group of electronic device terminals. Optionally, in this embodiment, the aforementioned electronic device may also be replaced by a terminal device such as a mobile terminal.

[0162] Optionally, in this embodiment, the aforementioned electronic device may be located in at least one of a plurality of network devices in a computer network.

[0163] In this embodiment, the aforementioned electronic device can execute program code for the following steps in the training method of the generative model and the image generation method: obtaining a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; performing a first stage of training on the first generative model using the first training sample set to obtain a second generative model; obtaining a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image and a second pose image corresponding to the first target image; performing a second stage of training on the second generative model using the second training sample set to obtain a target generative model.

[0164] The aforementioned electronic device can execute the program code for the following steps in the training method of the generative model and the image generation method: Before performing a first-stage training on the first generative model using a first training sample set to obtain a second generative model, the method further includes: segmenting the first sample reference image to obtain multiple image blocks; randomly shuffling the order of the multiple image blocks to obtain a processed sample reference image; performing a first-stage training on the first generative model using the first training sample set to obtain a second generative model includes: performing a first-stage training on the first generative model using the processed sample reference image, a first pose image, and the first sample reference image to obtain a second generative model.

[0165] The aforementioned electronic device can execute the program code for the following steps in the training method of the generative model and the image generation method: Training the first generative model in the first stage using the processed sample reference image, the first pose image, and the first sample reference image to obtain the second generative model includes: processing the first sample reference image, the processed sample reference image, and the first pose image using the encoder in the first generative model to obtain a first target token sequence, wherein the target token sequence consists of a first token sequence corresponding to the first pose image, a second token sequence corresponding to the first sample reference image and random noise, and a third token sequence corresponding to the processed sample reference image; processing the first target token sequence using the diffusion transformer model in the first generative model to obtain a first predicted velocity vector converted from the first noise quantity to the predicted sample reference image, wherein the predicted sample reference image is obtained by processing the first target token sequence using the diffusion transformer model; and training the diffusion transformer model in the first generative model in the first stage based on the first predicted velocity vector and the first true velocity vector to obtain the second generative model, wherein the first true velocity vector is calculated based on the first sample reference image and the first noise quantity.

[0166] The aforementioned electronic device can execute the program code for the following steps in the training method of the generative model and the image generation method: processing the processed sample reference image through an encoder to obtain a first initial token sequence; randomly discarding the first initial token sequence to obtain a third token sequence.

[0167] The aforementioned electronic device can execute program code for the following steps in the training method of the generative model and the image generation method: Training the second generative model in the second stage using a second training sample set to obtain a target generation model includes: processing the second training sample set using the encoder in the second generative model to obtain a second target token sequence; processing the second target token sequence using the diffusion transformer model in the second generative model to obtain a second predicted velocity vector converted from a second noise quantity into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence using the diffusion transformer model; and training the diffusion transformer model in the second generative model in the second stage based on the second predicted velocity vector and the second true velocity vector to obtain a target generation model, wherein the second true velocity vector is calculated based on the first target image and the second noise quantity.

[0168] The aforementioned electronic device can execute the following steps in the training method of the generative model and the image generation method: acquiring a reference image, a target pose image corresponding to the second target image to be generated, and a token sequence corresponding to random noise, wherein the token sequence corresponding to random noise is randomly sampled from the target noise distribution; processing the reference image and the target pose image through the encoder in the target generation model to obtain the processed token sequence, and obtaining the third target token sequence based on the processed token sequence and the token sequence corresponding to random noise; and processing the third target token sequence through the diffusion transformer model in the target generation model to obtain the second target image.

[0169] The aforementioned electronic device can execute the program code for the following steps in the training method of the generative model and the image generation method: processing the third target token sequence through the diffusion transformer model in the target generation model to obtain the second target image includes: calculating the first token and the second token in the third target token sequence to obtain the target attention score corresponding to the first token, wherein the first token is any token in the third target token sequence, and the second token is any token in the third target token sequence; and obtaining the second target image based on the target attention score.

[0170] The aforementioned electronic device can execute the following steps in the training method of the generative model and the image generation method: Calculating the target attention score corresponding to the first token and the second token in the third target token sequence includes: if the first token is a token in the token sequence corresponding to random noise and the second token is a token in the token sequence corresponding to the reference image, then calculating based on the first token and the second token to obtain the initial attention value; calculating based on the first position information of the first token in the random noise and the second position information of the second token in the reference image to obtain the position bias value; and calculating based on the initial attention value and the position bias value to obtain the target attention score.

[0171] Optionally, Figure 10 This is a structural block diagram of an electronic device according to an embodiment of this application. Figure 10 As shown, the electronic device 100 may include: one or more ( Figure 10 (Only one is shown in the image) Processor 1002 and memory 104. The electronic device 100 may also include a memory controller to control and manage the memory 1004; the electronic device 100 may also include a peripheral interface to connect to radio frequency models, audio models, and displays, etc.

[0172] The memory can be used to store software programs and models, such as the program instructions / models corresponding to the generative model training method and image generation method and apparatus in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and models stored in the memory, thereby implementing the aforementioned generative model training method and image generation method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to the electronic device 100 via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0173] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: acquiring a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; performing a first stage of training on a first generative model using the first training sample set to obtain a second generative model; acquiring a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image; performing a second stage of training on the second generative model using the second training sample set to obtain a target generative model.

[0174] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: before performing a first-stage training on the first generative model using a first training sample set to obtain a second generative model, the method further includes: segmenting the first sample reference image to obtain multiple image blocks; randomly shuffling the order of the multiple image blocks to obtain a processed sample reference image; performing a first-stage training on the first generative model using the first training sample set to obtain a second generative model includes: performing a first-stage training on the first generative model using the processed sample reference image, the first pose image, and the first sample reference image to obtain a second generative model.

[0175] The processor can invoke information and application programs stored in the memory via a transmission device to execute the following steps: Training the first generative model to obtain a second generative model through a first-stage process using the processed sample reference image, the first pose image, and the first sample reference image, including: processing the first sample reference image, the processed sample reference image, and the first pose image using the encoder in the first generative model to obtain a first target token sequence, wherein the target token sequence consists of a first token sequence corresponding to the first pose image, a second token sequence corresponding to the first sample reference image and random noise, and a third token sequence corresponding to the processed sample reference image; processing the first target token sequence using the diffusion transformer model in the first generative model to obtain a first predicted velocity vector converted from a first noise quantity to a predicted sample reference image, wherein the predicted sample reference image is obtained by processing the first target token sequence using the diffusion transformer model; and training the diffusion transformer model in the first generative model to obtain a second generative model based on the first predicted velocity vector and the first true velocity vector, wherein the first true velocity vector is calculated based on the first sample reference image and the first noise quantity.

[0176] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: process the processed sample reference image through the encoder to obtain a first initial token sequence; randomly discard the first initial token sequence to obtain a third token sequence.

[0177] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: Training the second generative model in a second stage using a second training sample set to obtain a target generative model includes: processing the second training sample set using the encoder in the second generative model to obtain a second target token sequence; processing the second target token sequence using the diffusion transformer model in the second generative model to obtain a second predicted velocity vector converted from a second noise quantity into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence using the diffusion transformer model; and training the diffusion transformer model in the second generative model in a second stage based on the second predicted velocity vector and the second true velocity vector to obtain a target generative model, wherein the second true velocity vector is calculated based on the first target image and the second noise quantity.

[0178] The processor can access the information and application program stored in the memory via the transmission device to perform the following steps: acquiring a reference image, a target pose image corresponding to the second target image to be generated, and a token sequence corresponding to random noise, wherein the token sequence corresponding to random noise is randomly sampled from the target noise distribution; processing the reference image and the target pose image through the encoder in the target generation model to obtain the processed token sequence, and obtaining a third target token sequence based on the processed token sequence and the token sequence corresponding to random noise; and processing the third target token sequence by calibrating it through the diffusion transformer model in the target generation model to obtain the second target image.

[0179] The processor can access the information and application program stored in the memory via the transmission device to perform the following steps: The second target image is obtained by processing the third target token sequence through the diffusion transformer model in the target generation model, including: calculating the target attention score corresponding to the first token and the second token in the third target token sequence, wherein the first token is any token in the third target token sequence, and the second token is any token in the third target token sequence; and obtaining the second target image based on the target attention score.

[0180] The processor can invoke information and application programs stored in the memory through the transmission device to perform the following steps: calculating the target attention score corresponding to the first token and the second token in the third target token sequence, including: if the first token is a token in the token sequence corresponding to random noise and the second token is a token in the token sequence corresponding to the reference image, then calculating based on the first token and the second token to obtain an initial attention value; calculating based on the first position information of the first token in the random noise and the second position information of the second token in the reference image to obtain a positional bias value; and calculating based on the initial attention value and the positional bias value to obtain the target attention score.

[0181] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only. Electronic device 100 can also be a smartphone, tablet computer, handheld computer, mobile internet device (MID), PAD and other terminal devices. Figure 10 This does not limit the structure of the aforementioned electronic device. For example, electronic device 100 may also include components that are more... Figure 10 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 10 The different configurations shown.

[0182] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0183] Example 7

[0184] Embodiments of this application also provide a computer program product. Optionally, in this embodiment, the computer program product can be used to store the program code executed by the training method and image generation method of the generative model provided in Embodiment 1.

[0185] Optionally, in this embodiment, the computer program product may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0186] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0187] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0188] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, units, or models, and may be electrical or other forms.

[0189] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0190] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0191] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0192] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A method for training a generative model, characterized in that, include: Obtain a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; The first generative model is trained in the first stage using the first training sample set to obtain the second generative model; Obtain a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image; The second generation model is trained in the second stage using the second training sample set to obtain the target generation model. The second generation model is trained in the second stage using the second training sample set to obtain the target generation model, which includes: The second training sample set is processed by the encoder in the second generative model to obtain the second target token sequence; The second target token sequence is processed by the diffusion transformer model in the second generation model to obtain a second predicted velocity vector that is converted from the second noise amount into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence through the diffusion transformer model; The diffusion transformer model in the second generation model is trained in the second stage based on the second predicted velocity vector and the second true velocity vector to obtain the target generation model, wherein the second true velocity vector is calculated based on the first target image and the second noise amount.

2. The method according to claim 1, characterized in that, Before performing a first-stage training on the first generative model using the first training sample set to obtain the second generative model, the method further includes: The first sample reference image is segmented to obtain multiple image blocks; The order of the multiple image blocks is randomly shuffled to obtain a processed sample reference image; The first generative model is trained in the first stage using the first training sample set to obtain the second generative model, which includes: The first generative model is trained in the first stage using the processed sample reference image, the first pose image, and the first sample reference image to obtain the second generative model.

3. The method according to claim 2, characterized in that, The first generative model is trained in the first stage using the processed sample reference image, the first pose image, and the first sample reference image to obtain the second generative model, which includes: The encoder in the first generative model processes the first sample reference image, the processed sample reference image, and the first pose image to obtain a first target token sequence, wherein the target token sequence is composed of a first token sequence corresponding to the first pose image, a second token sequence corresponding to the first sample reference image and random noise, and a third token sequence corresponding to the processed sample reference image. The first target token sequence is processed by the diffusion transformer model in the first generation model to obtain a first predicted velocity vector that is converted from a first noise quantity into a predicted sample reference image, wherein the predicted sample reference image is obtained by processing the first target token sequence through the diffusion transformer model; The diffusion converter model in the first generative model is trained in the first stage based on the first predicted velocity vector and the first true velocity vector to obtain the second generative model. The first true velocity vector is calculated based on the first sample reference image and the first noise level.

4. The method according to claim 3, characterized in that, The method further includes: The encoder processes the processed sample reference image to obtain a first initial token sequence; The first initial token sequence is randomly discarded to obtain the third token sequence.

5. An image generation method, characterized in that, include: Obtain the target pose image corresponding to the reference image and the second target image to be generated, and the token sequence corresponding to random noise, wherein the token sequence corresponding to random noise is randomly sampled from the target noise distribution; The reference image and the target pose image are processed by the encoder in the target generation model to obtain a processed token sequence, and a third target token sequence is obtained based on the processed token sequence and the token sequence corresponding to the random noise. The third target token sequence is processed by the diffusion transformer model in the target generation model to obtain the second target image.

6. The method according to claim 5, characterized in that, The second target image is obtained by processing the third target token sequence using the diffusion transformer model in the target generation model, including: The first token and the second token in the third target token sequence are calculated to obtain the target attention score corresponding to the first token, wherein the first token is any token in the third target token sequence and the second token is any token in the third target token sequence; The second target image is obtained based on the target attention score.

7. The method according to claim 6, characterized in that, The target attention score corresponding to the first token is calculated by combining the first token and the second token in the third target token sequence, including: If the first token is a token in the token sequence corresponding to the random noise, and the second token is a token in the token sequence corresponding to the reference image, then the initial attention value is calculated based on the first token and the second token. The position offset value is calculated based on the first position information corresponding to the first token in the random noise and the second position information of the second token in the reference image. The target attention score is calculated based on the initial attention value and the positional bias value.

8. An image generation method, characterized in that, include: The reference image uploaded by the client, the target pose image corresponding to the second target image to be generated, and the token sequence corresponding to random noise are obtained, wherein the token sequence corresponding to random noise is randomly sampled from the target noise distribution; In the cloud server, the encoder in the target generation model processes the reference image and the target pose image to obtain a processed token sequence. Based on the processed token sequence and the token sequence corresponding to the random noise, a third target token sequence is obtained. The third target token sequence is then processed by the diffusion transformer model in the target generation model to obtain a second target image. The second target image is returned to the client.

9. A training device for a generative model, characterized in that, include: The first acquisition unit is used to acquire a first training sample set, wherein the first training sample set includes at least: a first sample reference image and a first pose image corresponding to the first sample reference image; The first training unit is used to perform a first-stage training on the first generative model using the first training sample set to obtain the second generative model. The second acquisition unit is used to acquire a second training sample set, wherein the second training sample set includes at least: a second sample reference image, a first target image, and a second pose image corresponding to the first target image; The second training unit is used to perform a second-stage training on the second generative model using the second training sample set to obtain the target generative model. The second training unit includes: a third processing module for processing the second training sample set through the encoder in the second generation model to obtain a second target token sequence; a fourth processing module for processing the second target token sequence through the diffusion transformer model in the second generation model to obtain a second predicted velocity vector converted from a second noise quantity into a predicted target image, wherein the predicted target image is obtained by processing the second target token sequence through the diffusion transformer model; and a second training module for performing a second stage of training on the diffusion transformer model in the second generation model based on the second predicted velocity vector and the second true velocity vector to obtain the target generation model, wherein the second true velocity vector is calculated based on the first target image and the second noise quantity.

10. An image generation apparatus, characterized in that, include: The third acquisition unit is used to acquire a reference image, a target pose image corresponding to the second target image to be generated, and a token sequence corresponding to random noise, wherein the token sequence corresponding to random noise is randomly sampled from the target noise distribution; The fourth processing unit is used to process the reference image and the target pose image through the encoder in the target generation model to obtain the processed token sequence, and to obtain the third target token sequence based on the processed token sequence and the token sequence corresponding to the random noise. The fifth processing unit is used to process the third target token sequence by calibrating it through the diffusion transformer model in the target generation model to obtain the second target image.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the storage medium is located to perform the training method of the generative model according to any one of claims 1 to 4, or the image generation method according to any one of claims 5 to 8.

12. An electronic device, characterized in that, include: Memory, which stores executable programs; A processor for running the program, wherein the program, when running, executes the training method of the generative model according to any one of claims 1 to 4, or the image generation method according to any one of claims 5 to 8.

13. A computer program product, characterized in that, Includes a computer program or instructions that, when executed by a processor, implement the training method of the generative model according to any one of claims 1 to 4, or the image generation method according to any one of claims 5 to 8.

Images