Image generation method, image generation model, training method of image generation model

Abstract

The application provides an image generation method, an image generation model, a training method of the image generation model, an image generation device, a training device of the image generation model, a storage medium and a program product, and relates to the technical field of artificial intelligence. The method comprises the following steps: obtaining a feature vector of a face region; calling an nth diffusion processing module in N diffusion processing modules included in an image generation model to perform cross-attention processing on the feature vector of the face region and a first input, to obtain a first feature vector, extracting a second feature vector from the first input, performing feature learning on the second feature vector, and obtaining a third feature vector; fusing the third feature vector and the first feature vector to obtain a fourth feature vector, and generating an output of the nth diffusion processing module or a target image according to the fourth feature vector. The method can improve the fidelity of the face in the target image and improve the consistency between the face in the personalized image and the face in the original image.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image generation method, an image generation model, a training method for the image generation model, an image generation device, a training device for the image generation model, a storage medium, and a program product. Background Technology

[0002] With the development of artificial intelligence technology, Artificial Intelligence Generative Content (AIGC) has become a very cutting-edge research direction in recent years.

[0003] In some applications, users may want to generate a completely new image of a person based on their own original facial image. This new image could be a standing image, or an image with a completely new style of clothing, etc. Typically, the user provides the original image, which contains specific elements, such as a "face." This original image is then input into a pre-trained AIGC model to generate a personalized image. This personalized image includes the face from the original image. Ideally, the face in the personalized image should be identical to the face in the original image. However, in practical applications, the face in the personalized image often exhibits facial distortion compared to the face in the original image. Summary of the Invention

[0004] This application provides an image generation method, an image generation model, a training method for the image generation model, an image generation device, a training device for the image generation model, a storage medium, and a program product, aiming to solve the technical problem that faces in personalized images generated by AI models often exhibit distortion.

[0005] To achieve the above objectives, this application adopts the following technical solution:

[0006] Firstly, an image generation method is provided. This method is applied to a pre-trained image generation model, which includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. The method includes: obtaining the feature vector of the face region in the original face image; calling the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and a first input to obtain a first feature vector, where n iterates from 1 to N. When n iterates to 1, the first input is a random noise vector. During the process of n iterating from 2 to N, the first input is the output of the (n-1)th diffusion processing module; calling the nth diffusion processing module to extract a second feature vector from the first input based on the coordinates of the face region, and performing feature learning on the second feature vector to obtain a third feature vector; the coordinates of the face region are the position of the face region in the target image to be generated; calling the nth diffusion processing module to fuse the third feature vector with the first feature vector to obtain a fourth feature vector; generating the output of the nth diffusion processing module based on the fourth feature vector; and determining the target image based on the output of the nth diffusion processing module when n iterates to N.

[0007] In this embodiment, the face region is first learned through cross-attention operation, and then the face region is extracted from the first input and learned again. This helps the image generation model pay more attention to the features of the face region in the original face image when generating the target image, and allocate more "attention" or weight to the more important parts. Therefore, the face region in the target image obtained based on this scheme has richer details and improves the fidelity of the face region in the target image.

[0008] In one possible implementation of the first aspect, the method further includes: obtaining a descriptive feature vector of the descriptive information corresponding to the target image; wherein the descriptive information corresponding to the target image is used to describe the target image from different dimensions; and calling the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and the first input to obtain a first feature vector, including: calling the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector and the first input to obtain a first feature vector.

[0009] Based on the aforementioned technical features, on the one hand, descriptive information is introduced to guide the final image generation; on the other hand, the feature vector of the face region is subjected to cross-attention processing with the first input, and simultaneously, the descriptive feature vector is subjected to cross-attention processing with the first input. The results of the two cross-attention processing are then fused to obtain the first feature vector, thereby better guiding the final image generation. Furthermore, in this embodiment, the face region in the first input is relearned through the first target learning unit. This relearning obtains richer feature information about the face region. During the N traversals, the detail representation of the face region is enhanced in this way each time, resulting in richer detail representation of the face region in the final target image. This improves the fidelity of the face region in the target image and increases its consistency with the face region in the original face image.

[0010] Furthermore, the target image is a new image ultimately generated by the image generation method, which combines the facial features in the original face image with the descriptive feature vector corresponding to the descriptive information. Therefore, it can maintain the basic features of the original face image while incorporating the influence of descriptive information, thereby improving the consistency between the original face image and the target image.

[0011] In one possible implementation of the first aspect, generating the output of the nth diffusion processing module based on the fourth feature vector includes: performing convolution processing on the fourth feature vector to obtain the fifth feature vector; extracting the sixth feature vector from the fourth feature vector based on the coordinates of the face region; performing feature learning on the sixth feature vector to obtain the seventh feature vector; fusing the seventh feature vector with the fifth feature vector to obtain the eighth feature vector; and generating the output of the nth diffusion processing module based on the eighth feature vector.

[0012] Based on the aforementioned technical features, the face region and its feature vector in the first input are first learned. The face region in the first input is then learned again. The third feature vector obtained from this second learning is then fused into the first feature vector to obtain the fourth feature vector. On this basis, the fourth feature vector is convolved to obtain the fifth feature vector. The face region in the fourth feature vector is then learned again. Finally, the seventh feature vector obtained from this second learning is fused with the fifth feature vector to obtain the eighth feature vector. The output of the nth diffusion processing module is generated based on the eighth feature vector. Clearly, the face region in the eighth feature vector has better detail representation than the face region in the first feature vector, thus helping to improve the problem of facial image distortion. In the N traversals, the detail representation of the face region is enhanced in this way each time, resulting in richer detail representation of the face region in the final target image, thereby improving the fidelity of the face region in the target image and achieving higher consistency with the face region in the original face image.

[0013] In one possible implementation of the first aspect, feature learning is performed on the second feature vector to obtain the third feature vector, including: upsampling the second feature vector and performing feature learning on the upsampled second feature vector to obtain an intermediate feature vector; downsampling the intermediate feature vector to the size of the second feature vector to obtain the third feature vector.

[0014] In the above embodiments, the purpose of upsampling is to enlarge the face region. After enlarging, feature learning can learn more details, thereby achieving the goal of enhancing the face region.

[0015] In one possible implementation of the first aspect, the upsampling process for the second eigenvector includes: when the matrix corresponding to the second eigenvector is a non-block matrix, padding the second eigenvector with zeros, such that the matrix corresponding to the zero-padded second eigenvector is a block matrix; and then upsampling the zero-padded second eigenvector.

[0016] Based on the above technical features, through continuous operations such as zero padding, upsampling, downsampling, and zero removal, the resulting seventh feature vector contains richer features of the face region, thereby helping to improve the fidelity of the face region.

[0017] In one possible implementation of the first aspect, calling the nth diffusion processing module to fuse the third feature vector with the first feature vector to obtain the fourth feature vector includes: calling the nth diffusion processing module to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

[0018] In this embodiment, the fourth feature vector formed by weighted fusion has better transition between the face region and the non-face region.

[0019] In one possible implementation of the first aspect, the nth diffusion processing module is called to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector. This includes: calling the nth diffusion processing module to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; fusing the ninth feature vector with the third feature vector in a weighted manner, and updating the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

[0020] In one possible implementation of the first aspect, calling the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain a first feature vector includes: calling the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and the first input to obtain a first cross-attention feature vector; calling the nth diffusion processing module to perform cross-attention processing on the descriptive feature vector and the first input to obtain a second cross-attention feature vector; and calling the nth diffusion processing module to fuse the first cross-attention feature vector and the second cross-attention feature vector to obtain the first feature vector.

[0021] Secondly, an image generation model is provided, comprising N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. Each diffusion processing module includes a cross-attention layer, which comprises a cross-attention unit, a first target learning unit, and a first fusion unit. The outputs of the cross-attention unit and the first target learning unit serve as the inputs to the first fusion unit. Specifically, the cross-attention unit of the nth diffusion processing module performs cross-attention processing on the feature vector of the face region and the first input to obtain the first feature vector. Here, n iterates from 1 to N. When n iterates to 1, the first input is a random noise vector. As n iterates from 2 to N, the first input becomes the nth-th feature vector. The output of the nth diffusion processing module; the feature vector of the face region is the feature vector of the face region in the original face image; the first target learning unit of the nth diffusion processing module is used to extract the second feature vector from the first input according to the coordinates of the face region, and perform feature learning on the second feature vector to obtain the third feature vector; the coordinates of the face region are the position of the face region in the target image to be generated; the first fusion unit of the nth diffusion processing module is used to fuse the third feature vector with the first feature vector to obtain the fourth feature vector, and generate the output of the nth diffusion processing module according to the fourth feature vector. When n traverses to N, the target image is determined according to the output of the nth diffusion processing module.

[0022] In this embodiment, a cross-attention unit is used to learn the face region and its feature vector from the first input for the first time. Then, a first target learning unit learns the face region from the first input again. This second learning yields richer feature information about the face region. Based on this, a first fusion unit fuses the third feature vector obtained from the second learning into the first feature vector to obtain a fourth feature vector. The face region in the fourth feature vector has better detail representation than the face region in the first feature vector, thus helping to improve the problem of facial image distortion. During the N traversals, this method is used to enhance the detail representation of the face region each time, resulting in richer detail representation of the face region in the final target image, thereby improving the fidelity of the face region in the target image and achieving higher consistency with the face region in the original face image.

[0023] In one possible implementation of the second aspect, the cross-attention unit of the nth diffusion processing module is further used to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector; wherein the descriptive feature vector is obtained based on the descriptive information corresponding to the target image, and the descriptive information is used to describe the target image from different dimensions.

[0024] In one possible implementation of the second aspect, the diffusion processing module further includes a convolutional layer, which comprises a convolutional unit, a second target learning unit, and a second fusion unit. The input of the convolutional unit is the output of the cross-attention layer, and the output of the convolutional unit and the output of the second target learning unit are the inputs of the second fusion unit. Specifically, the convolutional unit of the nth diffusion processing module performs convolution processing on the fourth feature vector to obtain the fifth feature vector; the second target learning unit of the nth diffusion processing module extracts the sixth feature vector from the fourth feature vector based on the coordinates of the face region, performs feature learning on the sixth feature vector to obtain the seventh feature vector; and the second fusion unit of the nth diffusion processing module fuses the seventh feature vector with the fifth feature vector to obtain the eighth feature vector, and generates the output of the nth diffusion processing module based on the eighth feature vector.

[0025] Based on the aforementioned technical features, in the cross-attention layer, the first target learning unit and the cross-attention unit operate in parallel. The cross-attention unit performs the first learning, and the first target learning unit performs the second learning on the face region. The results of the two learning processes are then fused and fed into the convolutional layer. In the convolutional layer, the second target learning unit and the convolutional unit operate in parallel. The convolutional unit performs the first learning, and the second target learning unit performs the second learning on the face region. The results of the two learning processes are then fused to obtain the eighth feature vector. The output of the nth diffusion processing module is generated based on the eighth feature vector. Clearly, the face region in the eighth feature vector has better detail representation than the face region in the first feature vector, thus helping to improve the problem of facial image distortion. During the N traversals, the detail representation of the face region is enhanced in this way each time, resulting in richer detail representation of the face region in the final target image, thereby improving the fidelity of the face region in the target image and achieving higher consistency with the face region in the original face image.

[0026] In one possible implementation of the second aspect, the first target learning unit of the nth diffusion processing module is specifically used to upsample the second feature vector, and to perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector; the intermediate feature vector is downsampled to the size of the second feature vector to obtain a third feature vector.

[0027] In one possible implementation of the second aspect, the first target learning unit of the nth diffusion processing module is specifically used to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

[0028] In one possible implementation of the second aspect, the first fusion unit of the nth diffusion processing module is specifically used to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

[0029] In one possible implementation of the second aspect, the first fusion unit of the nth diffusion processing module is specifically used to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

[0030] In one possible implementation of the second aspect, the cross-attention unit of the nth diffusion processing module is specifically used to perform cross-attention processing on the feature vector of the face region and the first input to obtain a first cross-attention feature vector; perform cross-attention processing on the descriptive feature vector and the first input to obtain a second cross-attention feature vector; and fuse the first cross-attention feature vector and the second cross-attention feature vector to obtain a first feature vector.

[0031] Thirdly, a training method for an image generation model is provided. This method includes: acquiring noise samples and original face image samples; inputting the noise samples and original face image samples into an initial image generation model, where the noise samples are obtained by adding noise to the original face image samples; the initial image generation model includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2; each diffusion processing module includes a cross-attention layer, which includes a cross-attention unit, a first target learning unit, and a first fusion unit; the outputs of the cross-attention unit and the first target learning unit are the inputs of the first fusion unit; for the nth diffusion processing module, freezing the parameters of the cross-attention unit; and performing cross-attention processing on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention unit to obtain a first feature vector, where the feature vector of the face region is the feature vector obtained based on the face region in the original face image samples; n iterates from 1 to N; when n iterates to 1, the nth diffusion processing module... One input is a noise feature vector, which is a feature vector obtained based on noise samples. As n iterates from 2 to N, the first input is the output of the (n-1)th diffusion processing module. A loss is calculated based on the first feature vector and the target image to obtain the cross-entropy loss. The parameters of the first target learning unit and the first fusion unit are adjusted according to the cross-entropy loss. The target image is a zero-based image with the same size as the first feature vector and ground truth values ​​in the face region. The first target learning unit extracts the second feature vector from the first input based on the coordinates of the face region, performs feature learning on the second feature vector to obtain the third feature vector. The coordinates of the face region are the position of the face region in the target image to be generated. The first fusion unit fuses the third feature vector with the first feature vector to obtain the fourth feature vector. Based on the fourth feature vector, the output of the nth diffusion processing module is generated. When n iterates to N, the target image corresponding to the original face image is determined based on the output of the nth diffusion processing module.

[0032] The image generation model trained using this method can improve the fidelity of faces in personalized images and enhance the consistency between faces in personalized images and faces in original images when generating target images based on original images.

[0033] In one possible implementation of the third aspect, the method further includes: obtaining a descriptive feature vector, inputting the descriptive feature vector into an initial image generation model, wherein the descriptive feature vector is a feature vector of descriptive information corresponding to the target image, and the descriptive information is used to describe the target image from different dimensions; wherein, performing cross-attention processing on the feature vector of the face region and the first input based on the parameters of the frozen cross-attention unit to obtain the first feature vector includes: performing cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input based on the parameters of the frozen cross-attention unit to obtain the first feature vector.

[0034] In one possible implementation of the third aspect, the diffusion processing module further includes a convolutional layer, which comprises a convolutional unit, a second target learning unit, and a second fusion unit. The input of the convolutional unit is the output of the cross-attention layer, and the output of the convolutional unit and the output of the second target learning unit are the inputs of the second fusion unit. The method further includes: for the nth diffusion processing module, freezing the parameters of the convolutional unit, performing convolution processing on the fourth feature vector based on the frozen parameters of the convolutional unit to obtain the fifth feature vector; extracting the sixth feature vector from the fourth feature vector based on the coordinates of the face region based on the parameters of the second target learning unit, performing feature learning on the sixth feature vector to obtain the seventh feature vector; fusing the seventh feature vector with the fifth feature vector based on the parameters of the second fusion unit to obtain the eighth feature vector, generating the output of the nth diffusion processing module based on the eighth feature vector; and adjusting the parameters of the second target learning unit and the second fusion unit based on the cross-entropy loss.

[0035] In one possible implementation of the third aspect, the method further includes: a first target learning unit, specifically used to upsample the second feature vector and perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector; and to downsample the intermediate feature vector to the size of the second feature vector to obtain a third feature vector.

[0036] In one possible implementation of the third aspect, the method further includes: a first target learning unit, specifically used to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, wherein the matrix corresponding to the zero-padding second feature vector is a block matrix; and to perform upsampling on the zero-padding second feature vector.

[0037] In one possible implementation of the third aspect, the method further includes: a first fusion unit, specifically used to fuse the third feature vector with the first feature vector in a weighted manner to obtain a fourth feature vector.

[0038] In one possible implementation of the third aspect, the method further includes: a first fusion unit, specifically used to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain a ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain a fourth feature vector.

[0039] In one possible implementation of the third aspect, cross-attention processing is performed on the feature vector of the face region, the descriptive feature vector, and the first input based on the parameters of the frozen cross-attention unit to obtain a first feature vector. This includes: performing cross-attention processing on the feature vector of the face region and the first input based on the parameters of the frozen cross-attention unit to obtain a first cross-attention feature vector; performing cross-attention processing on the descriptive feature vector and the first input to obtain a second cross-attention feature vector; and fusing the first cross-attention feature vector and the second cross-attention feature vector to obtain the first feature vector.

[0040] Fourthly, an image generation apparatus is provided, the apparatus comprising a module / unit for performing the method in the first aspect or any implementation thereof.

[0041] Fifthly, a training apparatus for an image generation model is provided, the apparatus comprising a module / unit for performing the methods in the third aspect or any implementation thereof.

[0042] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third, fourth and fifth aspects.

[0043] In a sixth aspect, an image generation apparatus is provided, including a memory and a processor, the memory storing a computer program that executes the image generation method of the first aspect or any possible implementation thereof when the processor is running.

[0044] In a seventh aspect, a training apparatus for an image generation model is provided, including a memory and a processor, wherein the memory stores a computer program, and the computer program executes a training method for the image generation model in the third aspect or any possible implementation thereof when the processor is running.

[0045] The processor mentioned in the seventh aspect above can be a central processing unit (CPU) or a combination of a CPU and a neural network processing processor. The neural network processing processor can include a graphics processing unit (GPU), a neural network processing unit (NPU), and a tensor processing unit (TPU), etc. The TPU is a dedicated integrated circuit from Google, designed entirely for machine learning and artificial intelligence accelerators.

[0046] Eighthly, a computer-readable storage medium is provided storing a computer program that, when run on a processor, executes an image generation method of the first aspect or any possible implementation thereof, or, when run on a processor, executes a training method for an image generation model of the third aspect or any possible implementation thereof.

[0047] Ninthly, a computer program product is provided, the computer program product comprising: computer program code, which, when the computer program code is run on a computer, causes the computer to execute the image generation method of the first aspect or any possible implementation thereof, or the computer program, when run on a processor, executes the training method of the image generation model of the third aspect or any possible implementation thereof.

[0048] In a tenth aspect, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method in any one of the implementations of the first or third aspect described above.

[0049] The aforementioned chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC).

[0050] Eleventhly, an electronic device is provided, which includes the means of any one of the implementations of the fourth to seventh aspects described above.

[0051] In a twelfth aspect, a mobile terminal is provided, the mobile terminal including the apparatus of any one of the implementations of the fourth to seventh aspects described above.

[0052] In a thirteenth aspect, a cloud computing platform is provided, which includes the apparatus described in any of the fourth to seventh aspects above.

[0053] Based on the implementation methods provided in the above aspects, this application can be further combined to provide more implementation methods. The technical effects obtained by the second, third, fourth, fifth, sixth, and seventh aspects are similar to the technical effects obtained by the corresponding technical means in the first aspect, and will not be repeated here. Attached Figure Description

[0054] Figure 1 A schematic diagram of an implementation environment provided by an embodiment of this application is shown.

[0055] Figure 2 A schematic diagram is shown for generating a target image based on an original human face image.

[0056] Figure 3 A schematic diagram of an image generation model provided in an embodiment of this application is shown.

[0057] Figure 4 A flowchart illustrating an image generation method provided in an embodiment of this application is shown.

[0058] Figure 5 A flowchart illustrating a method for feature learning on a second feature vector is shown.

[0059] Figure 6 A flowchart illustrating another image generation method provided in an embodiment of this application is shown.

[0060] Figure 7 A schematic diagram of an image generation model provided in an embodiment of this application is shown.

[0061] Figure 8 A flowchart illustrating an image generation method provided in an embodiment of this application is shown.

[0062] Figure 9 A schematic diagram of another image generation model provided in an embodiment of this application is shown.

[0063] Figure 10 A flowchart illustrating an image generation method provided in an embodiment of this application is shown.

[0064] Figure 11 This is a flowchart illustrating a training method for an image generation model provided in an embodiment of this application.

[0065] Figure 12 A schematic diagram of the cross-entropy loss provided in an embodiment of this application is shown.

[0066] Figure 13 A flowchart illustrating another image generation model training method provided in an embodiment of this application is shown.

[0067] Figure 14 A schematic diagram of an optional structure of the image generation apparatus provided in an embodiment of this application is shown.

[0068] Figure 15 This illustration shows an optional structural diagram of a training apparatus for an image generation model provided in an embodiment of this application.

[0069] Figure 16 A schematic diagram of the hardware structure of an image generation apparatus provided in an embodiment of this application is shown.

[0070] Figure 17 This illustration shows a schematic diagram of the hardware structure of a training device for an image generation model provided in an embodiment of this application. Detailed Implementation

[0071] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0072] To facilitate a clear description of the technical solutions in the embodiments of this application, the terms "first" and "second" are used in the embodiments of this application to distinguish identical or similar items with essentially the same function and purpose. For example, "first chip" and "second chip" are used only to distinguish different chips and do not limit their order. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or execution order, and the terms "first" and "second" do not necessarily imply that they are different.

[0073] It should be noted that, in the embodiments of this application, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0074] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0075] Artificial intelligence-generated content (AIGC) has become a cutting-edge research area in recent years. AIGC is a technology based on machine learning and natural language processing that can automatically generate various types of content, including text, images, audio, and video. This content can be news articles, novels, pictures, music, and even software code.

[0076] Currently, AIGC is mainly divided into several major directions such as text generation, image generation, video generation, and audio generation. Among them, image generation (AI Image Generation) refers to the generation of images that are not works by human artists based on AIGC technology. This type of image is called "artificial intelligence generated image (hereinafter referred to as personalized image)".

[0077] For example, a user provides an original face image, which is then processed by an AIGC model to obtain a personalized image containing the face from the original image. Ideally, the face in the personalized image should be identical to the face in the original image. However, in practical applications, the face in the personalized image often exhibits facial distortion compared to the face in the original image.

[0078] Based on this, embodiments of this application provide an image generation method. The method includes inputting an original face image into an image generation model to obtain a target image. The target image contains the face in the original face image. In this method, the image generation model can learn the face region in the original face image multiple times, enhance the detail representation of the face region, improve the fidelity of the face in the target image, and improve the consistency between the face in the target image and the face in the original image.

[0079] This application also provides an image generation model, which, for example, is the image generation model used in the above-described image generation method.

[0080] This application also provides a method for training an image generation model. For example, the image generation model trained based on this method is the image generation model used in the above-described image generation method.

[0081] It should be noted that the technical solution of this application is not limited to improving the consistency between the face in the original image and the face in the personalized image, but can also be reasonably extended to improve the fidelity of specific elements in the personalized image and improve the consistency between specific elements in the personalized image and specific elements in the original image. The "face" in this solution can be reasonably extended to any of the "specific elements" mentioned above.

[0082] The implementation environment involved in the embodiments of this application will be described below.

[0083] Please refer to Figure 1 , Figure 1 This diagram illustrates an implementation environment provided in an embodiment of this application. The real-time environment includes a terminal 101 and a server 102, wherein the terminal 101 and the server 102 are connected via a communication network. There can be multiple terminals 101. Figure 1 Only one terminal is shown as an example.

[0084] Server 102 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 102 can also be a node server in a blockchain network.

[0085] Terminal 101 can also be referred to as terminal equipment, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. Terminal 101 can be a mobile phone, smart TV, wearable device, tablet computer, computer with wireless transceiver capabilities, virtual reality (VR) device, augmented reality (AR) device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc. The embodiments of this application do not limit the specific technology or device form used in terminal 101.

[0086] The following explanation uses the original face image as an example, where the specific element contained in the original face image is the "face".

[0087] In one implementation, during the training phase of the image generation model, server 102 can also acquire original face image samples and noise samples, input the noise samples and original face image samples into the initial image generation model, train the initial image generation model, and obtain a trained image generation model. The original face image samples and noise samples can be sent from terminal 101.

[0088] In another implementation, during the training phase of the image generation model, the terminal 101 can independently acquire original face image samples, add noise to the original face image samples to obtain noise samples, input the noise samples and the original face image samples into the initial image generation model, train the initial image generation model, and obtain the trained image generation model.

[0089] In one implementation, during the image generation stage, server 102 can obtain the feature vector of the face region of the original face image; then, it calls the locally deployed image generation model to process the image based on the image generation method provided in this application embodiment to obtain the target image. The original face image may be sent by terminal 101.

[0090] In another implementation, terminal 101 obtains the trained image generation model from server 102, deploys the image processing model on terminal 101, obtains the original face image, and then calls the image generation model provided in this application embodiment to process the image based on the image generation method provided in this application embodiment to obtain the target image.

[0091] Furthermore, embodiments of this application provide an execution device configured with a pre-trained image generation model provided in embodiments of this application. The execution device can run the image generation model to implement the functionality of the image generation method provided in embodiments of this application. The execution device can... Figure 1 Terminal 101 in the middle, or is Figure 1 Server 102 in the middle.

[0092] The application scenarios involved in the embodiments of this application are described below.

[0093] In some scenarios, users may desire personalized, AI-generated images. For example, a personalized image might be generated based on a user-provided original image containing specific elements, including but not limited to a face, hands, feet, arms, legs, chest, back, and head. These elements could also include, but are not limited to, an animal's face, head, limbs, tail, and ears. Similarly, the AI-generated personalized image contains the specific elements from the original image, but all other content in the personalized image besides these elements is AI-generated. This other content might include, for example, the person's (or animal's) style, posture, clothing, and actions.

[0094] The following example, taking a specific element in the original image as a "face" and the original image as an original face image, illustrates the application scenarios of this scheme.

[0095] For example, a personalized image can be a person's pose image. When it is necessary to generate a person's pose image, an original face image can be given and input into a pre-trained AIGC model to obtain a personalized image. The personalized image can be, for example, a person's image in a standing pose or a person's image in a jumping pose, etc. The image information such as the person's clothing style and background in the personalized image is generated by artificial intelligence.

[0096] like Figure 2 As shown, Figure 2 A schematic diagram is shown for generating a target image based on an original human face image. Figure 2The original face image 201 contains the specific element "face", and the target image 202 is a human image in motion, which contains the face in the original face image.

[0097] It should be noted that, in the embodiments of this application, the original face image can be obtained by the execution device in real time or obtained by the execution device from the memory. The original face image can refer to a hand-drawn image or a photographic image.

[0098] For example, personalized images can also be photographic images that meet specific requirements, such as ID photos. When an ID photo needs to be generated, an original face image can be given and input into a pre-trained AIGC model to obtain a personalized image, such as an ID photo, in which the background, hairstyle, clothing style, pose, etc., are generated by artificial intelligence.

[0099] For example, personalized images can also be images of specific styles, such as cartoon style, anime style, ink painting style, etc. When it is necessary to generate an image of a specific style, an original face image can be given and input into a pre-trained AIGC model to obtain a personalized image, such as an anime-style cartoon character image.

[0100] It should be noted that the embodiments of this application can also be applied to other scenarios requiring image compositing, and have broad application prospects in fields such as film, games, and advertising. For example, personalized images can be inserted into movie scenes to display image effects that cannot be achieved in the real world. Furthermore, personalized images can be inserted into game interfaces and advertising interfaces to display the effects desired by the user.

[0101] The image generation method and image generation model provided in the embodiments of this application will be described below. Figure 3 and Figure 4 As shown, Figure 3 A schematic diagram of an image generation model provided in an embodiment of this application is shown. Figure 4 A schematic flowchart of an image generation method provided in an embodiment of this application is shown. The method is applied to an execution device, which is configured with, for example... Figure 3 The image generation model shown is comprised of T U-net networks, each U-net network comprising N cascaded diffusion processing modules, where T is an integer greater than or equal to 1 and N is an integer greater than or equal to 2.

[0102] In this embodiment of the application, when T equals 1, the structure of the image generation model is as follows: Figure 3As shown. When T is greater than 1, the number of diffusion processing modules included in different U-net networks can be different.

[0103] by Figure 3 Taking the image generation model shown as an example, the different diffusion processing modules can process images with different resolutions (scales). For example, the scale corresponding to the (n-1)th diffusion processing module is 16x16, the scale corresponding to the nth diffusion processing module is 32x32, the scale corresponding to the (n+1)th diffusion processing module is 64x64, and so on. In this embodiment, the N cascaded diffusion processing modules included in the image generation model can capture feature information at each scale, which helps to retain more detail information.

[0104] In this embodiment of the application, the diffusion processing module includes a cross-attention layer, which includes a cross-attention unit, a first target learning unit, and a first fusion unit. The outputs of the cross-attention unit and the first target learning unit are the inputs of the first fusion unit.

[0105] Optionally, the diffusion processing module also includes a sampling layer, where the output of the cross-attention layer (the output of the first fusion unit) is the input of the sampling layer.

[0106] Optionally, the diffusion processing module also includes a convolutional layer, which is located between the cross-attention layer and the sampling layer, such as... Figure 3 As shown by the dashed line, the output of the cross-attention layer (the output of the first fusion unit) is the input of the convolutional layer, the output of the convolutional layer is the input of the sampling layer, and the output of the sampling layer is the input of the next diffusion processing module.

[0107] The image generation method provided in this application includes steps 401 to 404.

[0108] Step 401: Obtain the feature vector of the face region in the original face image.

[0109] The original face image is a photograph or image containing a face, used for subsequent image processing and synthesis tasks. In the image generation method provided in this application embodiment, the original face image is the input in the first step. By encoding and processing it, facial features are extracted. These facial features can be used for subsequent image generation, editing, and recognition tasks to achieve specific goals. The goal of this application embodiment is to perform personalized synthesis based on the input original face image to obtain a target image that meets the requirements.

[0110] After obtaining the original face image, a first encoding can be performed on the original face image. The first encoding is an encoding method for the original face image. After the first encoding, the features of the face region of the original face image can be obtained. Therefore, the first encoding is the process of encoding the original face image to extract the features of the face region. Furthermore, the features of the face region can be a feature vector composed of a set of numbers, that is, the feature vector of the face region in the embodiment of this application.

[0111] In one implementation, when performing the first encoding on the acquired original face image, a sub-image containing the face region can be extracted from the original face image. The first encoding is then performed based on the extracted sub-image to obtain the feature vector of the face region, thereby removing the background and improving the accuracy of the first encoding.

[0112] In this embodiment, an image encoder can be configured to perform a first encoding on the original face image to obtain the features of the face region. The image encoder can be a deep learning-based face feature model, such as the ArcFace model or FaceNet model, which can map the face region to a high-dimensional feature space. The deep learning-based face recognition model can then encode the original face image to obtain the feature vector of the face region. Alternatively, the image encoder can also be a face recognition model based on angular cosine distance, which can improve the accuracy of face region recognition by enhancing the similarity of samples within the same category and increasing the difference between samples from different categories. The angular cosine distance-based face recognition model can then encode the original face image to obtain the feature vector of the face region. This embodiment does not specifically limit the type of image encoder.

[0113] Step 402: Call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first feature vector.

[0114] Here, n iterates from 1 to N. When n iterates to 1, the first input is a random noise vector. During the process of n iterating from 2 to N, the first input is the output of the (n-1)th diffusion processing module.

[0115] Step 403: Call the nth diffusion processing module to extract the second feature vector from the first input based on the coordinates of the face region, perform feature learning on the second feature vector, and obtain the third feature vector.

[0116] The coordinates of the face region refer to the location of the face region in the target image to be generated.

[0117] Step 404: Call the nth diffusion processing module to fuse the third feature vector with the first feature vector to obtain the fourth feature vector. Based on the fourth feature vector, generate the output of the nth diffusion processing module.

[0118] When n traverses to N, the execution device determines the target image based on the output of the nth diffusion processing module.

[0119] In the embodiments of this application, such as Figure 3 As shown, the nth diffusion processing module includes a cross-attention layer, which includes a cross-attention unit, a first target learning unit, and a first fusion unit. The outputs of the cross-attention unit and the first target learning unit are the inputs of the first fusion unit.

[0120] In one implementation, the cross-attention layer further includes an input unit and an output unit. The input unit is connected to both the cross-attention unit and the first target learning unit, and is used to receive the output of the previous-level diffusion processing module and pass it to the cross-attention unit and the first target learning unit. The output unit is connected to the first fusion unit and is used to pass the output of the first fusion unit to the next diffusion processing module or to the next processing layer in this diffusion processing module.

[0121] In this embodiment of the application, after the execution device calls the cross-attention unit (hereinafter referred to as the cross-attention unit) in the cross-attention layer of the nth diffusion processing module to obtain the first input and the feature vector of the face region from the input unit, it can perform cross-attention processing on the feature vector of the face region and the first input based on the cross-attention algorithm to obtain the first feature vector.

[0122] Here, the feature vector of the face region is the feature vector of the face region in the original face image. The first input can be a random noise vector or the output of the previous diffusion processing module, wherein the matrix corresponding to the random noise vector has the same or matching size as the matrix corresponding to the first feature vector.

[0123] Specifically, when the nth diffusion processing module is the first diffusion processing module, the first input is a random noise vector. That is, the cross-attention unit processes the feature vector of the face region and the random noise vector based on the cross-attention algorithm to obtain the first feature vector. When the nth diffusion processing module is any one of the second to Nth diffusion processing modules, the first input is the output of the (n-1)th diffusion processing module. For example, for the third diffusion processing module, its cross-attention unit performs cross-attention processing based on the feature vector of the face region and the output of the second diffusion processing module to obtain the first feature vector.

[0124] In this embodiment of the application, after the execution device calls the first target learning unit (hereinafter referred to as the first target learning unit) in the cross-attention layer of the nth diffusion processing module to obtain the first input from the input unit, it can extract the second feature vector from the first input based on the coordinates of the face region.

[0125] The coordinates of the face region are used to indicate the position of the face region in the target image to be generated. In some implementations, the coordinates of the face region can be specified by the user or calculated based on the user-specified coordinates. In other implementations, the user can indicate a preference for the position of the face region, such as center, left, right, top, bottom, etc. The execution device can determine the coordinates of the face region automatically based on these preferences and provide these coordinates to the first target learning unit. The first target learning unit obtains the matrix corresponding to the first input and then extracts the second feature vector from the matrix corresponding to the first input based on the coordinates of the face region.

[0126] The coordinates of the face region can include multiple coordinate values, and the area enclosed by these multiple coordinate values ​​is the face region. For example, the coordinates of the face region can be the coordinates of the top left corner and the bottom right corner.

[0127] It should be noted that in this embodiment, since different diffusion processing modules correspond to different scales, that is, the scale of the diffusion processing module may be different from the scale of the target image, the coordinates of the face region in the target image cannot be directly used to determine the position of the face region in the first input.

[0128] For the nth diffusion processing module, the execution device needs to adaptively transform the coordinates of the face region according to the scale corresponding to the nth diffusion processing module, and then extract the second feature vector corresponding to the coordinates of the processed face region from the first input.

[0129] For example, if the target image has a pixel resolution of 64x64, the coordinates of the face region are (10, 20) at the top left and (70, 80) at the bottom right, and the scale corresponding to the nth diffusion processing module is 32x32, then the execution device can process the coordinates of the face region according to the proportional relationship between the target image and the scale corresponding to the nth diffusion processing module. Specifically, the execution device will compress the top-left (10, 20) and bottom-right (70, 80) coordinates, resulting in the top-left (5, 105) and bottom-right (35, 40) coordinates of the face region. Then, the execution device can extract the second feature vector corresponding to the processed face region coordinates from the first input.

[0130] In the application embodiment, the first target learning unit can perform feature learning on the second feature vector to obtain a third feature vector, which is also the process of relearning the face region. Specifically, the first target learning unit can learn the third feature vector from the second feature vector based on an attention mechanism.

[0131] In one implementation, please refer to Figure 5 , Figure 5 A flowchart illustrating a method for feature learning on a second feature vector is shown. The first target learning unit upsamples the second feature vector and then performs feature learning on the upsampled second feature vector to obtain an intermediate feature vector; the intermediate feature vector is then downsampled to the size of the second feature vector to obtain a third feature vector.

[0132] For example, the first target learning unit extracts the second feature vector from the first input based on the coordinates of the processed face region. For example, the matrix corresponding to the face region in the first input is an 8×8 matrix, which means that the matrix corresponding to the second feature vector is also an 8×8 matrix.

[0133] Then, the first target learning unit upsamples the second feature vector. The purpose of upsampling is to enlarge the face region. After enlarging, feature learning can learn more details, thereby achieving the goal of enhancing the face region.

[0134] In this embodiment of the application, the first target learning unit can upsample the second feature vector to any size. For example, the first target learning unit can upsample the 8×8 matrix corresponding to the second feature vector to a 16×16 matrix or to a 32×32 matrix, so as to achieve the purpose of amplifying the second feature vector.

[0135] Optionally, in this embodiment, the first target learning unit can upsample the second feature vector to the size of the matrix corresponding to the first input. For example, if the size of the matrix corresponding to the first input is 64×64, then the first target learning unit can upsample the 8×8 matrix corresponding to the second feature vector to a 64×64 matrix.

[0136] It should be noted that the size corresponding to the first input is matched with the scale corresponding to the nth diffusion processing module. By upsampling the matrix corresponding to the second feature vector to the size of the matrix corresponding to the first input, the adaptability of the upsampled second feature vector to the nth diffusion processing module can be improved, thereby improving the computational efficiency.

[0137] In this embodiment, the first target learning unit can perform feature learning on the upsampled second feature vector based on the attention mechanism to obtain an intermediate feature vector. Continuing with the example above, the size of the matrix corresponding to the intermediate feature vector is 64×64, and the feature learning process does not change the size of the feature vector.

[0138] It's important to understand that the second feature vector is composed of feature vectors from the face region in the first input, representing the features of the face region. The size of the matrix corresponding to the second feature vector represents the size of the face region in the first input. Accordingly, both the upsampled second feature vector and the intermediate feature vector after relearning represent the features of the face region. Based on this, in order to match the relearned feature vector with the face region in the first feature vector, the features of the face region need to be reduced to the size of the face region in the first input. Therefore, as... Figure 5 As shown, the first target learning unit can downsample the intermediate feature vector, that is, perform a reduction process, to obtain the third feature vector. The matrix corresponding to the third feature vector is an 8×8 matrix. The third feature vector is the result of reducing the intermediate feature vector, and it still represents the features of the face region. Since the third feature vector is the result after relearning, it can enrich the details of the face region and help improve the fidelity of the face region.

[0139] It should be noted that in some cases, when the first target learning unit extracts the second feature vector from the first input, the matrix corresponding to the extracted second feature vector is a non-block matrix. A block matrix is ​​a matrix with the same number of rows and columns, such as the 8×8 matrix in the example above. A non-block matrix is ​​a matrix with different numbers of rows and columns.

[0140] For example, the matrix corresponding to the second feature vector is a 7×8 matrix, i.e. a non-block matrix. In this embodiment, when a non-block matrix appears, it needs to be processed into a block matrix in order to facilitate subsequent upsampling processing.

[0141] In one implementation, the first target learning unit can pad the second feature vector with zeros if the matrix corresponding to the second feature vector is not a block matrix. After the zero-padding, the matrix corresponding to the second feature vector becomes a block matrix.

[0142] Then, the first target learning unit upsamples the zero-padded second feature vector, performs feature learning on the upsampled second feature vector to obtain an intermediate feature vector; the intermediate feature vector is downsampled to the size of the zero-padded second feature vector, and the downsampled result is de-zeroed to obtain the third feature vector.

[0143] For example, the first target learning unit can transform the 7×8 matrix corresponding to the second feature vector into an 8×8 matrix by padding with zeros. Then, the 8×8 matrix corresponding to the zero-padded second feature vector is upsampled, for example, to the size of the matrix corresponding to the first input, such as 64×64. Then, feature learning is performed on the upsampled second feature vector (64×64 matrix) to obtain an intermediate feature vector of size 64×64. Then, the matrix corresponding to the intermediate feature vector is downsampled from 64×64 to 8×8. Finally, the 8×8 matrix is ​​de-zeroed, that is, the part added in the previous zero-padded operation is removed, and the resulting 7×8 matrix is ​​the matrix corresponding to the third feature vector.

[0144] It should be noted that, as Figure 3 As shown, the first target learning unit and the cross-attention unit are processed in parallel.

[0145] In this embodiment, a cross-attention unit is used to learn the face region and its feature vector from the first input for the first time. Then, a first target learning unit learns the face region from the first input again. This second learning yields richer feature information about the face region. Based on this, the execution device can call the first fusion unit (hereinafter referred to as the first fusion unit) in the cross-attention layer of the nth diffusion processing module to fuse the third feature vector obtained from the second learning into the first feature vector, resulting in a fourth feature vector. The face region in the fourth feature vector has better detail representation than the face region in the first feature vector, thus helping to improve the problem of facial image distortion. During the N traversals, the detail representation of the face region is enhanced in this way each time, resulting in richer detail representation of the face region in the final target image, thereby improving the fidelity of the face region in the target image and achieving higher consistency with the face region in the original face image.

[0146] In this embodiment of the application, the process of the first fusion unit fusing the third feature vector with the first feature vector can be implemented in the following ways:

[0147] In the first approach, the first fusion unit can directly replace the region corresponding to the coordinates of the face region in the first feature vector with the third feature vector.

[0148] In the second approach, the first fusion unit can calculate the coordinates of the face region in the third feature vector and the first feature vector to obtain the fourth feature vector.

[0149] Optionally, in this embodiment, the first fusion unit may fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

[0150] The first fusion unit can extract the coordinates of the face region from the first feature vector to form the ninth feature vector.

[0151] As discussed above, the third eigenvector represents the features of the face region. Therefore, the size of the matrix corresponding to the third eigenvector is the same as the size of the matrix corresponding to the ninth eigenvector, providing a basis for fusion. Then, the first fusion unit can perform a weighted calculation on the third and ninth eigenvectors.

[0152] As shown in the following formula:

[0153]

[0154] in, This represents the ninth eigenvector. Represents the third eigenvector. Let α and β represent the features of the fused face region, respectively, and let α + β = 1. Generally, β is larger than α.

[0155] In this embodiment of the application, the first fusion unit obtains... Next, the fusion results need to be processed. Replace the face region in the first feature vector to obtain the fourth feature vector.

[0156] In this embodiment, the fourth feature vector formed by weighted fusion has better transition between the face region and the non-face region.

[0157] In this embodiment of the application, after obtaining the fourth feature vector, the execution device can generate the output of the nth diffusion processing module based on the fourth feature vector.

[0158] In one implementation, the execution device can directly use the fourth feature vector as the output of the nth diffusion processing module.

[0159] In another implementation, such as Figure 3 As shown, the nth diffusion processing module also includes a sampling layer, which is cascaded with the cross-attention layer. The output of the cross-attention layer serves as the input of the sampling layer, and the output of the sampling layer serves as the input of the next diffusion processing module. The function of the sampling layer is to make the resolution corresponding to the output of the nth diffusion processing module consistent with the resolution of the input of the (n+1)th diffusion processing module.

[0160] The execution device can call the sampling layer (hereinafter referred to as the sampling layer) of the nth diffusion processing module to sample the fourth feature vector and obtain the sampling result, which is the output of the nth diffusion processing module.

[0161] Sampling processing can be either upsampling or downsampling. Downsampling reduces the sampling rate of a signal or image, thus decreasing the number of data points. Downsampling (pooling operations) can reduce the size of feature maps, extract key features, reduce computational load, and improve computational efficiency. Common downsampling methods in image processing include: Average Pooling: averaging the values ​​within a local region of the image as the new pixel value; and Max Pooling: averaging the values ​​within a local region of the image as the new pixel value.

[0162] Upsampling increases the sampling rate of a signal or image, essentially increasing the number of data points. This is typically achieved through methods like interpolation to fill in the new data points. Common upsampling methods in image processing include:

[0163] Bilinear interpolation: estimates a new pixel value by using a weighted average of the weights of the four surrounding pixels.

[0164] It should be noted that whether the sampling process performed by the sampling layer is upsampling or downsampling depends on the position of the nth diffusion processing module in the N cascaded diffusion processing modules.

[0165] For example, if the resolution corresponding to the nth diffusion processing module is 16×16 and the resolution corresponding to the (n+1)th diffusion processing module is 8×8, then the sampling processing performed by the sampling layer included in the nth diffusion processing module is downsampling processing.

[0166] For example, if the resolution of the nth diffusion processing module is 16×16 and the resolution of the (n+1)th diffusion processing module is 32×32, then the sampling processing performed by the sampling layer included in the nth diffusion processing module is upsampling processing.

[0167] Optionally, in this embodiment, the multiple diffusion processing modules can employ a progressive downsampling method, decreasing the image size by a factor of 2, until the output of the last diffusion processing module represents the smallest diffusion processing module in the image generation model's processing. Alternatively, the multiple diffusion processing modules can employ a progressive upsampling method, increasing the image size by a factor of 2, until the output of the last diffusion processing module represents the output result required by the image generation model. In one implementation, the diffusion processing module corresponding to the upsampling process and the diffusion processing module corresponding to the downsampling process have a completely symmetrical structure.

[0168] In one implementation, such as Figure 3As shown in the dashed box, the nth diffusion processing module also includes a convolutional layer, which is located between the cross-attention layer and the sampling layer. The output of the cross-attention layer is the input of the convolutional layer, the output of the convolutional layer is the input of the sampling layer, and the output of the sampling layer is the input of the next diffusion processing module.

[0169] Continuing with the example above, if the size of the matrix corresponding to the fourth feature vector is 64×64, the convolutional layer can perform convolution processing on this 64×64 matrix to obtain the convolution result, which is also a 64×64 matrix. The convolutional layer transmits this convolution result to the sampling layer, which performs upsampling or downsampling processing on the convolution result to obtain the sampling result, which is the output of the nth diffusion processing module.

[0170] It should be noted that this application does not limit the number of convolutional layers, the kernel size, or the specific structure of the convolutional layers. Convolutional layers are used to extract image features and learn depth information. Convolutional layers may include accompanying pooling layers (i.e., sampling layers) to sample the convolved image, thereby achieving information abstraction, increasing the receptive field, and reducing parameter complexity.

[0171] In this embodiment, as n iterates from 1 to N-1, the output of the nth diffusion processing module is passed to the next diffusion processing module. When n reaches the last module, that is, when n reaches N, the execution device can determine the target image based on the output of the nth diffusion processing module.

[0172] Optionally, when the structure of the image generation model is as follows: Figure 3 As shown, when only one U-net network is included, the execution device can determine the target image based on the output of the Nth diffusion processing module.

[0173] Optionally, when the image generation model includes multiple U-net networks, the execution device can use the output of the Nth diffusion processing module as the input of the next U-net network and repeat the above embodiments. Figure 4 The steps of the image generation method shown are as follows: until the output of the Nth diffusion processing module in the last U-net network is obtained, the target image is then determined based on the output of the Nth diffusion processing module in the last U-net network.

[0174] It should be noted that, in the embodiments of this application, the input of the next U-net network includes the output of the Nth diffusion processing module in the previous U-net network and the feature vector of the face region of the original face image.

[0175] In this embodiment, the third feature vector obtained by relearning is fused into the first feature vector during the processing of each diffusion processing module to obtain the fourth feature vector. Through multiple traversal iterations, the feature representation capability of the face region is continuously enhanced, and the detail representation of the face region in the final target image is richer, thereby improving the fidelity of the face region in the target image and achieving higher consistency with the face region in the original face image.

[0176] Based on the above embodiments, please refer to Figure 6 , Figure 6 This illustration shows a flowchart of another image generation method provided in an embodiment of this application. The method is applied to an execution device, which is configured with, for example... Figure 6 The image generation model shown includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. The image generation method includes steps 601 to 605.

[0177] Step 601: Obtain the feature vector of the face region in the original face image.

[0178] In this embodiment, the process of obtaining the feature vector of the face region can refer to the content disclosed in the above embodiments, and will not be repeated here.

[0179] Step 602: Obtain the descriptive feature vector of the descriptive information corresponding to the target image.

[0180] In this context, descriptive information refers to other content or information that needs to be merged or integrated into the image generation process, besides the original face image. The descriptive information corresponding to the target image is used to describe the target image to be generated from different dimensions, such as the image's attributes, shape, features, style, color, pose, elements contained in the image (or objects contained in the image), and so on.

[0181] In one implementation, the descriptive information may be obtained based on specific requirements or instructions provided by the user, and may be used to guide or influence the final image generation result.

[0182] It should be noted that in the embodiments of this application, the description information can be empty, which means that the user has not provided any specific requirements or instructions. In this case, the description feature vector also does not exist.

[0183] In another implementation, the descriptive information can be pre-set in the image generation model. The image generation model can pre-set multiple candidate descriptive information sets, from which a selection can be made during image generation.

[0184] In another implementation, the descriptive information may include two parts: one part is obtained based on specific requirements or instructions provided by the user, and the other part is obtained by selecting from multiple candidate descriptive information pre-set in the image generation model.

[0185] In this embodiment of the application, the descriptive information may consist of one or more prompt words, such as yellow suit, pink shirt, curly hair, black hair, standing, full-body portrait, etc.

[0186] For example, the descriptive information can take many forms, depending on the nature of the target image and the desired effect. For instance, if a specific artistic style or visual effect is required in the target image, the descriptive information could include representative textures, color distributions, brushstroke patterns, etc., of that style.

[0187] In this scenario, the user can input one or more prompt words using the input device of the execution device. Alternatively, the execution device can display multiple prompt word options in the human-computer interaction interface, allowing the user to select from these options, and the execution device will obtain descriptive information based on the user's selected prompt word.

[0188] In this embodiment, the prompt words are not limited to words expressed in text form. Prompt words can also be text descriptions or speech fragments. Correspondingly, the descriptive information can be keywords, tags, or attributes parsed from text or speech. Alternatively, if the prompt word is an image containing changes in facial posture or expression, the descriptive information can include key point coordinates of the target posture or expression, muscle action units, etc. Or, when the prompt word is an image, the descriptive information can also include pixel-level color, texture, or shape information to guide detail processing during image generation. This embodiment does not impose specific limitations on the types of descriptive information and prompt words.

[0189] In this embodiment, after obtaining the descriptive information, the descriptive information can be further encoded to obtain a descriptive feature vector. The descriptive feature vector is obtained by the execution device through feature extraction of the descriptive information after acquisition. These features describe the attributes, style, shape, or other relevant information of the descriptive information and are used to fuse or map them with the feature vector of the face region of the original face image to guide the final image generation.

[0190] Step 603: Call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector.

[0191] Here, n iterates from 1 to N. When n iterates to 1, the first input is a random noise vector. During the process of n iterating from 2 to N, the first input is the output of the (n-1)th diffusion processing module.

[0192] In this embodiment, after obtaining the feature vector and description feature vector of the face region, the execution device can call the cross-attention unit (hereinafter referred to as the cross-attention unit) in the cross-attention layer of the nth diffusion processing module to perform cross-attention processing on the feature vector, description feature vector, and the first input of the face region. This implementation process includes:

[0193] S1, the cross-attention unit performs cross-attention processing on the feature vector of the face region and the first input to obtain the first cross-attention feature vector.

[0194] S2, the cross-attention unit performs cross-attention processing on the descriptive feature vector and the first input to obtain the second cross-attention feature vector.

[0195] S3, the cross-attention unit fuses the first cross-attention feature vector and the second cross-attention feature vector to obtain the first feature vector.

[0196] In this embodiment, when the diffusion processing module is the first diffusion processing module, the first input is a random noise vector. The cross-attention unit processes the feature vector of the face region and the random noise vector based on the cross-attention algorithm to obtain a first cross-attention feature vector. Then, the descriptive feature vector and the random noise vector are subjected to cross-attention processing to obtain a second cross-attention feature vector. Finally, the first cross-attention feature vector and the second cross-attention feature vector are fused to obtain a first feature vector.

[0197] When the diffusion processing module is any one of the 2nd to Nth diffusion processing modules, the first input is the output of the previous diffusion processing module. For example, for the 3rd diffusion processing module, its cross-attention unit performs cross-attention processing based on the feature vector of the face region and the output of the 2nd diffusion processing module to obtain the first cross-attention feature vector. Then, cross-attention processing is performed on the descriptive feature vector and the output of the 2nd diffusion processing module to obtain the second cross-attention feature vector. Finally, the first and second cross-attention feature vectors are fused to obtain the first feature vector.

[0198] Step 604: Call the nth diffusion processing module to extract the second feature vector from the first input based on the coordinates of the face region, perform feature learning on the second feature vector, and obtain the third feature vector.

[0199] Step 605: Call the nth diffusion processing module to fuse the third feature vector with the first feature vector to obtain the fourth feature vector, and generate the output of the nth diffusion processing module based on the fourth feature vector.

[0200] When n traverses to N, the execution device determines the target image based on the output of the nth diffusion processing module.

[0201] In this embodiment, the implementation process of steps 604 to 605 can refer to the content disclosed in the above embodiments, and will not be repeated here.

[0202] In this embodiment, descriptive information is introduced to guide the final image generation. Simultaneously, the feature vector of the face region is cross-attention processed with the first input, and the descriptive feature vector is also cross-attention processed with the first input. The results of these two cross-attention processes are then fused to obtain the first feature vector, thus better guiding the final image generation. Furthermore, in this embodiment, the face region in the first input is relearned by the first target learning unit. This relearning yields richer feature information about the face region. During N iterations, the detail representation of the face region is enhanced in this way each time, resulting in richer detail representation of the face region in the final target image. This improves the fidelity of the face region in the target image and increases its consistency with the face region in the original face image.

[0203] It should be noted that, in the embodiments of this application, when the image generation model includes multiple U-net networks, the execution device can use the output of the Nth diffusion processing module as the input of the next U-net network and repeat the above embodiments. Figure 4 The steps of the image generation method shown are as follows: until the output of the Nth diffusion processing module in the last U-net network is obtained, the target image is then determined based on the output of the Nth diffusion processing module in the last U-net network.

[0204] For the next U-net network, its input includes the output of the Nth diffusion processing module in the previous U-net network, the feature vector of the face region of the original face image, and the descriptive feature vector of the descriptive information corresponding to the target image.

[0205] Based on the above embodiments, please refer to Figure 7 and Figure 8 As shown, Figure 7 A schematic diagram of an image generation model provided in an embodiment of this application is shown. Figure 8 This illustration shows a flowchart of an image generation method provided in an embodiment of this application. The method is applied to an execution device, which is configured with, for example... Figure 7The image generation model shown.

[0206] The image generation model comprises T U-net networks, each of which includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2, optionally taking the value of 9. Each diffusion processing module includes the same constituent units. Different U-net networks may include different numbers of diffusion processing modules.

[0207] It should be noted that in this embodiment, traversing all N diffusion processing modules of a U-net network once signifies the completion of one denoising process from random noise to the target image. This process needs to be repeated T times to complete the denoising process from random noise to the target image and obtain the final target image. Since the processing of each U-net network is repetitive, this embodiment only describes the processing of the N diffusion processing modules in one U-net network, and the processing of the remaining U-net networks will not be elaborated upon.

[0208] The image generation method includes steps 801 to 808.

[0209] Step 801: Obtain the feature vector of the face region in the original face image.

[0210] In this embodiment, the process of obtaining the feature vector of the face region can be referred to the content disclosed in the above embodiments, and will not be repeated here.

[0211] Step 802: Obtain the descriptive feature vector of the descriptive information corresponding to the target image.

[0212] In this embodiment, the process of obtaining the descriptive feature vector can be referred to the content disclosed in the above embodiments, and will not be repeated here.

[0213] Step 803: Call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector.

[0214] Here, n iterates from 1 to N. When n iterates to 1, the first input is a random noise vector. During the process of n iterating from 2 to N, the first input is the output of the (n-1)th diffusion processing module.

[0215] In this embodiment, the process of obtaining the first feature vector can refer to the content disclosed in the above embodiments, and will not be repeated here.

[0216] Step 804: Call the nth diffusion processing module to extract the second feature vector from the first input based on the coordinates of the face region, perform feature learning on the second feature vector, and obtain the third feature vector.

[0217] Step 805: Call the nth diffusion processing module to fuse the third feature vector with the first feature vector to obtain the fourth feature vector.

[0218] When n traverses to N, the execution device determines the target image based on the output of the nth diffusion processing module.

[0219] In this embodiment, the implementation process of steps 804 to 805 can refer to the content disclosed in the above embodiments, and will not be repeated here.

[0220] In this embodiment of the application, after the first fusion unit fuses to obtain the fourth feature vector, it will pass the fourth feature vector to the convolutional layer cascaded with the cross attention layer.

[0221] Step 806: Call the nth diffusion processing module to perform convolution processing on the fourth feature vector to obtain the fifth feature vector.

[0222] Step 807: Call the nth diffusion processing module to extract the sixth feature vector from the fourth feature vector based on the coordinates of the face region, perform feature learning on the sixth feature vector, and obtain the seventh feature vector.

[0223] Step 808: Call the nth diffusion processing module to fuse the seventh feature vector with the fifth feature vector to obtain the eighth feature vector, and generate the output of the nth diffusion processing module based on the eighth feature vector.

[0224] In the embodiments of this application, such as Figure 7 As shown, the nth diffusion processing module includes a cross-attention layer, a convolutional layer, and a sampling layer. The cross-attention layer includes a cross-attention unit, a first target learning unit, and a first fusion unit. The convolutional layer includes a convolutional unit, a second target learning unit, and a second fusion unit. The outputs of the convolutional units and the second target learning unit are the inputs of the second fusion unit.

[0225] The convolutional layer is located between the cross-attention layer and the sampling layer. The output of the cross-attention layer is the input of the convolutional layer (i.e., the input of the convolutional unit), the output of the convolutional layer (the output of the second fusion unit) is the input of the sampling layer, and the output of the sampling layer is the input of the next diffusion processing module.

[0226] In this embodiment of the application, the execution device may call the convolution unit (hereinafter referred to as the convolution unit) in the convolution layer of the nth diffusion processing module to perform convolution processing on the fourth feature vector to obtain the fifth feature vector.

[0227] In one implementation, the process of the convolution unit performing convolution on the fourth feature vector includes: the convolution unit performs convolution operation on the fourth feature vector based on a pre-set convolution kernel, wherein the size of the convolution kernel is related to the size of the matrix corresponding to the fourth feature vector, and the resulting convolution result is the fifth feature vector.

[0228] In this embodiment, the execution device may further invoke the second target learning unit (hereinafter referred to as the second target learning unit) in the convolutional layer of the nth diffusion processing module to extract the sixth feature vector from the fourth feature vector based on the coordinates of the face region. It should be noted that the second target learning unit and the convolutional unit are processed in parallel, and there is no explicit order between them.

[0229] The coordinates of the face region are used to indicate its position in the target image to be generated. In some implementations, the coordinates of the face region can be user-specified or calculated based on user-specified coordinates. In other implementations, the user can indicate a preference for the position of the face region, such as center, left, right, top, bottom, etc. The execution device can determine the coordinates of the face region automatically based on these preferences and provide these coordinates to the second target learning unit. The second target learning unit obtains the matrix corresponding to the fourth feature vector, and then extracts the portion corresponding to the coordinates of the face region from the matrix corresponding to the fourth feature vector to obtain the sixth feature vector.

[0230] The coordinates of the face region can include multiple coordinate values, and the area enclosed by these multiple coordinate values ​​is the face region. For example, the coordinates of the face region can be the coordinates of the top left corner and the bottom right corner.

[0231] It should be noted that in this embodiment, since different diffusion processing modules correspond to different scales, that is, the scale of the diffusion processing module may be different from the scale of the target image, the coordinates of the face region in the target image cannot be directly used to determine the position of the face region in the first input.

[0232] For the nth diffusion processing module, the execution device needs to adaptively transform the coordinates of the face region according to the scale corresponding to the nth diffusion processing module, and then extract the sixth feature vector corresponding to the coordinates of the processed face region from the fourth feature vector.

[0233] For example, if the target image has a pixel resolution of 64x64, the coordinates of the face region are (10, 20) at the top left and (70, 80) at the bottom right, and the scale corresponding to the nth diffusion processing module is 32x32 (which also indicates that the scale of the convolutional layer is 32x32), then the execution device can process the coordinates of the face region based on the proportional relationship between the target image and the scale corresponding to the nth diffusion processing module. Specifically, the execution device will compress the top-left (10, 20) and bottom-right (70, 80) coordinates, resulting in the top-left (5, 105) and bottom-right (35, 40) coordinates of the face region. Then, the execution device can extract the sixth feature vector corresponding to the processed face region coordinates from the fourth feature vector.

[0234] In this embodiment of the application, the second target learning unit can perform feature learning on the sixth feature vector to obtain the seventh feature vector. This process is also the process of relearning the face region. The second target learning unit can learn the sixth feature vector based on an attention mechanism.

[0235] In one implementation, the second target learning unit can upsample the sixth feature vector and perform feature learning on the upsampled sixth feature vector to obtain an intermediate feature vector; then downsample the intermediate feature vector to the size of the sixth feature vector to obtain the seventh feature vector.

[0236] For example, if the matrix corresponding to the sixth eigenvector is a 7×8 matrix, which is a non-block matrix, the second objective learning unit can pad the matrix with zeros. The purpose of zero-padding is to convert the non-block matrix into a block matrix; this can be done by padding either the rows or columns. For example, the second objective learning unit can convert a 7×8 matrix into an 8×8 matrix. Furthermore, the second objective learning unit can also convert a 7×8 matrix into a 9×9 matrix or a 12×12 matrix, and so on.

[0237] Then, the second target learning unit can upsample the sixth feature vector after zero padding. The purpose of upsampling is to enlarge the face region. After enlarging, feature learning can learn more details, thereby achieving the purpose of enhancing the face region.

[0238] In this embodiment of the application, the second target learning unit can upsample the sixth feature vector to any size. For example, the second target learning unit can upsample the 8×8 matrix corresponding to the zero-padding sixth feature vector to a 16×16 matrix or to a 32×32 matrix, so as to achieve the purpose of amplifying the sixth feature vector.

[0239] Optionally, in this embodiment, the second target learning unit can upsample the sixth feature vector to the size of the matrix corresponding to the fourth feature vector. For example, if the size of the matrix corresponding to the fourth feature vector is 32×32, then the second target learning unit can upsample the 8×8 matrix corresponding to the sixth feature vector to the size of the matrix corresponding to the fourth feature vector, i.e., 32×32.

[0240] In this embodiment, the size of the matrix corresponding to the fourth feature vector is the same as the matrix processing size corresponding to the nth diffusion processing module. Therefore, expanding the sixth feature vector to the size of the matrix corresponding to the fourth feature vector before learning is more conducive to adapting to the computing power of the second target learning unit, thereby improving computing efficiency.

[0241] In this embodiment, the second target learning unit can perform feature learning on the upsampled sixth feature vector to obtain an intermediate feature vector; the size of the matrix corresponding to the intermediate feature vector is 32×32. The feature learning process does not change the size of the feature vector.

[0242] It's important to understand that the sixth eigenvector is composed of the feature vectors of the face region from the fourth eigenvector, representing the features of the face region. The size of the matrix corresponding to the sixth eigenvector represents the size of the face region within the fourth eigenvector. Accordingly, both the upsampled sixth eigenvector and the intermediate eigenvectors after relearning represent the features of the face region. Based on this, it's necessary to reduce the features of the face region to the size of the face region within the fourth eigenvector. Therefore, as... Figure 7 As shown, the second target learning unit can downsample the intermediate feature vector, that is, perform a reduction process, to obtain the seventh feature vector. The seventh feature vector is the result of reducing the intermediate feature vector, and it still represents the features of the face region. Since the seventh feature vector is the result after relearning, it can enrich the details of the face region and help improve the fidelity of the face region.

[0243] It should be noted that the matrix corresponding to the intermediate feature vector is a block matrix. In this embodiment, the seventh feature vector is also a block matrix. In this case, the seventh feature vector needs to be de-zeroed, that is, the part added in the previous zero-padding operation is removed to obtain a 7×8 matrix, which is the matrix corresponding to the seventh feature vector after de-zeroing.

[0244] In this embodiment, through continuous operations such as zero padding, upsampling, downsampling, and zero removal, the resulting seventh feature vector contains richer features of the face region, thereby helping to improve the fidelity of the face region.

[0245] In this embodiment of the application, the process of the second fusion unit fusing the seventh feature vector with the fifth feature vector can be implemented in the following ways:

[0246] In the first approach, the second fusion unit can directly replace the region corresponding to the face region coordinates in the fifth feature vector with the seventh feature vector.

[0247] In the second approach, the second fusion unit can calculate the coordinates of the face region in the seventh and fifth feature vectors to obtain the eighth feature vector.

[0248] Optionally, in this embodiment, the second fusion unit may fuse the seventh feature vector and the fifth feature vector in a weighted manner to obtain the eighth feature vector.

[0249] The second fusion unit can extract the feature vector corresponding to the coordinates of the face region from the fifth feature vector to obtain the tenth feature vector.

[0250] Furthermore, based on the preceding content, the seventh feature vector is a feature vector obtained after relearning the face region. Therefore, the size of the matrix corresponding to the seventh feature vector is the same as the size of the matrix corresponding to the tenth feature vector. On this basis, the second fusion unit can perform weighted calculations on the seventh and tenth feature vectors.

[0251] As shown in the following formula:

[0252]

[0253] in, This represents the tenth eigenvector. Represents the seventh eigenvector. Let α and β represent the features of the fused face region, respectively, and let α + β = 1. Generally, β is larger than α.

[0254] Among them, the second fusion unit obtains After that, Replace the face region in the fifth feature vector to obtain the eighth feature vector.

[0255] In this embodiment, the eighth feature vector is formed by weighted fusion, which improves the transition between the face region and the non-face region in the eighth feature vector.

[0256] In some embodiments of this application, the eighth feature vector can be directly determined as the output of the nth diffusion processing module.

[0257] In other cases, continue to combine Figure 7As can be seen, the convolutional layer can input the eighth feature vector into the sampling layer, and the execution device can call the sampling layer of the nth diffusion processing module (hereinafter referred to as the sampling layer) to sample the eighth feature vector and obtain the sampling result, which is the output of the nth diffusion processing module.

[0258] The sampling process can be either upsampling or downsampling, depending on the position of the nth diffusion processing module within the N cascaded diffusion processing modules.

[0259] Combination Figure 7 As can be seen in this embodiment, in the cross-attention layer, the first target learning unit and the cross-attention unit operate in parallel. The cross-attention unit performs the first learning, and the first target learning unit performs the second learning on the face region. The results of the two learnings are then fused and fed into the convolutional layer. In the convolutional layer, the second target learning unit and the convolutional unit operate in parallel. The convolutional unit performs the first learning, and the second target learning unit performs the second learning on the face region. The results of the two learnings are then fused to obtain the eighth feature vector. The output of the nth diffusion processing module is generated based on the eighth feature vector. Obviously, the face region in the eighth feature vector has better detail representation than the face region in the first feature vector, thus helping to improve the problem of face image distortion. In the N traversals, the detail representation of the face region is enhanced in this way each time, so the detail representation of the face region in the final target image is richer, thereby improving the fidelity of the face region in the target image and achieving higher consistency with the face region in the original face image.

[0260] Based on the above embodiments, please refer to Figure 9 As shown, Figure 9 A schematic diagram of another image generation model provided in an embodiment of this application is shown. This image generation model includes N cascaded diffusion processing modules, where each diffusion processing module includes one or more diffusion processing layers and a sampling layer.

[0261] When the diffusion processing module includes a diffusion processing layer, the diffusion processing layer includes a cross-attention layer and a convolutional layer. The cross-attention layer includes a cross-attention unit, a first target learning unit, and a first fusion unit, and the convolutional layer includes a convolutional unit, a second target learning unit, and a second fusion unit.

[0262] When the diffusion processing module includes multiple diffusion processing layers, at least one diffusion processing layer includes a cross-attention layer and a convolutional layer, and the remaining diffusion processing layers may include a cross-attention layer and / or a convolutional layer. The cross-attention layer includes a cross-attention unit, a first target learning unit and a first fusion unit, and the convolutional layer includes a convolutional unit or the convolutional layer includes a convolutional unit, a second target learning unit and a second fusion unit.

[0263] Different diffusion processing modules can process images with different resolutions (scales). However, the different diffusion processing layers within the same diffusion processing module correspond to the same scale.

[0264] In this embodiment, the number of diffusion processing layers included in different diffusion processing modules may vary. For example, the nth diffusion processing module may include m diffusion processing layers and one sampling layer, the (n+1)th diffusion processing module may include i diffusion processing layers and one sampling layer, and the (n+2)th diffusion processing module may include j diffusion processing layers and one sampling layer, where m, n, and j are positive integers greater than 1. This embodiment does not limit the number of diffusion processing layers included in a diffusion processing module.

[0265] Combination Figure 9 and Figure 10 The implementation process of the image generation method provided in the embodiments of this application will be described. Figure 10 The diagram shows a flowchart of an image generation method provided in an embodiment of this application. It is assumed that the nth diffusion processing module includes three diffusion processing layers and one sampling layer. The method includes steps 1001 to 1005.

[0266] Step 1001: Obtain the feature vector of the face region in the original face image and the descriptive feature vector of the descriptive information corresponding to the target image.

[0267] Step 1002: Call the first diffusion processing layer of the nth diffusion processing module to process the feature vector, descriptive feature vector and second input of the face region to obtain the output of the first diffusion processing layer.

[0268] When n iterates to 1, the second input is either a random noise vector or the output of the previous diffusion processing layer. As n iterates from 2 to N, the second input is either the output of the (n-1)th diffusion processing module or the output of the previous diffusion processing layer.

[0269] When the diffusion processing layer is the first diffusion processing layer in the nth diffusion processing module, the second input is a random noise vector or the output of the (n-1)th diffusion processing module.

[0270] When the diffusion processing layer is the second or subsequent diffusion processing layer in the nth diffusion processing module, the second input is the output of the previous diffusion processing layer.

[0271] In this embodiment of the application, for example, the first diffusion processing layer includes a cross-attention layer and a convolutional layer, wherein the cross-attention layer includes a cross-attention unit, a first target learning unit and a first fusion unit, and the convolutional layer includes a convolutional unit, a second target learning unit and a second fusion unit.

[0272] The execution device can call the cross attention unit in the cross attention layer of the first diffusion processing layer in the nth diffusion processing module to perform cross attention processing on the feature vector of the face region, the descriptive feature vector and the second input to obtain the A1 feature vector.

[0273] The execution device can call the first target learning unit in the cross-attention layer of the first diffusion processing layer in the nth diffusion processing module to perform feature learning on the A2 feature vector extracted from the second input, and obtain the A3 feature vector.

[0274] The execution device can call the first fusion unit in the cross-attention layer of the first diffusion processing layer in the nth diffusion processing module to fuse the A3 feature vector with the A1 feature vector to obtain the A4 feature vector.

[0275] Then, the execution device can call the convolution unit in the convolution layer of the first diffusion processing layer in the nth diffusion processing module to perform convolution processing on the A4 feature vector to obtain the A5 feature vector.

[0276] The execution device can call the second target learning unit in the convolutional layer of the first diffusion processing layer in the nth diffusion processing module to extract the A6th feature vector from the A4th feature vector according to the coordinates of the face region, and perform feature learning on the A6th feature vector to obtain the A7th feature vector.

[0277] The execution device can call the second fusion unit in the convolutional layer of the first diffusion processing layer in the nth diffusion processing module to fuse the A7 feature vector with the A5 feature vector to obtain the A8 feature vector, and generate the output of the first diffusion processing layer in the nth diffusion processing module based on the A8 feature vector.

[0278] Step 1003: Call the second diffusion processing layer of the nth diffusion processing module to process the feature vector, descriptive feature vector and second input of the face region to obtain the output of the second diffusion processing layer.

[0279] The second input is the output of the first diffusion processing layer.

[0280] In this embodiment of the application, for example, the second diffusion processing layer includes a cross attention layer and a convolutional layer, wherein the cross attention layer includes a cross attention unit, a first target learning unit and a first fusion unit, and the convolutional layer includes a convolutional unit.

[0281] The execution device can call the cross attention unit in the cross attention layer of the second diffusion processing layer in the nth diffusion processing module to perform cross attention processing on the feature vector, descriptive feature vector and output of the first diffusion processing layer of the face region to obtain the A9th feature vector.

[0282] The execution device can call the first target learning unit in the cross-attention layer of the second diffusion processing layer in the nth diffusion processing module to extract the A10 feature vector from the output of the first diffusion processing layer according to the coordinates of the face region, and perform feature learning on the A10 feature vector to obtain the A11 feature vector.

[0283] The execution device can call the first fusion unit in the cross-attention layer of the second diffusion processing layer in the nth diffusion processing module to fuse the A11 feature vector with the A9 feature vector to obtain the A12 feature vector.

[0284] Then, the execution device can call the convolution unit in the convolution layer of the second diffusion processing layer in the nth diffusion processing module to perform convolution processing on the A12 feature vector to obtain the A13 feature vector, and generate the output of the second diffusion processing layer in the nth diffusion processing module based on the A13 feature vector.

[0285] Step 1004: Call the third diffusion processing layer of the nth diffusion processing module to process the feature vector, descriptive feature vector and second input of the face region to obtain the output of the third diffusion processing layer.

[0286] The second input is the output of the second diffusion processing layer.

[0287] In this embodiment of the application, for example, the third diffusion processing layer includes a cross attention layer and a convolutional layer, wherein the cross attention layer includes a cross attention unit, a first target learning unit and a first fusion unit, and the convolutional layer includes a convolutional unit, a second target learning unit and a second fusion unit.

[0288] The execution device can call the cross attention unit in the cross attention layer of the third diffusion processing layer in the nth diffusion processing module to perform cross attention processing on the feature vector of the face region, the descriptive feature vector, and the output of the second diffusion processing layer to obtain the A14th feature vector.

[0289] The execution device can call the first target learning unit in the cross-attention layer of the third diffusion processing layer in the nth diffusion processing module to extract the A15th feature vector from the output of the second diffusion processing layer according to the coordinates of the face region, and perform feature learning on the A15th feature vector to obtain the A16th feature vector.

[0290] The execution device can call the first fusion unit in the cross-attention layer of the third diffusion processing layer in the nth diffusion processing module to fuse the A16th feature vector with the A14th feature vector to obtain the A17th feature vector.

[0291] Then, the execution device can call the convolution unit in the convolution layer of the third diffusion processing layer in the nth diffusion processing module to perform convolution processing on the A17 feature vector to obtain the A18 feature vector.

[0292] The execution device can call the second target learning unit in the convolutional layer of the third diffusion processing layer in the nth diffusion processing module to extract the A19th feature vector from the A17th feature vector according to the coordinates of the face region, and perform feature learning on the A19th feature vector to obtain the A20th feature vector.

[0293] The execution device can call the second fusion unit in the convolutional layer of the third diffusion processing layer in the nth diffusion processing module to fuse the A20 feature vector with the A18 feature vector to obtain the A21 feature vector, and generate the output of the third diffusion processing layer in the nth diffusion processing module based on the A21 feature vector.

[0294] Step 1005: Call the sampling layer of the nth diffusion processing module to sample the output of the 3rd diffusion processing layer to obtain the output of the nth diffusion processing module.

[0295] In this embodiment, the image generation model includes N diffusion processing modules, each including one or more diffusion processing layers. Each diffusion processing layer includes a cross-attention layer and a convolutional layer. The cross-attention layer includes a cross-attention unit, a first target learning unit, and a first fusion unit. The convolutional layer includes a convolutional unit, a second target learning unit, and a second fusion unit. The first and second target learning units can relearn the face region, thereby enhancing the facial region feature representation during the operation of each diffusion processing layer. Thus, the combined effect of the N diffusion processing modules and the combined effect of multiple diffusion processing layers within each module enriches the detail representation of the face region in the target image, thereby improving the fidelity of the face region in the target image and increasing its consistency with the face region in the original face image.

[0296] It should be noted that, Figure 9 The structure of the image generation model shown is as follows Figure 10 The image generation method shown is merely an illustrative example and does not constitute a limitation on the image generation model and image generation method in the embodiments of this application.

[0297] refer to Figure 11 , Figure 11 This is a flowchart illustrating a training method for an image generation model provided in an embodiment of this application. The training method for the image generation model can be executed by a server or by a terminal in cooperation with the server. The training method for the image generation model includes, but is not limited to, the following steps 1101 to 1103.

[0298] Step 1101: Obtain noise samples and original face image samples, and input the noise samples and original face image samples into the initial image generation model.

[0299] The initial image generation model includes T U-net networks, each of which includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. The number of diffusion processing modules included in different U-net networks may vary.

[0300] It should be noted that in this embodiment, traversing all N diffusion processing modules of a U-net network once signifies the completion of one denoising process from random noise to the target image. This process needs to be repeated T times to complete the denoising process from random noise to the target image and obtain the final target image. Since the processing of each U-net network is repetitive, this embodiment only describes the processing of the N diffusion processing modules in one U-net network, and the processing of the remaining U-net networks will not be elaborated upon.

[0301] In this embodiment of the application, the diffusion processing module includes a cross-attention layer, which includes a cross-attention unit, a first target learning unit, and a first fusion unit. The outputs of the cross-attention unit and the first target learning unit are the inputs of the first fusion unit.

[0302] The original face image sample is similar to the original face image, except that it is a photograph or image containing a face used in the training process for subsequent image processing and synthesis tasks. The original face image sample can be sent from the terminal to the server.

[0303] In the training method of image generation models, the original face image sample is the input in the first step. Through encoding and processing, feature vectors of the face regions in the original face image sample are extracted. These feature vectors can be used for subsequent training tasks. The original face image sample can be sent from the terminal to the server.

[0304] In this embodiment of the application, after obtaining the original face image sample, it is necessary to perform a first encoding on the original face image sample. The first encoding process is the same as the first encoding of the original face image in the above embodiment, and will not be described again here.

[0305] In this embodiment of the application, after obtaining the original face image sample, it is also necessary to add noise to the original face image sample to obtain a noise sample, and then extract the noise feature vector by encoding and processing the noise sample.

[0306] In this embodiment of the application, during the training process, the noise samples are obtained by adding noise to the original face image samples, and are not random noise.

[0307] Step 1102: For the nth diffusion processing module, freeze the parameters of the cross-attention unit, and perform cross-attention processing on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention layer to obtain the first feature vector. Based on the parameters of the first target learning unit, extract the second feature vector from the first input according to the coordinates of the face region, perform feature learning on the second feature vector to obtain the third feature vector; based on the parameters of the first fusion unit, fuse the third feature vector with the first feature vector to obtain the fourth feature vector, and generate the output of the nth diffusion processing module based on the fourth feature vector.

[0308] Among them, the feature vector of the face region is the feature vector obtained based on the face region in the original face image sample.

[0309] In this embodiment, after obtaining the feature vector and noise feature vector of the face region, the model is trained by traversing N diffusion processing modules. The training process of each diffusion processing module is similar. Taking the nth diffusion processing module as an example:

[0310] The parameters of the cross-attention units in the cross-attention layer of the nth diffusion processing module are frozen. Then, the nth diffusion processing module is called to perform cross-attention processing on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention layer, thus obtaining the first feature vector. Here, n traverses from 1 to N. When n traverses to 1, the first input is a noise feature vector, which is a feature vector obtained based on noise samples. During the process of n traversing from 2 to N, the first input is the output of the (n-1)th diffusion processing module.

[0311] The process of cross-attention processing by the cross-attention unit is similar to that in the above embodiments, and will not be described in detail here.

[0312] Simultaneously, the first target learning unit can extract a second feature vector from the first input based on the coordinates of the face region, and perform feature learning on the second feature vector to obtain a third feature vector. Here, the coordinates of the face region represent the location of the face region in the target image to be generated.

[0313] This process runs in parallel with the cross-attention processing performed by the cross-attention unit. The specific implementation of this process can be found in the above embodiments and will not be elaborated upon here.

[0314] In this embodiment of the application, since different diffusion processing modules correspond to different scales, that is, the scale corresponding to the diffusion processing module may be different from the scale of the target image, in this case, the coordinates of the face region in the target image cannot be directly used to determine the position of the face region in the first input.

[0315] For the nth diffusion processing module, the coordinates of the face region are enlarged or reduced according to the proportional relationship between the target image and the scale corresponding to the nth diffusion processing module, so that the coordinates of the face region after processing match the scale corresponding to the nth diffusion processing module. Then, the second feature vector corresponding to the coordinates of the face region after processing is extracted from the first input.

[0316] In this embodiment of the application, after obtaining the third feature vector and the first feature vector, it is necessary to fuse the third feature vector and the first feature vector. The fusion process is similar to the fusion process disclosed in the above embodiments, and will not be described in detail here.

[0317] It should be noted that the cross-attention unit is pre-trained and may have already learned effective feature extraction and representation capabilities during the training process. This application introduces a first target learning unit and a first fusion unit without changing the structure of the cross-attention unit. By freezing at least some of its parameters, these effective features can be retained in subsequent joint training, achieving a fine-tuning-free effect. This avoids destroying these learned features when training the first target learning unit and the first fusion unit, reducing the risk of overfitting. This helps ensure that the model learns feature representations with better generalization performance during training and is conducive to improving training efficiency.

[0318] In another implementation, the process of the nth diffusion processing module obtaining the first feature vector may further include: obtaining a descriptive feature vector and inputting the descriptive feature vector into the initial image generation model. Then, based on the parameters of the frozen cross-attention unit, cross-attention processing is performed on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector.

[0319] Among them, the descriptive feature vector is the feature vector of the descriptive information corresponding to the target image. The descriptive information is used to describe the target image from different dimensions.

[0320] Optionally, in this embodiment of the application, the process of performing cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input based on the parameters of the frozen cross-attention unit to obtain the first feature vector may further include: performing cross-attention processing on the feature vector of the face region and the first input based on the parameters of the frozen cross-attention unit to obtain the first cross-attention feature vector; performing cross-attention processing on the descriptive feature vector and the first input to obtain the second cross-attention feature vector; and fusing the first cross-attention feature vector and the second cross-attention feature vector to obtain the first feature vector.

[0321] Optionally, in this embodiment, the first target learning unit is specifically used to upsample the second feature vector, and to perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector; and to downsample the intermediate feature vector to the size of the second feature vector to obtain a third feature vector.

[0322] Optionally, the first target learning unit is specifically used to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

[0323] Optionally, the first fusion unit is specifically used to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

[0324] Optionally, the first fusion unit is specifically used to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

[0325] Step 1103: Calculate the loss based on the first feature vector and the target map to obtain the cross-entropy loss, and adjust the parameters of the first target learning unit and the first fusion unit according to the cross-entropy loss.

[0326] The target image is a zero-based image of the same size as the first feature vector, and has true values ​​in the face region.

[0327] In this embodiment, a face mask loss is added to the original loss during the loss calculation stage. The server can calculate the cross-entropy loss immediately after the nth diffusion processing module completes its execution. Alternatively, the server can calculate the cross-entropy loss during the loss calculation stage after traversing all diffusion processing modules.

[0328] Specifically, a cross-attention map is built based on the first feature vector, and then a target map is built. The target map is a zero map with the same size as the cross-attention map, where the face region in the zero map is the ground truth.

[0329] Please refer to Figure 12 As shown, Figure 12 A schematic diagram of the cross-entropy loss provided in an embodiment of this application is shown, wherein, Figure 12 The diagram shows the cross-attention map and target map corresponding to the first feature vector. The white areas in the target map represent face regions, and the target values ​​for these face regions are determined based on a two-dimensional Gaussian function. The cross-entropy loss can be calculated from the cross-attention map and the target map.

[0330] The target image includes target values. In non-face regions, the target values ​​are all set to 0. In face regions, the target values ​​are set based on a two-dimensional Gaussian function with the center of the face region as the maximum value of 1. The formula is as follows:

[0331]

[0332] Where x and y represent the horizontal and vertical coordinate values.

[0333] It should be noted that since different diffusion processing modules correspond to different scales, the sizes of the cross-attention maps corresponding to different diffusion processing modules are also different. In other words, for different diffusion processing modules, it is necessary to set a target map of a corresponding size so that it can match the first feature vector of that diffusion processing module.

[0334] Furthermore, for the nth diffusion processing module, based on the scale ratio between the target image and the nth diffusion processing module, the coordinates of the face region are magnified or reduced to match the scale of the nth diffusion processing module. Then, the face region in the target image is determined based on the coordinates of the processed face region. Figure 12 The white area in the diagram is shown, and a target value is set in the face region based on a two-dimensional Gaussian function with the center of the face region as the maximum value of 1, where the target value of the face region is the true value.

[0335] In this embodiment of the application, the process of calculating cross-entropy loss includes: for the nth diffusion processing module, calculating cross-entropy loss based on the first feature vector and the target map to obtain the loss result.

[0336] Taking the nth diffusion processing module with a scale of 32x32 as an example, a corresponding cross-attention map is established based on the first feature vector. A target map is established based on the size of the cross-attention map and the coordinates of the face region. The cross-entropy loss is calculated by comparing the target value in the 32x32 cross-attention map with the target value in the target map corresponding to the nth diffusion processing module. The formula is as follows:

[0337] loss facemask =CE(Annt) (32) ,target (32) )

[0338] Where CE represents the cross-entropy loss, and Annt (32) This represents a cross-attention map with a resolution of 32x32, and the target... (32) This represents a target image with a resolution of 32x32.

[0339] In one implementation, refer to the above embodiments. Figure 9 The image generation model shown includes a diffusion processing module comprising multiple diffusion processing layers, each of which includes a cross-attention layer and a convolutional layer. This means that during training, each diffusion processing module outputs multiple first feature vectors (i.e., multiple cross-attention maps). Based on this, the process of calculating the cross-entropy loss in this embodiment can be shown by the following formula:

[0340]

[0341] Where CE represents the cross-entropy loss, Let j represent the cross-attention map with a resolution of 32x32. Let j represent the target image with a resolution of 32x32, and n2 represent the total number of cross-attention maps, which is the number of diffusion processing layers included in the nth diffusion processing module.

[0342] In this embodiment of the application, after calculating the loss facemask Next, the average value can be calculated to obtain the loss result for this part. Then, the parameters of the first target learning unit and the first fusion unit are adjusted based on the loss result.

[0343] In one implementation, the server can adjust the parameters of the first target learning unit and the first fusion unit included in the nth diffusion processing module based on the loss result corresponding to the nth diffusion processing module.

[0344] In another implementation, the server can calculate the cross-entropy loss for each diffusion processing module, then average the total cross-entropy losses, and then adjust the parameters of the first target learning unit and the first fusion unit of each diffusion processing module based on the average value.

[0345] In another implementation, the server can select a portion of the diffusion processing modules from multiple diffusion processing modules, calculate the cross-entropy loss, and then adjust the parameters of the first target learning unit and the first fusion unit of each diffusion processing module based on the obtained cross-entropy loss.

[0346] In the embodiments of this application, cross-attention maps with large or small scales are generally avoided in the process of calculating cross-entropy loss.

[0347] For example, for a cross-attention map with a resolution of 8x8, the face region will be even smaller due to the small size of the cross-attention map itself. In this case, the cross-entropy loss of the face region calculated is of little reference value. Therefore, in order to reduce the computational cost during model training, it is better to avoid choosing a cross-attention map with a small scale to calculate the cross-entropy loss.

[0348] For example, the image generation model includes seven diffusion processing modules, each with a different scale. For instance, the first diffusion processing module corresponds to a scale of 64x64, the second to 32x32, the third to 16x16, the fourth to 8x8, the fifth to 16x16, the sixth to 32x32, and the seventh to 64x64.

[0349] The image generation model can include multiple U-net networks, and each U-net network includes multiple diffusion processing modules, each diffusion processing module includes multiple diffusion processing layers, and each diffusion processing layer includes a cross-attention layer. In other words, multiple first feature vectors (i.e. multiple cross-attention maps) can be obtained during training.

[0350] Based on this, in the embodiments of this application, cross-attention maps with resolutions of 32x32 and 16x16 can be selected to calculate the cross-entropy loss, as shown in the following formula:

[0351]

[0352] Where CE represents the cross-entropy loss, Let i represent the i-th cross-attention map with a resolution of 16x16. Let j represent the cross-attention map with a resolution of 32x32. This represents the i-th target image with a resolution of 16x16. Let n represent the j-th target image with a resolution of 32x32, n1 represent the total number of cross-attention maps with a resolution of 32x32, and n2 represent the total number of cross-attention maps with a resolution of 16x16.

[0353] In this embodiment of the application, after calculating the loss facemask Afterwards, the average value can be calculated to obtain the loss result, and then the parameters of the first target learning unit and the first fusion unit of each diffusion processing module can be adjusted according to the loss result.

[0354] In this embodiment of the application, by setting a ground value constraint on the face region in the target image to determine the position of the face region in the generated result, since the feature intensity is inconsistent within the face region, the use of a two-dimensional Gaussian function can better simulate the intensity of a real face, thereby controlling the face region in the generated target image to a predetermined position with a higher probability.

[0355] Based on the above embodiments, please refer to Figure 13 , Figure 13 The diagram illustrates a flowchart of another image generation model training method provided in this application embodiment. The image generation model training method can be executed by a server or by a terminal in cooperation with the server. The image generation model training method includes, but is not limited to, the following steps 1301 to 1304.

[0356] Step 1301: Obtain noise samples and original face image samples, and input the noise samples and original face image samples into the initial image generation model.

[0357] The initial image generation model includes T U-net networks, each of which includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. The number of diffusion processing modules included in different U-net networks may vary.

[0358] It should be noted that in this embodiment, traversing all N diffusion processing modules of a U-net network once signifies the completion of one denoising process from random noise to the target image. This process needs to be repeated T times to complete the denoising process from random noise to the target image and obtain the final target image. Since the processing of each U-net network is repetitive, this embodiment only describes the processing of the N diffusion processing modules in one U-net network, and the processing of the remaining U-net networks will not be elaborated upon.

[0359] In this embodiment of the application, the diffusion processing module includes a cross-attention layer, which includes a cross-attention unit, a first target learning unit, and a first fusion unit. The outputs of the cross-attention unit and the first target learning unit are the inputs of the first fusion unit.

[0360] In this embodiment of the application, the diffusion processing module further includes a convolutional layer, which includes a convolutional unit, a second target learning unit, and a second fusion unit. The input of the convolutional unit is the output of the cross-attention layer, and the output of the convolutional unit and the output of the second target learning unit are the inputs of the second fusion unit.

[0361] Step 1302: For the nth diffusion processing module, freeze the parameters of the cross-attention unit, and perform cross-attention processing on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention layer to obtain the first feature vector. Based on the parameters of the first target learning unit, extract the second feature vector from the first input according to the coordinates of the face region, perform feature learning on the second feature vector to obtain the third feature vector; based on the parameters of the first fusion unit, fuse the third feature vector with the first feature vector to obtain the fourth feature vector.

[0362] Step 1303: For the nth diffusion processing module, freeze the parameters of the convolutional unit, perform convolution processing on the fourth feature vector based on the frozen parameters of the convolutional unit to obtain the fifth feature vector; based on the parameters of the second target learning unit, extract the sixth feature vector from the fourth feature vector according to the coordinates of the face region, perform feature learning on the sixth feature vector to obtain the seventh feature vector; based on the parameters of the second fusion unit, fuse the seventh feature vector with the fifth feature vector to obtain the eighth feature vector, and generate the output of the nth diffusion processing module based on the eighth feature vector.

[0363] Step 1304: Calculate the loss based on the first feature vector and the target map to obtain the cross-entropy loss, and adjust the parameters of the first target learning unit, the first fusion unit, the second target learning unit, and the second fusion unit according to the cross-entropy loss.

[0364] In this embodiment, the convolutional units in the convolutional layer are pre-trained and may have already learned effective feature extraction and representation capabilities during training. The second target learning unit and the second fusion unit introduced in this embodiment do not change the structure of the convolutional units. By freezing the parameters of the convolutional units, these effective features can be retained in subsequent training, achieving a no-fine-tuning effect. This avoids destroying these already learned features when training the second target learning unit and the second fusion unit, reducing the risk of overfitting. This helps ensure that the model learns feature representations with better generalization performance during training and is beneficial to improving training efficiency.

[0365] For the nth diffusion processing module, during model training, the parameters of the cross-attention units in the cross-attention layer and the parameters of the convolutional units in the convolutional layer are frozen. Then, the first target learning unit, the first fusion unit, the second target learning unit, and the second fusion unit are jointly trained.

[0366] In this application embodiment, the process of calculating cross-entropy loss can refer to the content disclosed in the above embodiments, and will not be repeated here.

[0367] In this embodiment, compared to the model training method disclosed in the above embodiments, the difference lies in the addition of a convolutional layer during training. In this convolutional layer, the second target learning unit runs in parallel with the convolutional unit. During model training, the parameters of the convolutional unit are frozen, and the first learning is performed based on the frozen parameters of the convolutional unit. Then, the face region is relearned through the second target learning unit. The results of the two learnings are then fused to obtain the eighth feature vector. Obviously, the face region in the eighth feature vector has better detail representation than the face region in the first feature vector, thereby helping to improve the problem of facial image distortion.

[0368] Furthermore, the eighth feature vector is used to determine the output of the nth diffusion processing module, which is the first input corresponding to the (n+1)th diffusion processing module. Thus, the first feature vector determined by the (n+1)th diffusion processing module based on the first input includes the influence of the first target learning unit, the first fusion unit, the second target learning unit, and the second fusion unit in the nth diffusion processing module.

[0369] Therefore, when calculating the cross-entropy loss, the first feature vector corresponding to the nth diffusion processing module already includes the influence of the first target learning unit, the first fusion unit, the second target learning unit, and the second fusion unit in the previous diffusion processing module (i.e., the (n-1)th diffusion processing module). Thus, based on the cross-entropy loss, the parameters of the first target learning unit, the first fusion unit, the second target learning unit, and the second fusion unit can be adjusted.

[0370] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0371] The above combination Figures 1 to 13 The image generation method, image generation model, and image generation model training method provided in the embodiments of this application have been described. The apparatus for executing the above methods provided in the embodiments of this application is described below. Figure 14 As shown, Figure 14 This illustration shows an optional structural diagram of the image generation apparatus provided in an embodiment of this application. The image generation apparatus may be the execution device in the embodiments of this application, or a chip or chip system within the execution device. The image generation device 1400 includes an acquisition unit 1401 and a processing unit 1402. The acquisition unit 1401 is used to acquire the feature vector of the face region of the original face image. The processing unit 1402 is used to: call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first feature vector, where n traverses from 1 to N. When n traverses to 1, the first input is a random noise vector. During the process of n traversing from 2 to N, the first input is the output of the (n-1)th diffusion processing module; call the nth diffusion processing module to extract the second feature vector from the first input according to the coordinates of the face region, perform feature learning on the second feature vector to obtain the third feature vector; the coordinates of the face region are the position of the face region in the target image to be generated; call the nth diffusion processing module to fuse the third feature vector with the first feature vector to obtain the fourth feature vector, generate the output of the nth diffusion processing module according to the fourth feature vector, and determine the target image according to the output of the nth diffusion processing module when n traverses to N.

[0372] Optionally, as an embodiment, the acquisition unit 1401 is further used to acquire the description feature vector of the description information corresponding to the target image; wherein, the description information corresponding to the target image is used to describe the target image from different dimensions; the processing unit 1402 is specifically used to call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region, the description feature vector and the first input to obtain the first feature vector.

[0373] Optionally, as an embodiment, the processing unit 1402 is further configured to: call the nth diffusion processing module to perform convolution processing on the fourth feature vector to obtain the fifth feature vector; call the nth diffusion processing module to extract the sixth feature vector from the fourth feature vector according to the coordinates of the face region, perform feature learning on the sixth feature vector to obtain the seventh feature vector; call the nth diffusion processing module to fuse the seventh feature vector with the fifth feature vector to obtain the eighth feature vector, and generate the output of the nth diffusion processing module according to the eighth feature vector.

[0374] Optionally, as an embodiment, the processing unit 1402 is specifically used to call the nth diffusion processing module to perform upsampling processing on the second feature vector, and to perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector; and to downsample the intermediate feature vector to the size of the second feature vector to obtain a third feature vector.

[0375] Optionally, as an embodiment, the processing unit 1402 is specifically used to call the nth diffusion processing module to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

[0376] Optionally, as an embodiment, the processing unit 1402 is specifically used to call the nth diffusion processing module to fuse the third feature vector and the first feature vector in a weighted manner to obtain the fourth feature vector.

[0377] Optionally, as an embodiment, the processing unit 1402 is specifically used to call the nth diffusion processing module to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

[0378] Optionally, as an embodiment, the processing unit 1402 is specifically used to call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and the first input to obtain a first cross-attention feature vector; to perform cross-attention processing on the descriptive feature vector and the first input to obtain a second cross-attention feature vector; and to fuse the first cross-attention feature vector and the second cross-attention feature vector to obtain a first feature vector.

[0379] In summary, the image generation apparatus 1400 in this embodiment learns the face region and its feature vector from the first input through cross-attention processing. Then, the face region from the first input is learned again by the first target learning unit. This second learning yields richer feature information about the face region. Based on this, the third feature vector obtained from the second learning is fused into the first feature vector by the first fusion unit to obtain a fourth feature vector. The face region in the fourth feature vector has better detail representation than the face region in the first feature vector, thus helping to improve the problem of facial image distortion. During the N traversals, the detail representation of the face region is enhanced in this way each time, resulting in richer detail representation of the face region in the final target image, thereby improving the fidelity of the face region in the target image and achieving higher consistency with the face region in the original face image.

[0380] Reference Figure 15 , Figure 15This illustration shows an optional structural diagram of a training apparatus for an image generation model provided in an embodiment of this application. The training apparatus 1500 includes an acquisition unit 1501 and a processing unit 1502. The acquisition unit 1501 acquires noise samples and original face image samples, where the noise samples are obtained by adding noise to the original face image samples. The processing unit 1502 inputs the noise samples and original face image samples into an initial image generation model. The initial image generation model includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. Each diffusion processing module includes a cross-attention layer, which includes a cross-attention unit, a first target learning unit, and a first fusion unit. The outputs of the cross-attention unit and the first target learning unit are the inputs of the first fusion unit. For the nth diffusion processing module, the parameters of the cross-attention unit are frozen. Based on the frozen parameters of the cross-attention unit, cross-attention processing is performed on the feature vector of the face region and the first input to obtain a first feature vector. The feature vector of the face region is based on the original face image. The feature vector obtained from the face region in the image sample is processed by iterating n from 1 to N. When n reaches 1, the first input is a noise feature vector, which is a feature vector obtained based on the noise sample. As n iterates from 2 to N, the first input is the output of the (n-1)th diffusion processing module. The cross-entropy loss is calculated based on the first feature vector and the target image. The parameters of the first target learning unit and the first fusion unit are adjusted according to the cross-entropy loss. The target image is a zero-based image with the same size as the first feature vector and the ground truth value in the face region. The first target learning unit is used to extract the second feature vector from the first input based on the coordinates of the face region. The second feature vector is then used for feature learning to obtain the third feature vector. The coordinates of the face region are the position of the face region in the target image to be generated. The first fusion unit is used to fuse the third feature vector with the first feature vector to obtain the fourth feature vector. The output of the nth diffusion processing module is generated based on the fourth feature vector. When n iterates to N, the target image corresponding to the original face image is determined based on the output of the nth diffusion processing module.

[0381] Optionally, as an embodiment, the acquisition unit 1501 is further configured to acquire a descriptive feature vector, wherein the descriptive feature vector is a feature vector of the descriptive information corresponding to the target image, and the descriptive information is used to describe the target image from different dimensions; the processing unit 1502 is further configured to input the descriptive feature vector into the initial image generation model, and for the nth diffusion processing module, freeze the parameters of the cross-attention unit, and perform cross-attention processing on the feature vector of the face region, the descriptive feature vector and the first input based on the frozen parameters of the cross-attention unit to obtain the first feature vector.

[0382] Optionally, as an embodiment, the diffusion processing module further includes a convolutional layer, which includes a convolutional unit, a second target learning unit, and a second fusion unit. The input of the convolutional unit is the output of the cross-attention layer, and the output of the convolutional unit and the output of the second target learning unit are the input of the second fusion unit. The processing unit 1502 is also used to freeze the parameters of the convolutional unit, perform convolution processing on the fourth feature vector based on the frozen parameters of the convolutional unit to obtain the fifth feature vector; extract the sixth feature vector from the fourth feature vector based on the coordinates of the face region based on the parameters of the second target learning unit, perform feature learning on the sixth feature vector to obtain the seventh feature vector; fuse the seventh feature vector with the fifth feature vector based on the parameters of the second fusion unit to obtain the eighth feature vector, generate the output of the nth diffusion processing module based on the eighth feature vector; and adjust the parameters of the second target learning unit and the second fusion unit based on the cross-entropy loss.

[0383] Optionally, as an embodiment, the processing unit 1502 is specifically used to perform upsampling processing on the second feature vector, and to perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector; and to downsample the intermediate feature vector to the size of the second feature vector to obtain a third feature vector.

[0384] Optionally, as an embodiment, the processing unit 1502 is specifically used to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

[0385] Optionally, as an embodiment, the processing unit 1502 is specifically used to fuse the third feature vector and the first feature vector in a weighted manner to obtain the fourth feature vector.

[0386] Optionally, as an embodiment, the processing unit 1502 is specifically used to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector and the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

[0387] Optionally, as an embodiment, the processing unit 1502 is specifically used to perform cross-attention processing on the feature vector of the face region and the first input based on the parameters of the frozen cross-attention unit to obtain a first cross-attention feature vector; perform cross-attention processing on the descriptive feature vector and the first input to obtain a second cross-attention feature vector; and fuse the first cross-attention feature vector and the second cross-attention feature vector to obtain a first feature vector.

[0388] In summary, the model training device 1500 for the image generation model in this application embodiment freezes at least a portion of the parameters of the diffusion model during model training, and sets ground value constraints on the position of the face region in the generated result by setting the face region in the target image. Since the feature intensity is inconsistent in the face region, the use of a two-dimensional Gaussian function can better simulate the intensity of the real face, thereby controlling the face region in the generated target image to a predetermined position with a higher probability.

[0389] It should be noted that the aforementioned devices 1400 and 1500 are embodied in the form of functional units. The term "unit" here can be implemented in software and / or hardware, without specific limitation. For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include application-specific integrated circuits (ASICs), electronic circuits, a processor (e.g., a shared processor, a proprietary processor, or a group processor, etc.) and memory for executing one or more software or firmware programs, combined logic circuitry, and / or other suitable components supporting the described functions.

[0390] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0391] Figure 16 A schematic diagram of the hardware structure of an image generation apparatus provided in an embodiment of this application is shown. Figure 16 The image generation apparatus 1600 shown (which may specifically be a terminal or a server) includes a memory 1601, a processor 1602, a communication interface 1603, and a bus 1604. The memory 1601, processor 1602, and communication interface 1603 are interconnected via the bus 1604.

[0392] The memory 1601 may be a ROM, a static storage device, a dynamic storage device, or RAM. The memory 1601 may store a program, and when the program stored in the memory 1601 is executed by the processor 1602, the processor 1602 and the communication interface 1603 are used to execute the various steps of the image generation apparatus of the embodiments of this application.

[0393] The processor 1602 may be a general-purpose CPU, microprocessor, ASIC, GPU, or one or more integrated circuits, used to execute relevant programs to achieve the functions required by the units in the image generation apparatus of this application embodiment, or to execute the image generation apparatus of the method embodiment of this application.

[0394] The processor 1602 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the image generation apparatus of this application can be completed by the integrated logic circuitry in the hardware of the processor 1602 or by instructions in software form. The processor 1602 can also be a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1601. The processor 1602 reads the information in memory 1601 and, in conjunction with its hardware, completes the functions required by the units included in the image generation apparatus of this application, or executes the image generation apparatus of the method embodiments of this application.

[0395] The communication interface 1603 uses a transceiver device, such as, but not limited to, a transceiver, to enable communication between the device 1600 and other devices or communication networks. For example, input data (such as the original face image in the embodiments of this application) can be acquired through the communication interface 1603.

[0396] Bus 1604 may include a pathway for transmitting information between various components of device 1600 (e.g., memory 1601, processor 1602, communication interface 1603).

[0397] It should be understood that the acquisition unit 1401 in the image generating apparatus 1400 is equivalent to the communication interface 1603 in the image generating apparatus 1600; the processing unit 1402 in the image generating apparatus 1400 can be equivalent to the processor 1601.

[0398] Figure 17 This illustration shows a schematic diagram of the hardware structure of a training device for an image generation model provided in an embodiment of this application. Figure 17The training device 1700 for the image generation model shown (specifically, the device 1700 can be a terminal or a server) includes one or more processors 1701, a memory 1702, a communication interface 1703, and a bus 1704. The memory 1702, processor 1701, and communication interface 1703 are interconnected via the bus 1704.

[0399] The memory 1702 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1702 may store a program, and when the program stored in the memory 1702 is executed by the processor 1701, the processor 1701 and the communication interface 1703 are used to execute the various steps of the training method of the image generation model in the embodiments of this application.

[0400] The processor 1701 may be a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), graphics processor, or one or more integrated circuits, used to execute relevant programs to implement the functions required by the units in the image generation model training device of the present application embodiment, or to execute the image generation model training method of the method embodiment of the present application.

[0401] The processor 1701 can also be an integrated circuit chip with signal processing capabilities. In implementation, each step of the training method for the image generation model of this application can be completed through the integrated logic circuitry in the processor 1701 or through software instructions.

[0402] The processor 1701 described above can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly implemented by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 1702. The processor 1701 reads the information in memory 1702 and, in conjunction with its hardware, completes the functions required by the units included in the image generation model training device of the embodiments of this application, or executes the image generation model training method of the method embodiments of this application.

[0403] Communication interface 1703 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between device 1700 and other devices or communication networks. For example, training samples (such as original face image samples and noise samples in the method embodiment) can be acquired through communication interface 1703.

[0404] Bus 1704 may include a pathway for transmitting information between various components of device 1700 (e.g., memory 1702, processor 1701, communication interface 1703).

[0405] It should be understood that the acquisition unit 1401 in the training device 1400 of the image generation model is equivalent to the communication interface 1703 in the training device 1700 of the image generation model, and the processing unit 1402 can be equivalent to the processor 1701.

[0406] It should be noted that, although Figure 16 and Figure 17 The illustrated devices 1600 and 1700 only show the memory, processor, and communication interface. However, those skilled in the art should understand that in specific implementations, devices 1600 and 1700 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that devices 1600 and 1700 may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that devices 1600 and 1700 may also include only the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 16or Figure 17 All the devices shown.

[0407] In the above embodiments, the instructions stored in the memory for execution by the processor can be implemented in the form of a computer program product. This computer program product can be pre-written into the memory, or it can be downloaded and installed into the memory as software.

[0408] This application also provides a computer program product containing instructions, which includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. For example, available media may include magnetic media (e.g., floppy disk, hard disk, or magnetic tape), optical media (e.g., digital versatile disc (DVD)), or semiconductor media (e.g., solid-state disk (SSD)).

[0409] This application provides a chip. The chip includes a processor and a data interface. The processor is used to call a computer program in memory through the data interface to execute the technical solutions in the above embodiments. Its implementation principle and technical effects are similar to those in the related embodiments described above, and will not be repeated here.

[0410] This application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program or instructions. When the computer program or instructions are executed by a processor, they implement the methods described above. The methods described in the above embodiments can be implemented wholly or partially by software, hardware, firmware, or any combination thereof. If implemented in software, the functionality can be stored as one or more instructions or code on or transmitted over the computer-readable medium. The computer-readable medium can include computer storage media and communication media, and can also include any medium that can transfer a computer program from one place to another. The storage medium can be any target medium accessible by a computer.

[0411] This application also provides a mobile terminal, which includes any of the devices described in this application, such as... Figure 14 , Figure 15 , Figure 16 or Figure 17 The apparatus shown.

[0412] For example, the mobile terminal can be a drone or a robot.

[0413] This application also provides a cloud platform, which includes any of the devices described in this application, such as... Figure 14 , Figure 15 , Figure 16 or Figure 17 The apparatus shown.

[0414] This application also provides a wearable device, which includes any of the devices described in this application, such as... Figure 14 , Figure 15 , Figure 16 or Figure 17 The apparatus shown.

[0415] For example, wearable devices may include glasses or smartwatches.

[0416] This application also provides a vehicle that includes any of the devices described in this application, such as... Figure 14 , Figure 15 , Figure 16 or Figure 17 The apparatus shown.

[0417] This application also provides an electronic device, which includes any of the devices described in the embodiments of this application, such as... Figure 14 , Figure 15 , Figure 16 or Figure 17 The apparatus shown.

[0418] For example, electronic devices include televisions, computers, or mobile phones.

[0419] As one possible design, computer-readable media may include compact disc read-only memory (CD-ROM), RAM, ROM, EEPROM, or other optical disc storage; computer-readable media may include disk storage or other disk storage devices. Furthermore, any connecting cable may also be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of media. As used herein, disks and optical discs include optical discs (CD), laser discs, optical discs, DVDs, floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs optically reproduce data using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0420] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0421] The above specific embodiments further illustrate the purpose, technical solution and beneficial effects of this application. It should be understood that the above are only specific embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of this application should be included within the scope of protection of this application.

Claims

1. An image generation method, characterized in that, The method is applied to a pre-trained image generation model, which includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. The method includes: Obtain the feature vector of the face region in the original face image; The nth diffusion processing module is called to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first feature vector. Here, n traverses from 1 to N. When n traverses to 1, the first input is a random noise vector. During the process of n traversing from 2 to N, the first input is the output of the (n-1)th diffusion processing module. The nth diffusion processing module is invoked to extract a second feature vector from the first input based on the coordinates of the face region. The second feature vector is then upsampled, and feature learning is performed on the upsampled second feature vector to obtain an intermediate feature vector. The intermediate feature vector is then downsampled to the size of the second feature vector to obtain a third feature vector. The coordinates of the face region are the position of the face region in the target image to be generated. The nth diffusion processing module is invoked to fuse the third feature vector with the first feature vector to obtain a fourth feature vector. Based on the fourth feature vector, the output of the nth diffusion processing module is generated. When n traverses to N, the target image is determined based on the output of the nth diffusion processing module.

2. The method according to claim 1, characterized in that, The method further includes: Obtain the descriptive feature vector of the descriptive information corresponding to the target image; wherein, the descriptive information corresponding to the target image is used to describe the target image from different dimensions; The step of calling the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first feature vector includes: The nth diffusion processing module is invoked to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector.

3. The method according to claim 1 or 2, characterized in that, The step of generating the output of the nth diffusion processing module based on the fourth feature vector includes: The fourth feature vector is convolved to obtain the fifth feature vector; Based on the coordinates of the face region, a sixth feature vector is extracted from the fourth feature vector, and feature learning is performed on the sixth feature vector to obtain a seventh feature vector; The seventh feature vector is fused with the fifth feature vector to obtain the eighth feature vector, and the output of the nth diffusion processing module is generated based on the eighth feature vector.

4. The method according to claim 1 or 2, characterized in that, The upsampling process on the second feature vector includes: If the matrix corresponding to the second eigenvector is not a block matrix, the second eigenvector is padded with zeros, and the matrix corresponding to the second eigenvector after zero-padding is a block matrix. The second feature vector, after being padded with zeros, is then upsampled.

5. The method according to claim 1 or 2, characterized in that, The step of calling the nth diffusion processing module to fuse the third feature vector with the first feature vector to obtain the fourth feature vector includes: The nth diffusion processing module is invoked to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

6. The method according to claim 5, characterized in that, The step of calling the nth diffusion processing module to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector includes: The nth diffusion processing module is invoked to determine the feature vector corresponding to the coordinates of the face region from the first feature vector, thus obtaining the ninth feature vector; The ninth feature vector and the third feature vector are fused in a weighted manner, and the fusion result is used to update the ninth feature vector in the first feature vector to obtain the fourth feature vector.

7. The method according to claim 2, characterized in that, The step of calling the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector includes: The nth diffusion processing module is invoked to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first cross-attention feature vector; The nth diffusion processing module is invoked to perform cross-attention processing on the descriptive feature vector and the first input to obtain the second cross-attention feature vector; The nth diffusion processing module is invoked to fuse the first cross-attention feature vector and the second cross-attention feature vector to obtain the first feature vector.

8. An image generation model, characterized in that, The system comprises N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. Each diffusion processing module includes a cross-attention layer, which consists of a cross-attention unit, a first target learning unit, and a first fusion unit. The outputs of the cross-attention unit and the first target learning unit serve as the inputs to the first fusion unit. The cross-attention unit of the nth diffusion processing module is used to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first feature vector. Here, n traverses from 1 to N. When n traverses to 1, the first input is a random noise vector. During the process of n traversing from 2 to N, the first input is the output of the (n-1)th diffusion processing module. The feature vector of the face region is the feature vector of the face region in the original face image. The first target learning unit of the nth diffusion processing module is used to extract a second feature vector from the first input based on the coordinates of the face region, perform upsampling processing on the second feature vector, and perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector; and downsample the intermediate feature vector to the size of the second feature vector to obtain a third feature vector; the coordinates of the face region are the position of the face region in the target image to be generated; The first fusion unit of the nth diffusion processing module is used to fuse the third feature vector with the first feature vector to obtain a fourth feature vector, and generate the output of the nth diffusion processing module based on the fourth feature vector. When n traverses to N, the target image is determined based on the output of the nth diffusion processing module.

9. The image generation model according to claim 8, characterized in that, The cross-attention unit of the nth diffusion processing module is further used to perform cross-attention processing on the feature vector, descriptive feature vector and the first input of the face region to obtain the first feature vector; wherein the descriptive feature vector is obtained based on the descriptive information corresponding to the target image, and the descriptive information is used to describe the target image from different dimensions.

10. The image generation model according to claim 8 or 9, characterized in that, The diffusion processing module further includes a convolutional layer, which comprises a convolutional unit, a second target learning unit, and a second fusion unit. The input of the convolutional unit is the output of the cross-attention layer, and the outputs of the convolutional units and the second target learning unit are the inputs of the second fusion unit. The convolutional unit of the nth diffusion processing module is used to perform convolution processing on the fourth feature vector to obtain the fifth feature vector. The second target learning unit of the nth diffusion processing module is used to extract the sixth feature vector from the fourth feature vector according to the coordinates of the face region, and perform feature learning on the sixth feature vector to obtain the seventh feature vector; The second fusion unit of the nth diffusion processing module is used to fuse the seventh feature vector with the fifth feature vector to obtain the eighth feature vector, and generate the output of the nth diffusion processing module based on the eighth feature vector.

11. The image generation model according to claim 8 or 9, characterized in that, The first target learning unit of the nth diffusion processing module is specifically used to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

12. The image generation model according to claim 8 or 9, characterized in that, The first fusion unit of the nth diffusion processing module is specifically used to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

13. The image generation model according to claim 12, characterized in that, The first fusion unit of the nth diffusion processing module is specifically used to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

14. The image generation model according to claim 9, characterized in that, The cross-attention unit of the nth diffusion processing module is specifically used to perform cross-attention processing on the feature vector of the face region and the first input to obtain a first cross-attention feature vector; to perform cross-attention processing on the descriptive feature vector and the first input to obtain a second cross-attention feature vector; and to fuse the first cross-attention feature vector and the second cross-attention feature vector to obtain the first feature vector.

15. A training method for an image generation model, characterized in that, The method includes: Obtain noise samples and original face image samples, and input the noise samples and the original face image samples into an initial image generation model. The noise samples are obtained by adding noise to the original face image samples. The initial image generation model includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. The diffusion processing module includes a cross attention layer, which includes a cross attention unit, a first target learning unit, and a first fusion unit. The output of the cross attention unit and the output of the first target learning unit are the inputs of the first fusion unit. For the nth diffusion processing module, the parameters of the cross-attention unit are frozen, and cross-attention processing is performed on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention unit to obtain the first feature vector. The feature vector of the face region is the feature vector obtained based on the face region in the original face image sample. n traverses from 1 to N. When n traverses to 1, the first input is the noise feature vector, which is the feature vector obtained based on the noise sample. During the process of n traversing from 2 to N, the first input is the output of the (n-1)th diffusion processing module. Based on the first feature vector and the target image, a loss calculation is performed to obtain the cross-entropy loss. The parameters of the first target learning unit and the first fusion unit are adjusted according to the cross-entropy loss. The target image is a zero-based image with the same size as the first feature vector and the ground truth value in the face region. The first target learning unit is used to extract a second feature vector from the first input based on the coordinates of the face region, upsample the second feature vector, and perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector. The intermediate feature vector is downsampled to the size of the second feature vector to obtain a third feature vector. The coordinates of the face region are the position of the face region in the target image to be generated. The first fusion unit is used to fuse the third feature vector with the first feature vector to obtain a fourth feature vector. Based on the fourth feature vector, the output of the nth diffusion processing module is generated. When n traverses to N, the target image corresponding to the original face image is determined based on the output of the nth diffusion processing module.

16. The method according to claim 15, characterized in that, The method further includes: Obtain a descriptive feature vector and input the descriptive feature vector into the initial image generation model. The descriptive feature vector is the feature vector of the descriptive information corresponding to the target image. The descriptive information is used to describe the target image from different dimensions. The first feature vector is obtained by performing cross-attention processing on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention unit, including: Based on the parameters of the frozen cross-attention unit, cross-attention processing is performed on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector.

17. The method according to claim 15 or 16, characterized in that, The diffusion processing module further includes a convolutional layer, which comprises a convolutional unit, a second target learning unit, and a second fusion unit. The input of the convolutional unit is the output of the cross-attention layer, and the output of the convolutional unit and the output of the second target learning unit are the inputs of the second fusion unit. The method further includes: For the nth diffusion processing module, the parameters of the convolution unit are frozen, and the fourth feature vector is convolved based on the frozen parameters of the convolution unit to obtain the fifth feature vector; Based on the parameters of the second target learning unit, a sixth feature vector is extracted from the fourth feature vector according to the coordinates of the face region, and feature learning is performed on the sixth feature vector to obtain a seventh feature vector; Based on the parameters of the second fusion unit, the seventh feature vector and the fifth feature vector are fused to obtain the eighth feature vector, and the output of the nth diffusion processing module is generated according to the eighth feature vector. The parameters of the second target learning unit and the parameters of the second fusion unit are adjusted according to the cross-entropy loss.

18. The method according to claim 15 or 16, characterized in that, The method further includes: The first target learning unit is specifically used to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

19. The method according to claim 15 or 16, characterized in that, The method further includes: The first fusion unit is specifically used to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

20. The method according to claim 19, characterized in that, The method further includes: The first fusion unit is specifically used to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

21. The method according to claim 16, characterized in that, The cross-attention unit, based on its frozen parameters, performs cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector, including: Based on the parameters of the frozen cross-attention unit, cross-attention processing is performed on the feature vector of the face region and the first input to obtain a first cross-attention feature vector; cross-attention processing is performed on the description feature vector and the first input to obtain a second cross-attention feature vector; the first cross-attention feature vector and the second cross-attention feature vector are fused to obtain the first feature vector.

22. An image generation apparatus, characterized in that, The device includes: The acquisition unit is used to acquire the feature vector of the face region in the original face image; The processing unit comprises N cascaded diffusion processing modules, where N is an integer greater than or equal to 2, and is used for: The nth diffusion processing module is called to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first feature vector. Here, n traverses from 1 to N. When n traverses to 1, the first input is a random noise vector. During the process of n traversing from 2 to N, the first input is the output of the (n-1)th diffusion processing module. The nth diffusion processing module is invoked to extract a second feature vector from the first input based on the coordinates of the face region. The second feature vector is then upsampled, and feature learning is performed on the upsampled second feature vector to obtain an intermediate feature vector. The intermediate feature vector is then downsampled to the size of the second feature vector to obtain a third feature vector. The coordinates of the face region are the position of the face region in the target image to be generated. The nth diffusion processing module is invoked to fuse the third feature vector with the first feature vector to obtain a fourth feature vector. Based on the fourth feature vector, the output of the nth diffusion processing module is generated. When n traverses to N, the target image is determined based on the output of the nth diffusion processing module.

23. The apparatus according to claim 22, characterized in that, The acquisition unit is further configured to acquire the description feature vector of the description information corresponding to the target image; wherein, the description information corresponding to the target image is used to describe the target image from different dimensions; The processing unit is specifically used to call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region, the descriptive feature vector, and the first input to obtain the first feature vector.

24. The apparatus according to claim 22 or 23, characterized in that, The processing unit is also used to call the nth diffusion processing module to perform convolution processing on the fourth feature vector to obtain the fifth feature vector; The nth diffusion processing module is invoked to extract the sixth feature vector from the fourth feature vector based on the coordinates of the face region, and feature learning is performed on the sixth feature vector to obtain the seventh feature vector; The nth diffusion processing module is invoked to fuse the seventh feature vector with the fifth feature vector to obtain the eighth feature vector. Based on the eighth feature vector, the output of the nth diffusion processing module is generated.

25. The apparatus according to claim 22 or 23, characterized in that, The processing unit is specifically used to call the nth diffusion processing module to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

26. The apparatus according to claim 22 or 23, characterized in that, The processing unit is specifically used to call the nth diffusion processing module to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

27. The apparatus according to claim 26, characterized in that, The processing unit is specifically used to call the nth diffusion processing module to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

28. The apparatus according to claim 23, characterized in that, The processing unit is specifically used to call the nth diffusion processing module to perform cross-attention processing on the feature vector of the face region and the first input to obtain the first cross-attention feature vector. The descriptive feature vector and the first input are subjected to cross-attention processing to obtain a second cross-attention feature vector; The first cross-attention feature vector and the second cross-attention feature vector are fused to obtain the first feature vector.

29. A training device for an image generation model, characterized in that, The device includes: The acquisition unit is used to acquire noise samples and original face image samples, wherein the noise samples are obtained by adding noise to the original face image samples; The processing unit is used to input the noise samples and the original face image samples into the initial image generation model. The initial image generation model includes N cascaded diffusion processing modules, where N is an integer greater than or equal to 2. The diffusion processing module includes a cross attention layer, which includes a cross attention unit, a first target learning unit, and a first fusion unit. The output of the cross attention unit and the output of the first target learning unit are the inputs of the first fusion unit. For the nth diffusion processing module, the parameters of the cross-attention unit are frozen, and cross-attention processing is performed on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention unit to obtain the first feature vector. The feature vector of the face region is the feature vector obtained based on the face region in the original face image sample. n traverses from 1 to N. When n traverses to 1, the first input is the noise feature vector, which is the feature vector obtained based on the noise sample. During the process of n traversing from 2 to N, the first input is the output of the (n-1)th diffusion processing module. Based on the first feature vector and the target image, a loss calculation is performed to obtain the cross-entropy loss. The parameters of the first target learning unit and the first fusion unit are adjusted according to the cross-entropy loss. The target image is a zero-based image with the same size as the first feature vector and the ground truth value in the face region. The first target learning unit is used to extract a second feature vector from the first input based on the coordinates of the face region, upsample the second feature vector, and perform feature learning on the upsampled second feature vector to obtain an intermediate feature vector. The intermediate feature vector is downsampled to the size of the second feature vector to obtain a third feature vector. The coordinates of the face region are the position of the face region in the target image to be generated. The first fusion unit is used to fuse the third feature vector with the first feature vector to obtain a fourth feature vector. Based on the fourth feature vector, the output of the nth diffusion processing module is generated. When n traverses to N, the target image corresponding to the original face image is determined based on the output of the nth diffusion processing module.

30. The apparatus according to claim 29, characterized in that, The acquisition unit is further configured to acquire a descriptive feature vector, wherein the descriptive feature vector is a feature vector of the descriptive information corresponding to the target image, and the descriptive information is used to describe the target image from different dimensions; The processing unit is further configured to input the descriptive feature vector into the initial image generation model, and for the nth diffusion processing module, freeze the parameters of the cross attention unit, and perform cross attention processing on the feature vector of the face region, the descriptive feature vector and the first input based on the frozen parameters of the cross attention unit to obtain the first feature vector.

31. The apparatus according to claim 29 or 30, characterized in that, The diffusion processing module further includes a convolutional layer, which comprises a convolutional unit, a second target learning unit, and a second fusion unit. The input of the convolutional unit is the output of the cross-attention layer, and the outputs of the convolutional units and the second target learning unit are the inputs of the second fusion unit. The processing unit is further configured to freeze the parameters of the convolution unit, and perform convolution processing on the fourth feature vector based on the frozen parameters of the convolution unit to obtain the fifth feature vector. Based on the parameters of the second target learning unit, a sixth feature vector is extracted from the fourth feature vector according to the coordinates of the face region, and feature learning is performed on the sixth feature vector to obtain a seventh feature vector; Based on the parameters of the second fusion unit, the seventh feature vector and the fifth feature vector are fused to obtain the eighth feature vector, and the output of the nth diffusion processing module is generated according to the eighth feature vector. The parameters of the second target learning unit and the parameters of the second fusion unit are adjusted according to the cross-entropy loss.

32. The apparatus according to claim 29 or 30, characterized in that, The processing unit is specifically used to perform zero-padding on the second feature vector when the matrix corresponding to the second feature vector is a non-block matrix, so that the matrix corresponding to the second feature vector after zero-padding is a block matrix; and to perform upsampling on the second feature vector after zero-padding.

33. The apparatus according to claim 29 or 30, characterized in that, The processing unit is specifically used to fuse the third feature vector with the first feature vector in a weighted manner to obtain the fourth feature vector.

34. The apparatus according to claim 33, characterized in that, The processing unit is specifically configured to determine the feature vector corresponding to the coordinates of the face region from the first feature vector to obtain the ninth feature vector; to fuse the ninth feature vector with the third feature vector in a weighted manner, and to update the ninth feature vector in the first feature vector with the fusion result to obtain the fourth feature vector.

35. The apparatus according to claim 30, characterized in that, The processing unit is specifically configured to perform cross-attention processing on the feature vector of the face region and the first input based on the frozen parameters of the cross-attention unit to obtain a first cross-attention feature vector; perform cross-attention processing on the descriptive feature vector and the first input to obtain a second cross-attention feature vector; and fuse the first cross-attention feature vector and the second cross-attention feature vector to obtain the first feature vector.

36. An image generation apparatus, characterized in that, The system includes a memory and a processor, the memory storing a computer program that executes the image generation method as described in any one of claims 1 to 7 when the processor is running.

37. A training device for an image generation model, characterized in that, It includes a memory and a processor, the memory storing a computer program that executes a training method for the image generation model as described in any one of claims 15 to 21 when the processor is running.

38. A computer-readable storage medium, characterized in that, It stores a computer program that, when run on a processor, executes the image generation method as described in any one of claims 1 to 7, or, when run on a processor, executes the training method for the image generation model as described in any one of claims 15 to 21.

39. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the image generation method as described in any one of claims 1 to 7, or the computer program, when run on a processor, performs the training method for the image generation model as described in any one of claims 15 to 21.

Images