Diffusion processing method and apparatus, electronic device, and medium

Abstract

A diffusion processing method, executed by an electronic device. The diffusion processing method is executed in a plurality of denoising steps, and a diffusion processing model is executed in each denoising step. The diffusion processing model is divided into a first number of model tasks and assigned to a first number of diffusion processing units for execution, and when the diffusion processing unit in each denoising step executes an assigned model task, a first key value generated in an execution process of the diffusion processing model in a previous denoising step is used. The method comprises: in a preheating step before the plurality of denoising steps, dividing an image into a second number of preheating blocks, and allocating same to a second number of preheating processing units for processing, each preheating processing unit sequentially executing the first number of model tasks in the preheating step, to obtain a first sub-key value corresponding to each model task (410); transferring the first sub-key value corresponding to the model task to a diffusion processing unit corresponding to the model task, for the diffusion processing unit to synthesize, by using a second number of first sub-key values generated by the second number of preheating processing units in the preheating step, the first key value used by the first denoising step among the plurality of denoising steps, for the first denoising step to execute the diffusion processing model (420); and in any other denoising step after the first denoising step among the plurality of denoising steps, executing, by means of each diffusion processing unit, the diffusion processing model by using a first key value generated in a previous denoising step of the other denoising step (430).

Description

Diffusion processing method and device, electronic equipment and medium

[0001] Related applications

[0002] The present application claims priority to the Chinese patent application No. 202411056542.0, filed on August 01, 2024, and entitled “Diffusion processing method and device, electronic equipment and medium”, the contents of which are hereby incorporated by reference in its entirety. TECHNICAL FIELD

[0003] The present disclosure relates to the field of artificial intelligence, and in particular, to a diffusion processing method and device, electronic equipment and medium. BACKGROUND

[0004] A diffusion model can generate a high-resolution image by simulating the diffusion process of a substance in a medium. For example, in a text-to-image application, a diffusion guide vector is generated according to the text description of an image in a text-to-image. The image is input into the diffusion model, and the image is gradually denoised under the guidance of the diffusion guide vector, i.e., the diffusion process is divided into a plurality of denoising steps, and each denoising step removes a part of the noise from the image. In this way, after all the denoising steps are completed, a target image that meets the text description of the image is obtained.

[0005] Therefore, when the diffusion model is executed, the diffusion model is actually executed at each denoising step. In each denoising step, in order to promote parallelism of execution and improve execution efficiency, the diffusion model is divided into a plurality of model tasks, and each model task is assigned to a diffusion processing unit (such as a GPU) for execution. For example, the diffusion model has 10 layers, which are divided into 5 model tasks, and each model task has 2 layers. The 2 layers are executed by one diffusion processing unit, and a total of 5 diffusion processing units are required. When each model task is executed, it cannot rely only on the parameters of the model task itself, and sometimes needs parameters obtained by multiple model tasks together. For example, the diffusion transformer network (DiT), which is a mainstream diffusion model, has an important multi-head attention model layer, which needs key-value (KV) parameters of multiple model tasks as a whole, rather than KV parameters in a certain model task. However, when a certain model task is executed, the model tasks that are arranged in front of the model task in the plurality of model tasks have obtained the KV parameters in the denoising step, and the model tasks that are arranged behind the model task in the plurality of model tasks have not obtained the KV parameters in the denoising step, but the KV parameters have been obtained in the previous denoising step. Considering that the KV parameters in consecutive denoising steps are not very different, the KV parameters in the previous denoising step can be used.

[0006] However, there is no previous de-noising step for the first de-noising step. Therefore, a number of preheating steps are added before the first de-noising step. Each diffusion processing unit is provided with a preheating step. In the preheating step, the diffusion processing unit processes a plurality of image blocks to obtain the KV parameters of the image population. After a number of preheating steps equal to the number of diffusion processing units, the preliminary KV parameters of each diffusion processing unit are obtained. However, the more diffusion processing units there are, the more preheating steps are set in advance. In the preheating step, the resource utilization is insufficient, and the overall diffusion processing efficiency is low. SUMMARY

[0007] The present disclosure provides a diffusion processing method, device, electronic equipment and medium.

[0008] According to an aspect of the present disclosure, a diffusion processing method is provided, which is performed in a plurality of de-noising steps, a diffusion processing model is executed in each de-noising step, the diffusion processing model is divided into a first number of model tasks, and is executed by a first number of diffusion processing units, when the diffusion processing units execute the assigned model tasks in each de-noising step, the first key value generated during the execution of the diffusion processing model in the previous de-noising step is used, and the diffusion processing method comprises:

[0009] In the preheating step before the plurality of de-noising steps, the image is divided into a second number of preheating blocks, which are processed by a second number of preheating processing units, each preheating processing unit sequentially executes the first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each model task;

[0010] The first sub-key value corresponding to the model task is transmitted to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes the first key value used in the first de-noising step in the plurality of de-noising steps by using the second number of first sub-key values generated by the second number of preheating processing units in the preheating step, so that the diffusion processing model is executed in the first de-noising step; and

[0011] In any other de-noising step after the first de-noising step in the plurality of de-noising steps, the diffusion processing model is executed by each diffusion processing unit using the first key value generated in the previous de-noising step of the other de-noising step.

[0012] According to an aspect of the present disclosure, there is provided a diffusion processing apparatus, a method applied by the diffusion processing apparatus is performed in multiple denoising steps, a diffusion processing model is executed in each of the denoising steps, the diffusion processing model is divided into a first number of model tasks, and is executed by a first number of diffusion processing units, in each denoising step, the diffusion processing units execute the assigned model tasks, and a first key value generated in the execution of the diffusion processing model in the last denoising step is used, the diffusion processing apparatus comprises:

[0013] an assignment unit configured to divide an image into a second number of preheating blocks in a preheating step before the multiple denoising steps, and assign the preheating blocks to a second number of preheating processing units for processing, each of the preheating processing units sequentially executes the first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each of the model tasks;

[0014] a transmission unit configured to transmit the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes the first key value used in a first denoising step of the multiple denoising steps by using a second number of the first sub-key values generated by the second number of preheating processing units in the preheating step, so that the first denoising step executes the diffusion processing model; and

[0015] an execution unit configured to execute the diffusion processing model by each of the diffusion processing units by using the first key value generated in the last denoising step in any other denoising step after the first denoising step of the multiple denoising steps.

[0016] According to an aspect of the present disclosure, there is provided an electronic device comprising a memory and a processor, the memory stores a computer program, and the processor implements the diffusion processing method as described above when executing the computer program.

[0017] According to an aspect of the present disclosure, there is provided a computer readable storage medium, the storage medium stores a computer program, and the computer program is executed by a processor to implement the diffusion processing method as described above.

[0018] According to an aspect of the present disclosure, there is provided a computer program product comprising a computer program, the computer program is read and executed by a processor of a computer device, so that the computer device executes the diffusion processing method as described above.

[0019] Other features and advantages of the present disclosure will be set forth in the following description, and in part will become apparent from the description, or can be learned by practice of the present disclosure. The objects and other advantages of the present disclosure can be achieved and obtained by the structure particularly pointed out in the specification, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced as follows. Obviously, the drawings in the following description only belong to the embodiments of the present application, and other drawings can also be obtained by those skilled in the art without any creative effort based on the disclosed drawings.

[0021] FIG. 1 is a framework diagram of a system to which a diffusion processing method according to an embodiment of the present disclosure is applied;

[0022] FIGS. 2A-2C are application scenario interface diagrams of an embodiment of the present disclosure applied in a handwriting application scenario;

[0023] FIG. 3 is a general structure diagram of a diffusion model;

[0024] FIG. 4 is a main flow diagram of a diffusion processing method according to an embodiment of the present disclosure;

[0025] FIG. 5 is a schematic diagram of a diffusion processing model being divided into a first number of model tasks and assigned to a first number of diffusion processing units for execution according to an embodiment of the present disclosure;

[0026] FIG. 6 is a schematic diagram of an image being divided into a second number of preheating subblocks according to an embodiment of the present disclosure;

[0027] FIG. 7 is a schematic diagram of each preheating processing unit executing a first number of model tasks in a preheating step sequence to obtain a first sub-key value corresponding to each model task according to an embodiment of the present disclosure;

[0028] FIG. 8 is a schematic diagram of each preheating processing unit passing a first sub-key value corresponding to a model task to a diffusion processing unit corresponding to the model task according to an embodiment of the present disclosure;

[0029] FIG. 9 is a schematic diagram of steps executed by a preheating processing unit and steps executed by a diffusion processing unit according to an embodiment of the present disclosure;

[0030] FIG. 10 is a flow diagram of a first number of model tasks being executed in a preheating step sequence according to an embodiment of the present disclosure;

[0031] FIG. 11A is a schematic diagram of a first number of model tasks being unequally assigned to a plurality of preheating steps according to an embodiment of the present disclosure;

[0032] FIG. 11B is a schematic diagram of a first number of model tasks being equally assigned to a plurality of preheating steps according to an embodiment of the present disclosure;

[0033] FIG. 12 is a flow diagram of a first number of model tasks being assigned to a plurality of preheating steps according to an embodiment of the present disclosure;

[0034] FIG. 13 is a schematic diagram of assigning a first number of model tasks to a plurality of warm-up steps according to an embodiment of the present disclosure;

[0035] FIG. 14 is a flowchart of determining a number of warm-up steps according to an embodiment of the present disclosure;

[0036] FIG. 15 is a flowchart of acquiring a diffusion processing complexity according to an embodiment of the present disclosure;

[0037] FIG. 16A is a schematic diagram of diffusion target description information and a target image according to an embodiment of the present disclosure;

[0038] FIG. 16B is a schematic diagram of another diffusion target description information and a target image according to an embodiment of the present disclosure;

[0039] FIG. 17 is a flowchart of determining a number of warm-up steps based on a model size and a diffusion processing complexity according to an embodiment of the present disclosure;

[0040] FIG. 18 is a flowchart of determining a second number according to an embodiment of the present disclosure;

[0041] FIG. 19 is a flowchart of synthesizing a first key value used in a first denoising step of a plurality of denoising steps using a second number of first sub-key values generated by a second number of warm-up processing units in a warm-up step according to an embodiment of the present disclosure;

[0042] FIG. 20A is a schematic diagram of sequentially dividing an image into a second number of warm-up patches according to an embodiment of the present disclosure;

[0043] FIG. 20B is a schematic diagram of integrating first sub-matrices corresponding to respective warm-up patches generated by a second number of warm-up processing units into a first matrix according to a sequence of the warm-up patches according to an embodiment of the present disclosure;

[0044] FIG. 21 is a flowchart of transferring a first sub-key value corresponding to a model task to a diffusion processing unit corresponding to the model task according to an embodiment of the present disclosure;

[0045] FIG. 22 is a schematic diagram of sequentially transferring first sub-key values corresponding to model tasks to diffusion processing units corresponding to the model tasks according to a sequence of the model tasks in a last warm-up step of a plurality of warm-up steps according to an embodiment of the present disclosure;

[0046] FIG. 23 is a flowchart of transferring a first sub-key value corresponding to a model task to a diffusion processing unit corresponding to the model task at a transmission start time according to an embodiment of the present disclosure;

[0047] FIG. 24 is a schematic diagram of passing the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task at the transmission start time according to an embodiment of the present disclosure;

[0048] FIG. 25 is a flowchart of performing diffusion processing model by each diffusion processing unit using the first key value generated by the previous de-noising step of the other de-noising step after the first de-noising step of the multiple de-noising steps according to an embodiment of the present disclosure;

[0049] FIG. 26 is a schematic diagram of starting to perform each model task in different de-noising sub-steps of the other de-noising step according to an embodiment of the present disclosure;

[0050] FIG. 27 is a flowchart of performing each model task by each diffusion processing unit using the first synthetic key value according to an embodiment of the present disclosure;

[0051] FIG. 28A is a schematic diagram of dividing the image into a fourth number of image patches according to an embodiment of the present disclosure;

[0052] FIG. 28B is a schematic diagram of performing each model task by each diffusion processing unit using the first synthetic key value according to an embodiment of the present disclosure;

[0053] FIG. 29 is a flowchart of processing the second image patch to the last image patch of the fourth number of image patches using the first synthetic key value respectively according to an embodiment of the present disclosure;

[0054] FIG. 30 is a detailed flowchart of an application example of the diffusion processing method according to an embodiment of the present disclosure;

[0055] FIG. 31 is a block diagram of a diffusion processing device according to an embodiment of the present disclosure;

[0056] FIG. 32 is a terminal structure diagram of performing the diffusion processing method shown in FIG. 4 according to an embodiment of the present disclosure;

[0057] FIG. 33 is a server structure diagram of performing the diffusion processing method shown in FIG. 4 according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0058] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, but not all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present disclosure.

[0059] Before the embodiments of the present disclosure are described in further detail, the terms and names involved in the embodiments of the present disclosure are explained, and the terms and names involved in the embodiments of the present disclosure are applicable to the following explanations:

[0060] Diffusion model: A generative model that generates high-resolution images by simulating the diffusion process of matter in a medium. Diffusion models are inspired by non-equilibrium thermodynamics. They define a Markov chain of diffusion steps to slowly add random noise to the data, and then learn to reverse the diffusion process to construct the desired data samples from noise. Unlike VAE or Flow-based models, diffusion models are learned in a fixed procedure, and the latent variables have high dimensionality (same as the original data). Diffusion models include forward diffusion and reverse diffusion.

[0061] Forward diffusion: Gradually add noise to the original image until the original image becomes completely noisy. The original image can be an arbitrary image. Adding noise to the original image is done through a noise adding step. In each noise adding step, the added noise and the image before adding noise are recorded for use in reverse diffusion.

[0062] Reverse diffusion: Starting from a completely noisy state, gradually remove noise to restore a clear image. This process is achieved through a series of denoising steps, which correspond to the corresponding noise adding steps. Noise adding steps and denoising steps are collectively referred to as diffusion steps. For example, in forward diffusion, the noise adding steps are 1, 2, …, T, and in reverse diffusion, the denoising steps are T, T-1, …, 1. Denoising step T corresponds to noise adding step T, denoising step T-1 corresponds to noise adding step T-1, and so on.

[0063] Multi-head attention mechanism: Multi-head attention mechanism is an improved method based on self-attention mechanism. Self-attention mechanism is a method that can calculate the weight of each position in the input sequence, which can well handle the problem of long-distance dependence in the sequence. However, in practical applications, there may be multiple attention points, so multiple self-attention mechanisms are needed to handle different attention points. Multi-head attention mechanism is a method that uses multiple self-attention mechanisms on an input sequence to get multiple sets of attention results, and then concatenates and linearly projects these results to get the final output. The advantage of multi-head attention is that it can handle multiple attention points, and can better handle complex semantic relationships.

[0064] The diffusion model can generate a high-resolution image by simulating the diffusion process of a substance in a medium. For example, in a text-to-image application, a diffusion guide vector is generated according to the text description of the image in the text-to-image. The image is input into the diffusion model, and the image is denoised step by step under the guidance of the diffusion guide vector, that is, the diffusion process is divided into a plurality of denoising steps, and each denoising step removes a part of noise from the image. In this way, after all the denoising steps, the target image that meets the text description of the image is obtained.

[0065] Therefore, when the diffusion model is executed, the diffusion model is actually executed once at each denoising step. In each denoising step, in order to promote the parallelism of execution and improve the efficiency of execution, the diffusion model is divided into a plurality of model tasks, and each model task is assigned to a diffusion processing unit (such as a GPU) for execution. For example, the diffusion model has 10 layers and is divided into 5 model tasks, and each model task has 2 layers. The 2 layers are executed by one diffusion processing unit, and a total of 5 diffusion processing units are required. When each model task is executed, the model task cannot only rely on the parameters of the model task itself, and sometimes needs parameters obtained by a plurality of model tasks together. For example, the diffusion transformer network (DiT) which is a mainstream diffusion model has an important multi-head attention model layer, and the key value (KV) parameters of the plurality of model tasks as a whole are needed instead of the KV parameters in a certain model task. However, when a certain model task is executed, the model tasks in front of the model task in the plurality of model tasks have obtained the KV parameters in the denoising step, and the model tasks behind the model task in the plurality of model tasks have not obtained the KV parameters in the denoising step, but the KV parameters have been obtained in the previous denoising step. Considering that the KV parameters in the continuous denoising steps are not much different, the KV parameters in the previous denoising step can be used.

[0066] However, for the first denoising step, there is no previous denoising step. Therefore, before the first denoising step, a plurality of warm-up steps are added. Each diffusion processing unit is provided with a warm-up step. In the warm-up step, the diffusion processing unit processes a plurality of image blocks to obtain the KV parameters of the image as a whole. After a number of warm-up steps equal to the number of diffusion processing units, the preliminary KV parameters of each diffusion processing unit are obtained. However, the more diffusion processing units there are, the more warm-up steps are set in advance, and the resource utilization rate in the warm-up step is insufficient, and the efficiency of the entire diffusion processing is low.

[0067] Based on this, the diffusion processing method, device, electronic equipment and medium provided by the embodiments of the present disclosure can provide preliminary global KV parameters to a plurality of diffusion processing units in one warm-up step, reduce the resource idling in the warm-up process, and improve the efficiency of the entire diffusion processing.

[0068] System architecture and scene to which the embodiments of the present disclosure are applied

[0069] Fig. 1 is a system architecture diagram to which the diffusion processing method according to the embodiment of the present disclosure is applied. It includes: an object terminal 110, the Internet 120, a gateway 130, and a server 140.

[0070] The object terminal 110 is a device used by an object to input requirements for the diffusion processing to a specific application, and to display an image finally generated by the diffusion processing. In addition, the actual diffusion processing of the multiple preheating steps and the multiple denoising steps of the diffusion model in the embodiment of the present disclosure can also be completed by the object terminal 110. In this case, the object terminal 110 can perform the diffusion processing method of the embodiment of the present disclosure alone. The object terminal 110 includes various forms such as a desktop computer, a laptop computer, a PDA (Personal Digital Assistant), a mobile phone, a car terminal, a home theater terminal, a dedicated terminal, and the like. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network, share one display device to work cooperatively, and collectively constitute an object terminal. The object terminal 110 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0071] The gateway 130 is also called an internetworking connector or a protocol converter. The gateway 130 realizes network interconnection at the transport layer and is a computer system or device that acts as a conversion function. In the case of two systems using different communication protocols, data formats or languages, or even having completely different architectures, the gateway 130 is a translator. At the same time, the gateway 130 can also provide filtering and security functions. The messages sent by the object terminal 110 to the server 140 are sent to the corresponding server 140 through the gateway 130. The messages sent by the server 140 to the object terminal 110 are also sent to the corresponding object terminal 110 through the gateway 130.

[0072] The server 140 is a device that provides overall services to all object terminals 110 connected in a network. In the case where the diffusion model is installed in the server 140, the server 140 is responsible for the actual diffusion processing of the multiple preheating steps and the multiple denoising steps of the diffusion model in the embodiment of the present disclosure. Compared with the object terminal 110, the server 140 has higher requirements in stability, security, performance, and the like. The server 140 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a part (such as a virtual machine) of a high-performance computer, a combination of parts (such as virtual machines) of multiple high-performance computers, and the like. The server 140 can also communicate with the Internet 120 in a wired or wireless manner to exchange data.

[0073] The embodiments of the present disclosure can be applied in various scenarios, such as the text-to-image scenario shown in FIGS. 2A-2C, molecule graph modeling, time series modeling, and the like. In the following, the application of the diffusion processing process of the embodiments of the present disclosure will be described in detail taking the text-to-image scenario as an example. The text-to-image application refers to an application of generating an image corresponding to the input description text of the generated image. For example, when the input description text is “There is a crescent moon in the sky, and a person is walking on the road with a bag”, the output is the image shown in FIG. 2C. At present, text-to-image is widely used in conversational robots, large language model processing, engineering drawing, games, and the like.

[0074] As described above, when the diffusion model is used in a specific application, the diffusion guide vector can be generated according to the specific requirements in the specific application. In the text-to-image application, the image to be generated is the target image, and the specific requirement in the application is the target image description, such as “There is a crescent moon in the sky, and a person is walking on the road with a bag”. Based on the target image description, the diffusion guide vector is generated. The diffusion guide vector is a vector specially used to guide the diffusion model to generate the target image. The image is the image before diffusion processing, and in the text-to-image application, it can be any image. Although the image is any image, it is gradually denoised under the guidance of the diffusion guide vector, and the target image shown in FIG. 2C is obtained.

[0075] As shown in FIG. 2A, on the object terminal 110, the object opens the text-to-image application. A prompt “Please enter the text description of the image you want to generate:” appears on the interface. The object inputs “There is a crescent moon in the sky, and a person is walking on the road with a bag” in the chat box.

[0076] As shown in FIG. 2B, after the object inputs the target image description, the object clicks the “Confirm” control next to it. At this time, as shown in FIG. 2C, the finally generated target image is displayed. The target image is obtained by the diffusion processing method of the embodiments of the present disclosure. The specific processing process of the embodiments of the present disclosure will be described in detail in the following.

[0077] It should be understood that the above only illustrates the description of part of the application scenarios of the present disclosure. The business scenarios to which the present disclosure can be applied can include but are not limited to the specific embodiments mentioned above.

[0078] General description of embodiments of the present disclosure

[0079] It should be emphasized that the embodiments of the present disclosure can be applied to various application scenarios, such as text-to-image scenarios, molecular graph modeling scenarios, and the like. In the related art, a number of preheating steps are added before the first denoising step. Each diffusion processing unit is provided with a preheating step. In the preheating step, the diffusion processing unit processes a plurality of image blocks to obtain the KV parameters of the image as a whole. After a number of preheating steps equal to the number of diffusion processing units, the preliminary KV parameters of each diffusion processing unit are obtained. However, the number of preheating steps is set in advance according to the number of diffusion processing units, and the resource utilization rate in the preheating step is insufficient, and the overall diffusion processing efficiency is low. One embodiment of the present disclosure provides a diffusion processing method, device, electronic equipment and medium, which can divide the image into a second number of preheating blocks and assign them to a second number of preheating processing units for processing. Each preheating processing unit can execute a plurality of model tasks in the same preheating step sequence. Thus, the KV parameters of a plurality of model tasks are obtained through one preheating step, and the preliminary overall KV parameters are provided for a plurality of diffusion processing units, reducing the resource idling in the preheating process and improving the efficiency of the overall diffusion processing.

[0080] According to an embodiment of the present disclosure, diffusion processing refers to comprehensive scheduling of various computing resources to execute specific processes in the diffusion model according to the target image description. The diffusion processing method of the embodiments of the present disclosure can be executed by the object terminal 110, or by the server 140, or by a part of the object terminal 110 and a part of the server 140.

[0081] Before describing the diffusion processing method of the embodiments of the present disclosure in detail, the basic structure of the diffusion model will be introduced with reference to FIG. 3.

[0082] The diffusion model is a generative model that generates high-resolution images by simulating the diffusion process of matter in a medium. The diffusion space of matter in the medium is called latent space. The space in which image processing is performed is called image space. To process image data using the diffusion process, it is necessary to convert the data in the image space into data in the latent space, and therefore, an encoder and a decoder are required. The encoder converts the basic image data into a vector z targeted by the diffusion processing, and the decoder converts the target image vector generated by the diffusion model into a diffused target image.

[0083] The entire diffusion process is divided into forward diffusion and reverse diffusion. In forward diffusion, a number of noise adding steps are divided. In each noise adding step, noise is added until the vector z corresponding to the original image becomes a vector z corresponding to a noise image t . After the vector z corresponding to the original image is processed by the first noise adding step, the vector z1 is obtained, and after the second noise adding step, the vector z2 is obtained, and so on. After t noise adding steps, the vector z t is finally obtained. The vector z tThe corresponding image is a completely noisy image. In each denoising step, the added noise and the image vector before the noise is added are recorded for use in the training of the corresponding denoising step in the reverse diffusion.

[0084] In the reverse diffusion, from the completely noisy image corresponding to the vector z t Starting, by a series of denoising steps, the noise is gradually removed, and the clear image vector is recovered. The denoising steps are one-to-one corresponding to the noise adding steps. For example, in the forward diffusion, the noise adding steps are 1, 2, …, T, then in the reverse diffusion, the denoising steps are T, T-1, …, 1. Denoising step T corresponds to noise adding step T, denoising step T-1 corresponds to noise adding step T-1, … It starts from the input vector of the denoising step, each time uses the trained denoising step to predict the noise of the current denoising step, and subtracts the predicted noise from the input vector of the denoising step to obtain the denoised vector. Since in the corresponding noise adding step, the noise added in the corresponding noise adding step and the image vector before the noise is added have been recorded, in the corresponding denoising step, the image vector before the noise is added in the corresponding noise adding step can be compared with the denoised vector, and the difference between the two is used to train the denoising process of each denoising step, so that the diffusion model becomes more accurate.

[0085] In actual use of the diffusion model, the target image description and the image are input on the real input side. The target image description is a description of the target image to be generated, such as "a crescent moon hangs in the sky, and a person carrying a bag is walking on the road". The image is the image that the diffusion model is directed at. Since the diffusion model has been trained to have the ability to gradually denoise according to requirements to obtain the required target image for any image, in principle, the image can be any image. Since the diffusion model can only process vectorized data, the target image description needs to be converted into a diffusion guide vector. The diffusion guide vector guides each denoising step of the diffusion model so that the final target image vector is generated. In actual use of the diffusion model, only the denoising step processing is performed, and the noise adding step processing is not performed. The noise adding step is used in the training process.

[0086] As shown in FIG. 3, in each denoising step, a denoising encoder and a denoising decoder are included. The denoising encoder and the denoising decoder each include two or more mutual attention models. Through the two or more mutual attention models in the denoising encoder, the input image vector is encoded in the denoising aspect. Through the two or more mutual attention models in the denoising decoder, the result of the encoding by the denoising encoder is decoded in the denoising aspect. Through the denoising encoder and the denoising decoder, the denoising processing of one denoising step is completed. After T denoising steps of denoising processing, the target image vector corresponding to the diffusion target image is recovered.

[0087] As shown in FIG. 4, according to one embodiment of the present disclosure, the diffusion processing method is performed in multiple denoising steps, a diffusion processing model is executed in each denoising step, the diffusion processing model is divided into a first number of model tasks, and is executed by a first number of diffusion processing units, when each diffusion processing unit executes the assigned model task in each denoising step, the first key value generated in the execution process of the diffusion processing model of the last denoising step is used, and the diffusion processing method comprises:

[0088] Step 410, in a preheating step before the multiple denoising steps, the image is divided into a second number of preheating blocks, which are assigned to a second number of preheating processing units for processing, each preheating processing unit sequentially executes the first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each model task;

[0089] Step 420, the first sub-key value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes the first key value used in the first denoising step of the multiple denoising steps by using the second number of first sub-key values generated by the second number of preheating processing units in the preheating step, so that the first denoising step executes the diffusion processing model;

[0090] Step 430, in any other denoising step after the first denoising step in the multiple denoising steps, each diffusion processing unit executes the diffusion processing model by using the first key value generated in the previous denoising step of the other denoising step.

[0091] The above steps 410-430 are briefly described below.

[0092] In step 410, in a preheating step before the multiple denoising steps, the image is divided into a second number of preheating blocks, which are assigned to a second number of preheating processing units for processing, each preheating processing unit sequentially executes the first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each model task.

[0093] The division method of the image can adopt a uniform division method, for example, for a two-dimensional image, if the second number is n, the image can be equally divided in the horizontal direction or the vertical direction. Assuming that the size of the image is HxW (heightxwidth), if it is uniformly divided in the horizontal direction, the size of each preheating block is If it is uniformly divided in the vertical direction, the size of each preheating block is Non-uniform division can also be performed according to the content characteristics of the image, for example, the area with rich features in the image is divided more finely, and the area with simple features is divided more coarsely.

[0094] According to an embodiment of the present disclosure, the diffusion processing model, i.e., the model mentioned above that generates a high-resolution image by simulating the diffusion process of a substance in a medium, is not described again here. The denoising step refers to the step of removing noise by the diffusion processing model. The reverse process of the diffusion processing model includes multiple denoising steps, and each denoising step includes a denoising encoder and a denoising decoder, and each denoising encoder and denoising decoder includes two or more cross-attention models.

[0095] It can be understood that the diffusion processing model is divided into a first number of model tasks, which are assigned to the first number of diffusion processing units for execution. Among them, the model task is a part of the task that needs to be executed in the diffusion processing model. One diffusion processing model can be divided into multiple model tasks, and the first number is the number of model tasks in the diffusion processing model. In one division manner, each denoising step can be regarded as a model task. If there are N denoising steps, there are N model tasks. In this case, the first number is N. In another division manner, the denoising encoders in all denoising steps can be regarded as a model task, and the denoising decoders in all denoising steps can be regarded as another model task. In this case, the first number is 2. In another division manner, the cross-attention models with a certain attribute in all denoising steps can be integrated into a model task. In this case, the first number is equal to the number of attributes. The division manner of the model task is not limited here.

[0096] The first number of model tasks is in order. In the case of regarding each denoising step as a model task, the order of the model tasks is the order of the denoising steps themselves. For example, if the image is input into the denoising step N, the denoising step N-1, …, and the denoising step 1 in series, then the denoising step N, the denoising step N-1, …, and the denoising step 1 are the order of the model tasks. In the case of regarding the denoising encoders in all denoising steps as a model task and regarding the denoising decoders in all denoising steps as another model task, the model task of the denoising encoder and the model task of the denoising decoder are the order of the model tasks. In the case of integrating the cross-attention models with a certain attribute in all denoising steps into a model task, all attributes can be sorted, and the order of the attributes is the order of the model tasks.

[0097] The diffusion processing unit refers to a processing unit for performing the model task of the diffusion processing model in the denoising step, and the preheating processing unit refers to a processing unit for performing the model task of the diffusion processing model in the preheating step. Moreover, the number of diffusion processing units and the number of preheating processing units can be pre-set by experience. The processing unit is a hardware structure that performs the model task. For example, in a processor structure with CPU and GPU, the CPU is responsible for the scheduling of instructions, and each GPU can be the smallest hardware unit that performs the model task, i.e., the processing unit. The denoising encoder, denoising decoder, etc. shown in FIG. 3 are not hardware structures, and they can be deployed on the same hardware structure or on different hardware structures. For example, when each denoising step is deployed on a processing unit, the denoising encoder and the denoising decoder are deployed on the same processing unit. In the case where the denoising encoder in all denoising steps is taken as one model task and the denoising decoder in all denoising steps is taken as another model task, the denoising encoder and the denoising decoder are deployed on different processing units. Each model task corresponds to a processing unit.

[0098] According to an embodiment of the present disclosure, the first key value refers to a prepared KV parameter of the diffusion processing unit, and the first key value can also be understood as a kind of cache technology. When each diffusion processing unit executes the assigned model task at each denoising step, the previous calculation result can be reused by using the first key value to reduce the amount of repeated calculation and improve the processing speed of the diffusion processing model. For example, the first key value generated in the execution process of the diffusion processing model at the last denoising step is used when each diffusion processing unit executes the assigned model task at each denoising step. It is assumed that the diffusion processing model is divided into three model tasks and is assigned to three diffusion processing units for execution, and the forward process of the diffusion processing model includes denoising step 1, denoising step 2 and denoising step 3 in order. In denoising step 1, diffusion processing unit 1 executes the first model task of the diffusion processing model for the image, diffusion processing unit 2 executes the second model task of the diffusion processing model for the image, and diffusion processing unit 3 executes the third model task of the diffusion processing model for the image. Similarly, in denoising step 2 after denoising step 1, diffusion processing unit 1 executes the first model task of the diffusion processing model for the image, diffusion processing unit 2 executes the second model task of the diffusion processing model for the image, and diffusion processing unit 3 executes the third model task of the diffusion processing model for the image. Further, diffusion processing unit 1 is taken as an example for illustration: it is assumed that diffusion processing unit 1 generates first key value A in the process of executing the first model task of the diffusion processing model at denoising step 1, and diffusion processing unit 1 needs to use the first key value A in the process of executing the first model task of the diffusion processing model at denoising step 2. It is assumed that diffusion processing unit 1 generates first key value B in the process of executing the first model task of the diffusion processing model at denoising step 2, and diffusion processing unit 1 needs to use the first key value B in the process of executing the first model task of the diffusion processing model at denoising step 3.

[0099] According to an embodiment of the present disclosure, since the first key value generated in the diffusion processing model execution process of the last denoising step is needed when each diffusion processing unit performs the assigned model task in each denoising step, the first key value for the first denoising step of the forward process of the diffusion processing model is not available. Therefore, a number of warm-up steps can be added before the first denoising step to provide the first key value for the first denoising step of the forward process of the diffusion processing model. In the warm-up steps, the image is divided into a second number of warm-up patches for processing by a second number of warm-up processing units. The image refers to the input image of the diffusion processing model. As described above, it can be any image because the diffusion processing model has the ability to turn any image into the target image through step-by-step denoising. In some applications, the image can be a specified image. The warm-up patches are parts of the image. In an embodiment of the present disclosure, the image is divided into a second number of warm-up patches. As shown in FIG. 6, the second number is 2. The image is divided into warm-up patch 1 and warm-up patch 2. For example, a 480*800 image can be divided into 4 warm-up patches of 120*800.

[0100] It can be understood that, in order to provide the first key value for the first denoising step, it is necessary to ensure that the number of model tasks sequentially executed by the warm-up processing units in the warm-up steps and the number of model tasks sequentially executed by the diffusion processing units in the denoising steps are consistent to obtain the first key value corresponding to each model task. However, because each warm-up processing unit only processes a part of the warm-up patches, the key value corresponding to the model task obtained by each warm-up processing unit is only a part of the first key value, i.e., a first sub-key value. For example, the “each warm-up processing unit sequentially executes a first number of model tasks in the warm-up steps to obtain a first sub-key value corresponding to each model task” is described in combination with FIG. 7: assuming that the image is divided into 2 warm-up patches for processing by warm-up processing unit 1 and warm-up processing unit 2. For warm-up patch 1, warm-up processing unit 1 sequentially executes model task 1, model task 2, model task 3, and model task 4 on warm-up patch 1 in the warm-up steps to obtain a first sub-key value corresponding to model task 1, a first sub-key value corresponding to model task 2, a first sub-key value corresponding to model task 3, and a first sub-key value corresponding to model task 4. For warm-up patch 2, warm-up processing unit 2 sequentially executes model task 1, model task 2, model task 3, and model task 4 on warm-up patch 2 in the warm-up steps to obtain a first sub-key value corresponding to model task 1, a first sub-key value corresponding to model task 2, a first sub-key value corresponding to model task 3, and a first sub-key value corresponding to model task 4.

[0101] The specific process of step 410 will be described in detail hereinafter.

[0102] In step 420, the first sub-key value corresponding to the model task is transmitted to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes the first key value used in the first denoising step of the plurality of denoising steps by using the second number of first sub-key values generated by the second number of preheating processing units in the preheating step, so that the diffusion processing unit performs the diffusion processing model of the first denoising step.

[0103] According to an embodiment of the present disclosure, since the plurality of diffusion processing units need to use the first key value corresponding to the model task calculated in the preheating step when performing the model task in the first denoising step, each preheating processing unit needs to transmit the first sub-key value corresponding to each model task to the diffusion processing unit of the model task after obtaining the first sub-key value corresponding to each model task. For example, referring to FIG. 7, the preheating processing unit 1 and the preheating processing unit 2 respectively transmit the first sub-key value corresponding to the model task 1 to the diffusion processing unit 1 corresponding to the model task 1 after obtaining the first sub-key value corresponding to the model task 1. Similarly, the preheating processing unit 1 and the preheating processing unit 2 respectively transmit the first sub-key value corresponding to the model task 2 to the diffusion processing unit 2 corresponding to the model task 2 after obtaining the first sub-key value corresponding to the model task 2. The preheating processing unit 1 and the preheating processing unit 2 respectively transmit the first sub-key value corresponding to the model task 3 to the diffusion processing unit 3 corresponding to the model task 3 after obtaining the first sub-key value corresponding to the model task 3. The preheating processing unit 1 and the preheating processing unit 2 respectively transmit the first sub-key value corresponding to the model task 4 to the diffusion processing unit 4 corresponding to the model task 4 after obtaining the first sub-key value corresponding to the model task 4.

[0104] Further, the preheating processing unit can transmit the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task in a broadcast manner. The preheating processing unit can also transmit the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task in a peer-to-peer (P2P) communication manner.

[0105] According to the embodiments of the present disclosure, because the first sub-key value corresponding to the model task obtained by each preheating processing unit is only a part of the first key value, after receiving the first sub-key value corresponding to the model task sent by the preheating processing unit, each diffusion processing unit needs to synthesize the first sub-key value corresponding to the model task sent by the preheating processing unit to obtain the complete first key value. The diffusion processing unit can use the complete first key value to perform the corresponding model task in the first denoising step of the diffusion processing model. For example, referring to FIG. 8, after receiving the first sub-key value corresponding to the first model task sent by the preheating processing unit 1 and the preheating processing unit 2, the diffusion processing unit 1 synthesizes the first key value A used in the first denoising step to perform the first model task of the diffusion processing model using the first key value A. After receiving the first sub-key value corresponding to the second model task sent by the preheating processing unit 1 and the preheating processing unit 2, the diffusion processing unit 2 synthesizes the first key value B used in the first denoising step to perform the second model task of the diffusion processing model using the first key value B. After receiving the first sub-key value corresponding to the third model task sent by the preheating processing unit 1 and the preheating processing unit 2, the diffusion processing unit 3 synthesizes the first key value C used in the first denoising step to perform the third model task of the diffusion processing model using the first key value C. After receiving the first sub-key value corresponding to the fourth model task sent by the preheating processing unit 1 and the preheating processing unit 2, the diffusion processing unit 4 synthesizes the first key value D used in the first denoising step to perform the fourth model task of the diffusion processing model using the first key value D.

[0106] It can be understood that after determining the first sub-key value corresponding to the model task, the preheating processing unit transmits the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can synthesize the first key value corresponding to the model task using the plurality of first sub-key values corresponding to the model task to perform the denoising step of the diffusion processing model using the first key value. Referring to FIG. 9, the embodiments of the present disclosure hide the communication between the preheating processing unit and the diffusion processing unit (i.e., the preheating processing unit transmits the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task) in the preheating step and the denoising step, so that the denoising step can immediately follow the preheating step, greatly improving the efficiency of the diffusion processing.

[0107] According to the embodiments of the present disclosure, in order to verify that the processing efficiency of the diffusion processing model will be improved when using the diffusion processing method of the embodiments of the present disclosure, the embodiments of the present disclosure also measure the processing efficiency of the diffusion processing model through a plurality of experiments to obtain the processing efficiency of the diffusion processing model for different images. Referring to Table 1 as follows:

[0108] Table 1

[0109] Referring to Table 1, taking the pixels of the image as 1024px as an example, it can be obtained that, compared with the time delay of the diffusion processing model without setting the preheating step, the time delay of the diffusion processing model using the prior art is increased by 67%, and the time delay of the diffusion processing model using the diffusion processing method of the present disclosure is increased by 60% compared with the time delay of the diffusion processing model without setting the preheating step. Therefore, it can be determined that the time delay of the diffusion processing model using the diffusion processing method of the present disclosure is less than the time delay of the diffusion processing model using the prior art. Moreover, with the increase of the pixels of the diffusion image, the time delay of the diffusion processing model using the diffusion processing method of the present disclosure will be significantly reduced compared with the time delay of the diffusion processing model using the prior art. Therefore, it can be determined that, compared with the prior art of "setting as many preheating steps as diffusion processing units", the embodiment of the present disclosure can provide the preliminary overall KV parameters for multiple diffusion processing units in one preheating step, reduce the resource idling in the preheating process, and improve the efficiency of the entire diffusion processing.

[0110] In step 430, any other denoising step after the first denoising step in the plurality of denoising steps, the diffusion processing model is executed by each diffusion processing unit using the first key value generated by the previous denoising step of the other denoising step.

[0111] According to the embodiment of the present disclosure, in each denoising step after the first denoising step, the diffusion processing unit needs to use the first key value generated by the previous denoising step when executing the diffusion processing model to reuse the previous calculation result and reduce the repeated calculation amount. For example, assuming that in denoising step 1, for an image, diffusion processing unit 1 generates first key value 1 when executing the first model task of the diffusion processing model, diffusion processing unit 2 generates first key value 2 when executing the second model task of the diffusion processing model, diffusion processing unit 3 generates first key value 3 when executing the third model task of the diffusion processing model, and diffusion processing unit 4 generates first key value 4 when executing the fourth model task of the diffusion processing model. It can be determined that in denoising step 2 after denoising step 1, diffusion processing unit 1 needs to use first key value 1 when executing the first model task of the diffusion processing model, diffusion processing unit 2 needs to use first key value 2 when executing the second model task of the diffusion processing model, diffusion processing unit 3 needs to use first key value 3 when executing the third model task of the diffusion processing model, and diffusion processing unit 4 needs to use first key value 4 when executing the fourth model task of the diffusion processing model.

[0112] The embodiments of steps 410-430 do not set up a number of preheating steps equal to the number of diffusion processing units and obtain the preliminary overall KV parameters of the corresponding diffusion processing unit in each preheating step as in the prior art, but adopt the method of dividing the image into a second number of preheating blocks. The image is divided into a second number of preheating blocks and assigned to a second number of preheating processing units for processing. Each preheating processing unit can sequentially perform a plurality of model tasks in the same preheating step (only these model tasks are performed for a preheating block instead of the entire image, and it is impossible to directly perform a plurality of model tasks on the entire image in the same preheating step due to too heavy processing load). In this way, the KV parameters of a plurality of model tasks can be obtained in one preheating step and transmitted to a plurality of diffusion processing units corresponding to the plurality of model tasks. The diffusion processing units can obtain the respective KV parameters of the second number of preheating blocks from the second number of preheating processing units respectively, and integrate them to obtain the KV parameters of the entire image. Compared with the prior art in which one preheating step can only provide KV parameters for one diffusion processing unit, the embodiments of the present disclosure can provide preliminary overall KV parameters for a plurality of diffusion processing units in one preheating step, reduce resource idling in the preheating process, and improve the efficiency of the entire diffusion processing.

[0113] The above is a general description of steps 410-430. The specific implementation process of steps 410-430 is described in detail below.

[0114] Detailed description of step 410

[0115] Step 410, in the preheating step before a plurality of denoising steps, the image is divided into a second number of preheating blocks, which are assigned to a second number of preheating processing units for processing. Each preheating processing unit sequentially performs a first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each model task.

[0116] In an embodiment, the preheating steps are only one. In this case, each preheating processing unit sequentially executes the first number of model tasks in the only preheating step, which can achieve the purpose of obtaining the first sub-key value corresponding to multiple model tasks in one preheating step. For example, assuming that the image is divided into 2 preheating blocks, which are respectively allocated to 2 preheating processing units for processing, preheating processing unit 1 and preheating processing unit 2 sequentially execute 3 model tasks in the only preheating step. For preheating block 1, preheating processing unit 1 can sequentially obtain the first sub-key value corresponding to the first model task, the first sub-key value corresponding to the second model task, and the first sub-key value corresponding to the third model task in the preheating step. For preheating block 2, preheating processing unit 2 can sequentially obtain the first sub-key value corresponding to the first model task, the first sub-key value corresponding to the second model task, and the first sub-key value corresponding to the third model task in the preheating step. In the denoising step after the only preheating step, the diffusion processing unit 1 corresponding to the first model task can synthesize the first key value by using the first sub-key value corresponding to the first model task, and execute the first model task of the diffusion processing model by using the first key value. The diffusion processing unit 2 corresponding to the second model task can synthesize the first key value by using the first sub-key value corresponding to the second model task, and execute the second model task of the diffusion processing model by using the first key value. The diffusion processing unit 3 corresponding to the third model task can synthesize the first key value by using the first sub-key value corresponding to the third model task, and execute the third model task of the diffusion processing model by using the first key value.

[0117] In an embodiment, the preheating steps are multiple preheating steps, wherein the first number of model tasks are allocated to the multiple preheating steps, so that the number of model tasks allocated to each preheating step is the third number, and the third number in different preheating steps is not completely the same.

[0118] Referring to FIG. 10, step 410 includes:

[0119] Step 1010: In each preheating step, the third number of model tasks corresponding to the preheating step are executed, so that the first number of model tasks are executed in the multiple preheating steps.

[0120] Step 1010 is described in detail as follows:

[0121] According to an embodiment of the present disclosure, the third number refers to the number of model tasks allocated to each preheating step. The third number in different preheating steps can not be completely the same, and the sum of the number of model tasks in the multiple preheating steps is the first number.

[0122] For example, referring to FIG. 11A, it is assumed that two preheating steps are included before the denoising step, the diffusion processing model is divided into four model tasks, and the four model tasks are allocated to the two preheating steps, such that the model tasks allocated to the preheating step 1 are model task 1, and the model tasks allocated to the preheating step 2 are model task 2, model task 3, and model task 4. It can be determined that, in the preheating step 1, each preheating processing unit executes the model task 1, and in the preheating step 2 after the preheating step 1, each preheating processing unit sequentially executes the model task 2, the model task 3, and the model task 4.

[0123] According to another embodiment of the present disclosure, in the case where a plurality of preheating steps are included before the denoising step, the third number in different preheating steps can also be completely the same, and at this time, the sum of the number of model tasks in the plurality of preheating steps is still the first number.

[0124] For example, referring to FIG. 11B, it is assumed that two preheating steps are included before the denoising step, the diffusion processing model is divided into four model tasks, and the four model tasks are allocated to the two preheating steps, such that the model tasks allocated to the preheating step 1 are model task 1 and model task 2, and the model tasks allocated to the preheating step 2 are model task 3 and model task 4. It can be determined that, in the preheating step 1, each preheating processing unit executes the model task 1 and the model task 2, and in the preheating step 2 after the preheating step 1, each preheating processing unit sequentially executes the model task 3 and the model task 4.

[0125] The embodiment of the above step 1010 allocates the first number of model tasks to the plurality of preheating steps, such that the number of model tasks allocated to each preheating step is the third number, so as to execute the third number of model tasks corresponding to each preheating step in each preheating step. By making the preheating processing unit execute the model tasks in batches in turn, the risk of excessive load caused by too many model tasks executed in a short time is reduced.

[0126] In an embodiment, the first number of model tasks is in order, and referring to FIG. 12, the first number of model tasks in step 410 is allocated to the plurality of preheating steps in the following manner:

[0127] Step 1210, obtaining the task size of each model task;

[0128] Step 1220, merging a part of sequentially adjacent model tasks in each of a plurality of first candidate merging schemes, so that the number of generated merged task blocks is equal to the number of preheating steps;

[0129] Step 1230, calculating the total task size of each merged task block in each first candidate merging scheme based on the task size, and calculating the first variance of the total task size of each merged task block;

[0130] Step 1240, the first candidate merging scheme with the first minimum variance is determined as the target merging scheme, and a part of sequentially adjacent model tasks are merged into one preheating step based on the target merging scheme.

[0131] The first candidate merging scheme refers to a scheme of merging a part of sequentially adjacent model tasks, so that the number of generated merged task blocks is equal to the number of preheating steps. The merged task block refers to a task block generated by merging a part of sequentially adjacent model tasks by the first candidate merging scheme. The target merging scheme refers to the merging scheme of the sequentially adjacent model tasks finally used for dividing the model tasks, which is the first candidate merging scheme with the first minimum variance.

[0132] The steps 1210 to 1240 are described in detail as follows:

[0133] In step 1210, the task size of each model task is obtained.

[0134] According to an embodiment of the present disclosure, the diffusion model includes a plurality of denoising steps, and each denoising step includes at least two mutual attention models of each denoising decoder. Each mutual attention model includes a plurality of layers. In order to reasonably allocate model tasks and balance the processing burden of each preheating processing unit, the model tasks need to be divided to the level of layers. The diffusion processing model can be divided into layers first, and then some adjacent layers are merged to aggregate model tasks. Further, the task size of each model task refers to the size of the layers aggregated by the model task, and the size of the layers aggregated by the model task can be the layer parameter size of the layers aggregated by the model task.

[0135] For example, assuming that the diffusion processing model includes layer 1, layer 2, layer 3, layer 4 and layer 5, layer 1 and layer 2 can be aggregated into model task 1, and layer 3, layer 4 and layer 5 can be aggregated into model task 2. The task size of model task 1 depends on the layer parameter size of layer 1 and layer 2, and the task size of model task 2 depends on the layer parameter size of layer 3, layer 4 and layer 5.

[0136] In step 1220, a part of sequentially adjacent model tasks are merged by a plurality of first candidate merging schemes, so that the number of generated merged task blocks is equal to the number of preheating steps.

[0137] According to an embodiment of the present disclosure, there can be many layers in the diffusion processing model, and accordingly, there can be many model tasks in the diffusion processing model, and in most cases, the first number is greater than the number of warm-up steps. At this time, a part of the sequentially adjacent model tasks need to be merged so that the number of generated merged task blocks is equal to the number of warm-up steps. There are many merging schemes for this merging. Each scheme that merges sequentially adjacent model tasks so that the number of generated merged tasks is equal to the number of warm-up steps is called a first candidate merging scheme.

[0138] For example, referring to FIG. 13, it is assumed that the diffusion processing model includes model task 1, model task 2, model task 3, and model task 4, and the task size of model task 1 is A, the task size of model task 2 is B, the task size of model task 3 is C, and the task size of model task 4 is D. It is assumed that the number of warm-up steps is 2, and a part of the sequentially adjacent model tasks can be merged to obtain a plurality of first candidate merging schemes, and the number of merged task blocks in the first candidate merging scheme is equal to the number of warm-up steps. Among them, in the first candidate merging scheme A, the model tasks are assigned to warm-up step 1, and sequentially adjacent model tasks 2, 3, and 4 are merged and assigned to warm-up step 2. In the first candidate merging scheme B, sequentially adjacent model tasks 1 and 2 are merged and assigned to warm-up step 1, and sequentially adjacent model tasks 3 and 4 are merged and assigned to warm-up step 2. In the first candidate merging scheme C, sequentially adjacent model tasks 1, 2, and 3 are merged and assigned to warm-up step 1, and model task 4 is assigned to warm-up step 2.

[0139] In step 1230, based on the task size, the total task size of each merged task block in each first candidate merging scheme is calculated, and the first variance of the total task size of each merged task block is calculated.

[0140] According to an embodiment of the present disclosure, the total task size of the merged task block refers to the sum of the sizes of all model tasks merged in the merged task block. The first variance of the total task size of each merged task block refers to the difference between the total task sizes of different merged task blocks, and the first variance can reflect whether the total task size of each merged task block in each first candidate merging scheme is balanced.

[0141] For example, it is assumed that in the plurality of first candidate merging schemes of FIG. 13, the first variance of the total task size of each merged task block in the first candidate merging scheme A is 400Kb 2 , the first variance of the total task size of each merged task block in the first candidate merging scheme B is 1296Kb 2 , and the first variance of the total task size of each merged task block in the first candidate merging scheme C is 225Kb.2 Thus, the balance degree of each merged task block in the first candidate merging scheme C is determined to be > the balance degree of each merged task block in the first candidate merging scheme A > the balance degree of each merged task block in the first candidate merging scheme B.

[0142] In step 1240, the first candidate merging scheme with the minimum first variance is determined as the target merging scheme, and a part of sequentially adjacent model tasks are merged into one pre-warming step based on the target merging scheme.

[0143] According to an embodiment of the present disclosure, the target merging scheme refers to the merging scheme of sequentially adjacent model tasks finally used for dividing the model tasks. The first minimum variance means that the total task size of each merged task block in the first candidate merging scheme is the most balanced. Determining the first candidate merging scheme with the minimum first variance as the target merging scheme can make the computational amount of model tasks in each pre-warming step relatively balanced, which is beneficial to dispersing the processing load of each pre-warming processing unit.

[0144] For example, assuming that the first variance of the total task size of each merged task block in the first candidate merging scheme A in FIG. 13 is 400Kb 2 , the first variance of the total task size of each merged task block in the first candidate merging scheme B is 1296Kb 2 , and the first variance of the total task size of each merged task block in the first candidate merging scheme C is 225Kb 2 Thus, the first candidate merging scheme C can be determined as the target merging scheme, and sequentially adjacent model task 1, model task 2 and model task 3 are merged and assigned to pre-warming step 1, and model task 4 is assigned to pre-warming step 2.

[0145] The embodiments of the above steps 1210 to 1240 can select the target merging scheme according to the principle that the first variance of the total task size of each merged task block in the first candidate merging scheme is minimum, so that the computational amount of model tasks in each pre-warming step is relatively balanced, which is beneficial to dispersing the processing load of each pre-warming processing unit and improving resource utilization.

[0146] In an embodiment, the specific implementation of assigning the first number of model tasks to the plurality of pre-warming steps in step 410 has been described above, and the way of determining the number of pre-warming steps is described here. Referring to FIG. 14, the number of pre-warming steps is determined in the following way:

[0147] In step 1410, the model size of the diffusion processing model is obtained.

[0148] In step 1420, the diffusion processing complexity is obtained.

[0149] Step 1430, determining the number of preheating steps based on the model size and the diffusion processing complexity.

[0150] The steps 1410 to 1430 are described in detail as follows:

[0151] In step 1410, the model size of the diffusion processing model is obtained.

[0152] According to an embodiment of the present disclosure, the diffusion processing model includes a plurality of denoising steps, and each denoising step includes a plurality of mutual attention models, etc. There can be multiple layers in the mutual attention model, and each layer has some layer parameters. The model size of the diffusion processing model can be understood as the sum of the sizes of the layer parameters of the mutual attention models in the denoising steps included in the diffusion processing model. The model size of the diffusion processing model can be obtained by querying the size of the diffusion processing model in the object interface.

[0153] Moreover, the model size of the diffusion processing model is directly proportional to the number of preheating steps. The larger the model size of the diffusion processing model, the more complex the diffusion processing model is, and the first number of model tasks in the diffusion processing model can be more, and more preheating steps need to be set.

[0154] In step 1420, the diffusion processing complexity is obtained.

[0155] According to an embodiment of the present disclosure, the diffusion processing complexity refers to the complexity of the target image to be generated. Although the target image has not been generated, it can be basically determined whether the target image is complex according to the target image description describing the target image. Moreover, the diffusion processing complexity is directly proportional to the number of preheating steps. The larger the diffusion processing complexity, the more complex the target image to be generated is, and the more preheating steps need to be set.

[0156] The diffusion processing complexity is mainly determined by the diffusion post-image element number and the diffusion post-image element relationship information, etc. Specifically, the diffusion post-image element number refers to the number of elements in the diffusion target description information, and the diffusion post-image element relationship information refers to the information of the interaction relationship between the elements in the diffusion target description information. The specific method of obtaining the diffusion processing complexity based on these factors will be described in detail later.

[0157] In step 1430, the number of preheating steps is determined based on the model size and the diffusion processing complexity.

[0158] According to embodiments of this disclosure, both the model size and the diffusion processing complexity are proportional to the number of preheating steps. A larger model size results in greater diffusion processing complexity, indicating a more complex target image to be generated and a higher computational load on the preheating processing unit. To minimize the computational load on the preheating processing unit and prevent it from malfunctioning due to excessive computational load in a short period, a larger number of preheating steps is required when the number of preheating processing units remains constant.

[0159] For example, suppose there exists a diffusion processing model A and a diffusion processing model B, where the model size of diffusion processing model A is larger than that of diffusion processing model B, and the diffusion processing complexity of diffusion processing model A is greater than that of diffusion processing model B. Therefore, it can be determined that the number of preheating steps for diffusion processing model A is greater than the number of preheating steps for diffusion processing model B. By determining a larger number of preheating steps for diffusion processing model A, the computational load on the preheating processing units used to execute the preheating steps of diffusion processing model A is reduced, avoiding the situation where too much computational load is concentrated in a single preheating step.

[0160] The specific method for "determining the number of preheating steps based on model size and diffusion processing complexity" will be described in detail below.

[0161] The embodiments of steps 1410 to 1430 above quantify the processing difficulty of the diffusion processing model by obtaining the model size and diffusion processing complexity of the diffusion processing model, and determining the number of preheating steps based on the model size and diffusion processing complexity of the diffusion processing model, thereby improving the accuracy of determining the number of preheating steps.

[0162] In one embodiment, referring to FIG15, step 1420 includes:

[0163] Step 1510: Obtain diffusion target description information;

[0164] Step 1520: Based on the diffusion target description information, obtain the number of elements in the diffused image and the relationship information between the elements in the diffused image;

[0165] Step 1530: Determine the first sub-complexity based on the number of elements in the diffused image;

[0166] Step 1540: Determine the second sub-complexity based on the element relationship information of the diffused image;

[0167] Step 1550: Determine the diffusion processing complexity based on the first and second sub-complexities.

[0168] The first sub-complexity refers to a complexity determined based on the number of image elements after diffusion, and the number of image elements after diffusion is directly proportional to the first sub-complexity. The first sub-complexity can be obtained according to the number of image elements after diffusion by means of table lookup or formula substitution.

[0169] The second sub-complexity refers to a complexity determined based on the relationship information of image elements after diffusion, and the relationship information of image elements after diffusion is directly proportional to the second sub-complexity. The second sub-complexity can be obtained according to the relationship information of image elements after diffusion by means of table lookup or formula substitution.

[0170] The steps 1510 to 1550 are described in detail as follows:

[0171] In step 1510, the diffusion target description information is obtained.

[0172] According to an embodiment of the present disclosure, the diffusion target description information refers to a textual description of a target image to be generated. For example, referring to FIG. 16A, the diffusion target description information can be “There is a white cloud in the blue sky, and there is a house below, and there is a tree beside the house”. Further, the diffusion target description information can be input through an object interface, for example, the object enters the voice by using the voice, so that the terminal converts the voice of the object into the diffusion target description information. Moreover, the diffusion target description information can also be obtained from a specified part of the corpus, for example, a picture of a novel is generated according to a specified paragraph of the novel.

[0173] In step 1520, the number of image elements after diffusion and the relationship information of image elements after diffusion are obtained based on the diffusion target description information.

[0174] According to an embodiment of the present disclosure, the number of image elements after diffusion refers to the number of elements in the diffusion target description information. Referring to FIG. 16A, the diffusion target description information is “There is a white cloud in the blue sky, and there is a house below, and there is a tree beside the house”, it can be determined that the elements in the diffusion target description information include “blue sky”, “white cloud”, “house” and “tree”, and further it is determined that the number of image elements after diffusion is 4. Referring to FIG. 16B, the diffusion target description information is “There is a white cloud in the blue sky, and there is a house below, and there is a tree beside the house, and there is a person under the tree, and the person holds a bag in his hand”, it can be determined that the elements in the diffusion target description information include “blue sky”, “white cloud”, “house”, “tree”, “person” and “bag”, and further it is determined that the number of image elements after diffusion is 6.

[0175] The post-diffusion image element relationship information refers to the information of the interaction between the elements in the diffusion target description information. Referring to FIG. 16A, the diffusion target description information is "there is a white cloud in the blue sky, there is a house below the white cloud, and there is a tree beside the house", and it can be determined that the post-diffusion image element information includes "there is a white cloud in the blue sky", "there is a house below the white cloud", and "there is a tree beside the house". Referring to FIG. 16B, the diffusion target description information is "there is a white cloud in the blue sky, there is a house below the white cloud, there is a tree beside the house, there is a person below the tree, and the person holds a bag in his hand", and it can be determined that the post-diffusion image element relationship information includes "there is a white cloud in the blue sky", "there is a house below the white cloud", "there is a tree beside the house", and "the person holds a bag in his hand". It can be obtained that the post-diffusion image element relationship information of FIG. 16B is more complex than that of FIG. 16A.

[0176] In step 1530, the first sub-complexity is determined based on the post-diffusion image element number.

[0177] According to an embodiment of the present disclosure, the post-diffusion image element number and the first sub-complexity are directly proportional. Based on the post-diffusion image element number, there can be multiple implementation manners.

[0178] In one implementation manner, a lookup table can be used. The post-diffusion image element number is used to look up a pre-set post-diffusion image element number and first sub-complexity corresponding relationship table to obtain the first sub-complexity corresponding to the post-diffusion image element number. Table 2 below is an actual example of the post-diffusion image element number and first sub-complexity corresponding relationship table:

[0179] Table 2

[0180] For example, the post-diffusion image element number is 15, and the first sub-complexity is obtained by looking up Table 2 as 8. The lookup table is simple and easy to implement, and improves the efficiency of determining the diffusion processing complexity.

[0181] In another implementation manner, the first sub-complexity can also be determined by substituting the formula based on the post-diffusion image element number. The post-diffusion image element number is substituted into a pre-set post-diffusion image element number and first sub-complexity formula to obtain the first sub-complexity corresponding to the post-diffusion image element number.

[0182] For example, it is assumed that the pre-set post-diffusion image element number and first sub-complexity formula is: first sub-complexity = post-diffusion image element number / 2. For example, the post-diffusion image element number is 10, and the first sub-complexity can be determined by the pre-set post-diffusion image element number and first sub-complexity formula as 5.

[0183] For example, assuming that the preset formula is C1=0.5N1+10, where C1 represents the first sub-complexity and N1 represents the number of image elements after diffusion, the number of image elements after diffusion N1 is substituted into the formula to obtain the first sub-complexity C1 corresponding to the number of image elements after diffusion. For example, if N1=20, then C1=0.5×20+10=20.

[0184] In step 1540, the second sub-complexity is determined based on the image element relationship information after diffusion.

[0185] According to an embodiment of the present disclosure, to more accurately measure the complexity of the image element relationship information after diffusion, the image element relationship information after diffusion can be quantified first. Assuming that the image element relationship information after diffusion includes "there is a white cloud in the blue sky", "there is a house under the white cloud", and "there is a tree beside the house", the image element relationship information after diffusion can be quantified as 3. Assuming that the image element relationship information after diffusion includes "there is a white cloud in the blue sky", "there is a house under the white cloud", "there is a tree beside the house", and "a person holds a bag in his hand", the image element relationship information after diffusion can be quantified as 4.

[0186] It can be understood that the image element relationship information after diffusion is proportional to the second sub-complexity. There are multiple implementation manners for determining the second sub-complexity based on the image element relationship information after diffusion.

[0187] In one implementation manner, a lookup table can be used. The preset correspondence table between the image element relationship information after diffusion and the second sub-complexity is looked up according to the image element relationship information after diffusion to obtain the second sub-complexity corresponding to the image element relationship information after diffusion. Table 3 below is an actual example of the correspondence table between the image element relationship information after diffusion and the second sub-complexity:

[0188] Table 3

[0189] For example, the image element relationship information after diffusion is 8, and the second sub-complexity is obtained by looking up Table 3 as 8. The lookup table is simple and easy to implement, and improves the efficiency of determining the diffusion processing complexity.

[0190] In another implementation manner, the second sub-complexity can also be determined by substituting the image element relationship information after diffusion into a preset formula between the image element relationship information after diffusion and the second sub-complexity to obtain the second sub-complexity corresponding to the image element relationship information after diffusion.

[0191] For example, assuming that the pre-set formula of the post-diffusion image element relationship information and the second sub-complexity is: second sub-complexity = post-diffusion image element relationship information * 5, if the post-diffusion image element relationship information is 4, the second sub-complexity can be determined as 20 according to the pre-set formula of the post-diffusion image element relationship information and the second sub-complexity.

[0192] For example, assuming that the pre-set formula is C2 = 0.3R1 + 5, where C2 represents the second sub-complexity and R1 represents the post-diffusion image element relationship information (here, it is assumed that R1 is a quantified value, for example, obtained by quantifying the complexity of the element relationship through an algorithm), the second sub-complexity C2 corresponding to the post-diffusion image element relationship information can be obtained by substituting the post-diffusion image element relationship information R1 into the formula. For example, if R1 = 10, then C2 = 0.3 * 10 + 5 = 8.

[0193] In step 1550, the diffusion processing complexity is determined based on the first sub-complexity and the second sub-complexity.

[0194] According to embodiments of the present disclosure, the diffusion processing complexity can be determined based on the first sub-complexity and the second sub-complexity in various ways.

[0195] In one implementation, the sum of the first sub-complexity and the second sub-complexity can be determined as the diffusion processing complexity. For example, assuming that the first sub-complexity is 5 and the second sub-complexity is 10, the diffusion processing complexity can be determined as 15.

[0196] In another implementation, the average of the first sub-complexity and the second sub-complexity can be determined as the diffusion processing complexity. For example, assuming that the first sub-complexity is 10 and the second sub-complexity is 20, the diffusion processing complexity can be determined as 15.

[0197] In another implementation, different weights can be set for the first sub-complexity and the second sub-complexity, and the weighted sum of the first sub-complexity and the second sub-complexity can be determined as the diffusion processing complexity. Assuming that the formula is C = w1C1 + w2C2, where C represents the diffusion processing complexity, C1 represents the first sub-complexity, C2 represents the second sub-complexity, w1 and w2 are the weights of the first sub-complexity and the second sub-complexity, respectively, and w1 + w2 = 1. For example, assuming that the first sub-complexity is 5, the second sub-complexity is 10, the weight of the first sub-complexity is 0.2, and the weight of the second sub-complexity is 0.3, the diffusion processing complexity can be determined as 4.

[0198] The embodiments of steps 1510 to 1550 above determine the first sub-complexity based on the number of elements in the diffused image, determine the second sub-complexity based on the relationship information of the elements in the diffused image, and determine the diffusion processing complexity based on the first and second sub-complexities. This approach considers both the number of elements in the diffusion target description information and the interaction relationships between these elements, determining the diffusion processing complexity from multiple dimensions. This is more accurate than determining it from a single dimension and improves the efficiency of the diffusion processing.

[0199] In one embodiment, referring to FIG17, step 1430 includes:

[0200] Step 1710: Determine the first score based on the model size;

[0201] Step 1720: Determine the second score based on the diffusion processing complexity;

[0202] Step 1730: Determine the number of preheating steps based on the first score and the second score.

[0203] The first score is a quantitative indicator used to measure the influence of the diffusion treatment model size on the number of preheating steps. It is related to the model size of the diffusion treatment model; the larger the model size, the higher the first score usually is. The first score is determined based on the model size of the diffusion treatment model using a pre-defined table or formula that correlates model size with the first score, and then combined with other factors to determine the number of preheating steps.

[0204] The second score is a quantitative indicator used to measure the impact of diffusion process complexity on the number of preheating steps. It is related to diffusion process complexity; the higher the diffusion process complexity, the higher the second score typically is. The second score can be determined based on the diffusion process complexity using a pre-defined table or formula that establishes a correspondence between diffusion process complexity and the second score. This second score, combined with the first score, determines the number of preheating steps.

[0205] Steps 1710 to 1730 are described in detail below:

[0206] In step 1710, the first score is determined based on the model size.

[0207] According to embodiments of this disclosure, determining the first score based on the model size can be implemented in various ways.

[0208] In one implementation, a lookup table can be used. The model size is searched against a pre-defined table mapping model size to the first score to obtain the corresponding first score. Table 4 below shows a practical example of this mapping:

[0209] Table 4

[0210] For example, the model size is 15 GB, and the first score is 30 according to the lookup table 4. The lookup table is simple and easy to implement, and greatly reduces the processing overhead.

[0211] In another implementation, the first score is determined based on the model size, and a formula can also be used. The model size is substituted into a pre-set formula of model size and first score to obtain the first score corresponding to the model size.

[0212] For example, it is assumed that the pre-set formula of model size and first score is: first score = model size / 1 GB. It is assumed that the model size is 8 GB, and the first score can be determined by the pre-set formula of model size and first score to obtain the first score of 8.

[0213] For example, it is assumed that the pre-set formula is S1 = 0.2M1 + 3, where S1 represents the first score, and M1 represents the model size (unit: MB). The model size M1 is substituted into the formula to obtain the first score S1 corresponding to the model size. For example, if M1 = 50, then S1 = 0.2x50 + 3 = 13.

[0214] In step 1720, the second score is determined based on the diffusion processing complexity.

[0215] According to an embodiment of the present disclosure, the second score is determined based on the diffusion processing complexity, and there can be various implementation manners.

[0216] In one implementation, a lookup table can be used. A pre-set correspondence table of diffusion processing complexity and second score is looked up according to the diffusion processing complexity to obtain the second score corresponding to the diffusion processing complexity.

[0217] In another implementation, the second score is determined based on the diffusion processing complexity, and a formula can also be used. The diffusion processing complexity is substituted into a pre-set formula of diffusion processing complexity and second score to obtain the second score corresponding to the diffusion processing complexity.

[0218] For example, the diffusion processing complexity C d is substituted into a pre-set formula S2 = C d *2, where S2 represents the second score, to obtain the second score corresponding to the diffusion processing complexity.

[0219] For example, it is assumed that the pre-set formula is S2 = 0.1C d +2, where S2 represents the second score, and C d represents the diffusion processing complexity. The diffusion processing complexity C dSubstitute the formula, the diffusion processing complexity corresponding to the second score S2 is obtained. For example, if C d = 28, S2 = 0.1 x 28 + 2 = 4.8.

[0220] In step 1730, the number of preheating steps is determined based on the first score and the second score.

[0221] It can be understood that determining the number of preheating steps based on the first score and the second score includes: determining a third score based on the first score and the second score; determining the number of preheating steps based on the third score.

[0222] The third score is a quantitative index obtained by comprehensively considering the first score and the second score, and is used to finally determine the number of preheating steps. It is obtained by summing, averaging or weighted summing the first score and the second score, and reflects the combined effect of the model size of the diffusion processing model and the diffusion processing complexity on the number of preheating steps.

[0223] According to embodiments of the present disclosure, there are various implementation manners for determining the third score based on the first score and the second score.

[0224] In one implementation manner, the sum of the first score and the second score can be determined as the third score. For example, assuming that the first score is 50 and the second score is 30, the third score can be determined as 80.

[0225] In another implementation manner, the average of the first score and the second score can be determined as the third score. For example, the average of the first score S1 and the second score S2 can be determined as the third score S3, i.e. S3 = (S1 + S2) / 2. For example, assuming that the first score is 50 and the second score is 30, the third score can be determined as 40.

[0226] In another implementation manner, different weights can be set for the first score and the second score, and the weighted sum of the first score and the second score can be determined as the third score. Assuming that the formula is S3 = w s1 S1 + w s2 S2, where S3 represents the third score, S1 represents the first score, S2 represents the second score, w s1 and w s2 are the weights of the first score and the second score respectively, and w s1 +w s2 = 1. For example, assuming that the first score is 50, the second score is 30, the weight of the first score is 0.2, and the weight of the second score is 0.3, the third score can be determined as 19.

[0227] According to embodiments of the present disclosure, there are various implementation manners for determining the number of preheating steps based on the third score.

[0228] In an implementation, the third fraction is used to look up a pre-set correspondence table of the third fraction and the number of pre-warming steps to obtain the number of pre-warming steps corresponding to the third fraction.

[0229] In another implementation, the number of pre-warming steps is determined based on the third fraction. The formula can also be used. The third fraction is substituted into the pre-set formula of the third fraction and the number of pre-warming steps to obtain the number of pre-warming steps corresponding to the third fraction.

[0230] For example, assuming the pre-set formula is N p = 0.5S3+5, where N p represents the number of pre-warming steps, S3 represents the third fraction, and the third fraction S3 is substituted into the formula to obtain the number of pre-warming steps N p corresponding to the third fraction. For example, if S3 = 10.54, then N p = 0.5x10.54+5 = 10.27 = 10 (rounded).

[0231] For another example, the third fraction S3 can be substituted into the pre-set formula of the third fraction and the number of pre-warming steps N p = S3 / 10 (the result is rounded up) to obtain the number of pre-warming steps corresponding to the third fraction.

[0232] The above embodiments of steps 1710 to 1730 determine the first fraction based on the model size, determine the second fraction based on the diffusion processing complexity, and finally determine the number of pre-warming steps based on the first fraction and the second fraction. The above method comprehensively considers various factors affecting the number of pre-warming steps, thereby improving the accuracy of determining the number of pre-warming steps.

[0233] Detailed description of step 420

[0234] In step 420, the first sub-key value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit uses the second number of first sub-key values generated by the second number of pre-warming processing units in the pre-warming step to synthesize the first key value used in the first denoising step of the plurality of denoising steps, so that the first denoising step executes the diffusion processing model.

[0235] In an embodiment, referring to FIG. 18, the second number in step 420 is determined as follows:

[0236] Step 1810, obtaining the image size of the image;

[0237] Step 1820, obtaining the service type for diffusion processing;

[0238] Step 1830, obtaining a target diffusion processing efficiency;

[0239] Step 1840, determining a second number based on the image size, the service type, and the target diffusion processing efficiency.

[0240] The steps 1810 to 1840 are described in detail as follows:

[0241] In step 1810, an image size of an image is obtained.

[0242] According to an embodiment of the present disclosure, the image refers to an input image of a diffusion processing model, and the image size refers to the size of the input image of the diffusion processing model. The image size affects the number of pre-warming partitions obtained after the image is divided. The larger the size of the image, the greater the amount of computation when the diffusion processing method is subsequently executed. At this time, the image needs to be divided into more pre-warming partitions, and the pre-warming partitions are distributed to multiple pre-warming processing units for execution, so as to balance the computation amount of each pre-warming processing unit and avoid that a certain pre-warming processing unit bears too much computational load.

[0243] In step 1820, a service type for diffusion processing is obtained.

[0244] According to an embodiment of the present disclosure, the service type of diffusion processing refers to the application scenario of the diffusion processing model, and the service type of diffusion processing includes a text image scene, a molecule detection scene, and an image processing scene, etc. The complexity required by diffusion processing corresponding to different service types of diffusion processing is also different.

[0245] For example, when the service type of diffusion processing is the application scenario of molecule detection, the accuracy of generating a molecule image is required to be very high, and a higher processing complexity than the ordinary text image service type is required, and the generated molecule image needs to have a higher pixel. Based on this, the diffusion image needs to be divided into more pre-warming partitions, and the pre-warming partitions are distributed to multiple pre-warming processing units for execution, so as to adapt to the accuracy requirement of the service type.

[0246] In step 1830, a target diffusion processing efficiency is obtained.

[0247] According to an embodiment of the present disclosure, the target diffusion processing efficiency refers to the processing efficiency of the image expected by the object. The shorter the processing time of the image expected by the object, the higher the target diffusion processing efficiency, and the longer the processing time of the image expected by the object, the lower the target diffusion processing efficiency. Further, if there is a higher requirement for the target diffusion processing efficiency, the diffusion image can be divided into more pre-warming partitions, so that multiple pre-warming processing units process multiple pre-warming partitions in parallel, so as to achieve the purpose of improving the target diffusion processing efficiency.

[0248] In step 1840, the second number is determined based on the image size, the service type, and the target diffusion processing efficiency.

[0249] It can be understood that determining the second number based on the image size, the service type, and the target diffusion processing efficiency includes: determining a fourth score based on the image size; determining a fifth score based on the service type, and determining a sixth score based on the target diffusion processing efficiency; determining a seventh score based on the fourth score, the fifth score, and the sixth score, and determining the second number based on the seventh score.

[0250] The fourth score is a quantitative index for measuring the influence degree of the image size on the number of pre-warming partitions (i.e., the second number) of the image. It is related to the image size, and the larger the image size, the higher the fourth score is generally. The fourth score can be determined according to the image size by using a pre-set image size and fourth score corresponding relationship table or formula, and then the second number is determined in combination with other factors.

[0251] The fifth score is a quantitative index for measuring the influence degree of the service type of diffusion processing on the number of pre-warming partitions (i.e., the second number) of the image. Different diffusion processing service types (such as a text-to-image scene, a molecule detection scene, and an image processing scene, etc.) correspond to different fifth scores, which reflect the differences in the requirements of processing complexity and accuracy of the service type, and then affect the determination of the second number.

[0252] The sixth score is a quantitative index for measuring the influence degree of the target diffusion processing efficiency on the number of pre-warming partitions (i.e., the second number) of the image. It is related to the target diffusion processing efficiency, and the higher the target diffusion processing efficiency, the higher the sixth score is generally. The sixth score can be determined according to the target diffusion processing efficiency by using a pre-set target diffusion processing efficiency and sixth score corresponding relationship table or formula, and then the second number is determined in combination with other factors.

[0253] The seventh score is a quantitative index obtained by comprehensively considering the fourth score, the fifth score, and the sixth score, and is used to finally determine the number of pre-warming partitions (i.e., the second number) of the image. It is obtained by summing, averaging, or weighted summing the fourth score, the fifth score, and the sixth score, and reflects the influence of the image size, the service type, and the target diffusion processing efficiency on the second number.

[0254] According to an embodiment of the present disclosure, the fourth score can be determined based on the image size in various ways.

[0255] In one implementation, a lookup table can be used. According to the image size, a pre-set image size and fourth score corresponding relationship table is looked up to obtain the fourth score corresponding to the image size.

[0256] In another implementation, the fourth score can also be determined by substituting into a formula based on the image size. The image size is substituted into a pre-set formula of image size and fourth score to obtain the fourth score corresponding to the image size.

[0257] For example, the image size I1 (unit: MB) can be substituted into a pre-set formula of image size and fourth score S4 = I1 / 2 MB, where S4 represents the fourth score, to obtain the fourth score corresponding to the image size.

[0258] For another example, assuming that the pre-set formula is S4 = 0.01I1 + 1, where S4 represents the fourth score and I1 represents the image size (unit: KB), the image size I1 is substituted into the formula to obtain the fourth score S4 corresponding to the image size. For example, if I1 = 200, then S4 = 0.01 x 200 + 1 = 3.

[0259] According to an embodiment of the present disclosure, the fifth score can be determined based on the service type, and there can be various implementation manners.

[0260] In one implementation, a lookup table can be used. A pre-set correspondence table of service type and fifth score is looked up according to the service type to obtain the fifth score corresponding to the service type.

[0261] In another implementation, the fifth score can also be determined by substituting into a formula based on the service type. The service type is substituted into a pre-set formula of service type and fifth score to obtain the fifth score corresponding to the service type.

[0262] For example, if the service type T is a text-to-image scene, S5 = 8; if the service type T is a molecule detection scene, S5 = 12; and if the service type T is an image processing scene, S5 = 10, where S5 represents the fifth score, to obtain the fifth score corresponding to the service type.

[0263] For another example, assuming that the pre-set formula is S5 = 2T1 + 2, where S5 represents the fifth score and T1 represents a quantized value of the service type, the quantized value of the service type T1 is substituted into the formula to obtain the fifth score S5 corresponding to the service type. For example, if T1 = 2, then S5 = 2 x 2 + 2 = 6.

[0264] According to an embodiment of the present disclosure, the sixth score can be determined based on the target diffusion processing efficiency, and there can be various implementation manners.

[0265] In one implementation, a lookup table can be used. A pre-set correspondence table of target diffusion processing efficiency and sixth score is looked up according to the target diffusion processing efficiency to obtain the sixth score corresponding to the target diffusion processing efficiency.

[0266] In another implementation, the sixth fraction can also be determined by substituting the target diffusion processing efficiency into a formula. The target diffusion processing efficiency is substituted into a formula pre-set for the target diffusion processing efficiency and the sixth fraction to obtain the sixth fraction corresponding to the target diffusion processing efficiency.

[0267] For example, the target diffusion processing efficiency E (unit: image / second) can be substituted into a formula pre-set for the target diffusion processing efficiency and the sixth fraction S6=E1*3, where S6 represents the sixth fraction, to obtain the sixth fraction corresponding to the target diffusion processing efficiency.

[0268] For another example, assume that the formula pre-set is S6=0.3E1+1, where S6 represents the sixth fraction and E1 represents the target diffusion processing efficiency (unit: times / second), the target diffusion processing efficiency E1 is substituted into the formula to obtain the sixth fraction S6 corresponding to the target diffusion processing efficiency. For example, if E1=10, then S6=0.3*10+1=4.

[0269] According to an embodiment of the present disclosure, the seventh fraction can be determined based on the fourth fraction, the fifth fraction and the sixth fraction in various ways.

[0270] In one implementation, the sum of the fourth fraction, the fifth fraction and the sixth fraction can be determined as the seventh fraction. For example, assume that the fourth fraction is 10, the fifth fraction is 20 and the sixth fraction is 30, the seventh fraction can be determined as 60.

[0271] In another implementation, the average of the fourth fraction, the fifth fraction and the sixth fraction can be determined as the seventh fraction. For example, assume that the fourth fraction is 50, the fifth fraction is 30 and the sixth fraction is 40, the seventh fraction can be determined as 40.

[0272] In another implementation, different weights can be set for the fourth fraction, the fifth fraction and the sixth fraction respectively, and the weighted sum of the fourth fraction, the fifth fraction and the sixth fraction can be determined as the seventh fraction. Assume that the formula is S7=w s4 S4+w s5 S5+w s6 S6, where S7 represents the seventh fraction, S4 represents the fourth fraction, S5 represents the fifth fraction, S6 represents the sixth fraction, and w s4 , w s5 and w s6 are the weights of the fourth fraction, the fifth fraction and the sixth fraction respectively, and w s4 +w s5 +w s6= 1. For example, assuming that the fourth score is 50, the fifth score is 30, the sixth score is 40, the weight of the fourth score is 0.2, the weight of the fifth score is 0.3, and the weight of the sixth score is 0.1, the seventh score can be determined as 23.

[0273] According to an embodiment of the present disclosure, the second number can be determined based on the seventh score, and there can be various implementation manners.

[0274] In one implementation manner, a lookup table can be used to look up a pre-set corresponding relationship table of the seventh score and the second number according to the seventh score, so as to obtain the second number corresponding to the seventh score.

[0275] In another implementation manner, the second number can be determined based on the seventh score. A formula can also be used. The seventh score is substituted into a pre-set formula of the seventh score and the second number, so as to obtain the second number corresponding to the seventh score.

[0276] The second number can be determined based on the seventh score in a linear mapping manner. Assuming that the formula is N2 = a x F7 + b, where N2 represents the second number, and a and b are constants, which can be determined through experiments or experience. For example, if a = 0.5, b = 10, and the seventh score F7 = 20, then the second number N2 = 0.5 x 20 + 10 = 20.

[0277] The embodiments of the above steps 1810 to 1840 can determine the second number based on the size of the image, the service type, and the target processing efficiency, comprehensively consider various factors affecting the second number, and improve the accuracy of determining the second number.

[0278] In one embodiment, the first sub-key value is embodied as a first sub-matrix corresponding to a preheating block, and the first key value is embodied as a first matrix corresponding to the image.

[0279] Referring to FIG. 19, the step 420 of “synthesizing the first key value used in the first denoising step in the plurality of denoising steps by using the second number of first sub-key values generated by the second number of preheating processing units in the preheating step” includes:

[0280] The step 1910 integrates the first sub-matrix corresponding to each preheating block generated by each preheating processing unit into the first matrix according to the order of the preheating blocks.

[0281] The step 1910 is described in detail as follows:

[0282] According to an embodiment of the present disclosure, the first matrix refers to the form of the first key value. The first key value can be embodied as the first matrix corresponding to the image, and each element in the first matrix is equivalent to a sampling value of the image.

[0283] The first sub-matrix refers to the form of the first sub-key value. After the image is divided into the second number of pre-warming blocks, the first sub-key value obtained by the pre-warming processing unit processing the pre-warming block can be embodied as the first sub-matrix corresponding to the pre-warming block, and each element in the first sub-matrix is equivalent to the sampling value in the pre-warming block corresponding to the first sub-matrix.

[0284] It can be understood that the first key value used in the first denoising step can be synthesized by using the second number of first sub-key values generated by the second number of pre-warming processing units in the pre-warming step, and the first sub-matrix can also be naturally integrated into the first matrix. It can be envisaged that the division order of the pre-warming block when the image is divided into the second number of pre-warming blocks is the order of integrating the first sub-matrix into the first matrix.

[0285] The integration operation can be performed according to the dimension and structure of the matrix. If the first sub-matrix is a two-dimensional matrix and the number of columns of the first sub-matrix is the same, the sub-matrix can be spliced together by row. Assuming that the second number is n, and the first sub-matrix generated by the i-th pre-warming processing unit is M i , the size of which is h i ×w (height × width), then the size of the integrated first matrix M is , that is, M = [M n ](The semicolon here represents splicing by row). If the number of rows of the sub-matrix is the same, the sub-matrix can be spliced by column. If the dimensions of the sub-matrix are inconsistent, the dimensions can be adjusted first, for example, by interpolation or dimension reduction, so that the dimensions match before integration.

[0286] For example, referring to FIG. 20A, assuming that when the image is divided into the second number of pre-warming blocks, the image is divided into four equal parts from top to bottom to obtain pre-warming block 1, pre-warming block 2, pre-warming block 3, and pre-warming block 4. Correspondingly, referring to FIG. 20B, after obtaining the first sub-matrix 1 corresponding to the pre-warming block 1 generated by the pre-warming processing unit 1, the first sub-matrix 2 corresponding to the pre-warming block 2 generated by the pre-warming processing unit 2, the first sub-matrix 3 corresponding to the pre-warming block 3 generated by the pre-warming processing unit 3, and the first sub-matrix 4 corresponding to the pre-warming block 4 generated by the pre-warming processing unit 4, the first sub-matrix 1, the first sub-matrix 2, the first sub-matrix 3, and the first sub-matrix 4 need to be integrated according to the division order of the pre-warming block 1, the pre-warming block 2, the pre-warming block 3, and the pre-warming block 4 in FIG. 20A to obtain the first matrix.

[0287] The embodiment of the step 1910 integrates the first sub-matrices respectively generated by the second number of preheating processing units and corresponding to the preheating sub-blocks into the first matrix according to the preheating sub-block order, considers the division order when the image is divided into the preheating sub-blocks when integrating the first matrix, improves the integration accuracy, and ensures the integrity of the determined first matrix.

[0288] In an embodiment, the preheating step is a plurality of preheating steps.

[0289] Referring to FIG. 21, the step 420 of “delivering the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task” includes:

[0290] The step 2110 delivers the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task in the order of the model task.

[0291] The step 2110 is described in detail as follows:

[0292] According to an embodiment of the present disclosure, in the case where there are a plurality of preheating steps, for each preheating sub-block, the preheating processing unit sequentially obtains the first sub-key value corresponding to each model task and delivers the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task. Further, the preheating processing unit delivers the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task, and there are a plurality of implementation manners. The first implementation manner is that in the last preheating step of the plurality of preheating steps, the first sub-key value corresponding to the model task is sequentially delivered to the diffusion processing unit corresponding to the model task in the order of the model task.

[0293] It can be understood that in the last preheating step of the plurality of preheating steps, the preheating processing unit has determined the first sub-key value corresponding to the plurality of model tasks in the sequence of the diffusion processing model. Since the diffusion processing unit first executes the model task in the sequence of the diffusion processing model, and then sequentially executes other model tasks, in order to ensure that the diffusion processing unit can timely synthesize the first key value corresponding to the model task in the sequence of the diffusion processing model, and utilize the first key value to execute the model task, the preheating processing unit needs to sequentially deliver the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task in the order of the model task. Based on this, the diffusion processing unit can sequentially synthesize the first key value corresponding to the model task in the order of the model task, and utilize the first key value to execute the model task.

[0294] Further, referring to FIG. 22, it can be determined that the preheating processing unit has not obtained the first sub-key value corresponding to all model tasks of the diffusion processing model at the beginning of the last preheating step of the plurality of preheating steps. However, the preheating processing unit has obtained the first sub-key value corresponding to all model tasks of the diffusion processing model at the end of the last preheating step. After obtaining the first sub-key value corresponding to all model tasks of the diffusion processing model, the preheating processing unit sends the first sub-key value corresponding to the model task that has not been sent to the diffusion processing unit, so that the diffusion processing unit can synthesize the first key value by using the first sub-key value before performing the denoising step of the diffusion processing model. It can be understood that, by making the preheating step and the delivery of the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task be performed in the same time period and in sequence, the embodiments of the present disclosure do not need to wait until the end of the last preheating step before delivering the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task, thereby reducing the time for the diffusion processing model to process the image and improving the execution efficiency of the diffusion processing model.

[0295] According to another embodiment of the present disclosure, the second implementation of “the preheating processing unit delivers the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task” is that, after the end of each preheating step of the plurality of preheating steps, the preheating processing unit delivers the first sub-key value corresponding to the model task allocated to the preheating step to the diffusion processing unit corresponding to the model task.

[0296] It can be understood that, since each preheating step will be allocated to a part of the model tasks in the case of multiple preheating steps, the preheating processing unit can obtain the first sub-key value corresponding to the model task allocated to each preheating step at the end of each preheating step. In order to avoid the situation that the communication pressure surges due to the centralized sending of the first sub-key values of all model tasks, the preheating processing unit can deliver the first sub-key value corresponding to the model task allocated to the preheating step to the diffusion processing unit corresponding to the model task at the end of each preheating step. By delivering the first sub-key values corresponding to multiple model tasks in batches, the communication pressure between the preheating processing unit and the diffusion processing unit can be alleviated.

[0297] For example, referring to FIG. 7, it is assumed that the preheating process includes three preheating steps, in the preheating step 1, the preheating processing unit executes the model task 1, in the preheating step 2, the preheating processing unit sequentially executes the model task 2 and the model task 3, and in the preheating step 3, the preheating processing unit executes the model task 4. It can be obtained that after the preheating step 1 ends, the preheating processing unit transmits the first sub-key value corresponding to the model task 1 to the diffusion processing unit 1. After the preheating step 2 ends, the preheating processing unit transmits the first sub-key value corresponding to the model task 2 to the diffusion processing unit 2, and transmits the first sub-key value corresponding to the model task 3 to the diffusion processing unit 3. After the preheating step 3 ends, the preheating processing unit transmits the first sub-key value corresponding to the model task 4 to the diffusion processing unit 4.

[0298] The embodiment of the above step 2110 transmits the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task in sequence according to the order of the model task in the last preheating step of the plurality of preheating steps, so that the preheating step and the transmission of the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task are performed in sequence in the same time period. Without waiting for the last preheating step to end before transmitting the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task, the time for the diffusion processing model to process the image is reduced, and the execution efficiency of the diffusion processing model is improved.

[0299] It can be understood that the above has described the first two embodiments of “the preheating processing unit transmitting the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task”, and the third embodiment of “the preheating processing unit transmitting the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task” is described here:

[0300] In an embodiment, each model task starts to be executed in a different denoising sub-step of the first denoising step;

[0301] Referring to FIG. 23, the step 420 of “transmitting the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task” includes:

[0302] The step 2310 of acquiring the first size of the first sub-key value;

[0303] The step 2320 of predicting the first transmission time of the first sub-key value based on the first size;

[0304] The step 2330 of acquiring the execution start time of the denoising sub-step corresponding to the first sub-key value;

[0305] The step 2340 of determining the transmission start time based on the execution start time and the first transmission time;

[0306] Step 2350, at the transmission start time, the first sub-key value corresponding to the model task is transmitted to the diffusion processing unit corresponding to the model task.

[0307] The steps 2310 to 2350 are described in detail as follows:

[0308] In step 2310, the first size of the first sub-key value is obtained.

[0309] According to an embodiment of the present disclosure, the first size refers to the size of the first sub-key value, and the first size affects the time used by the preheating processing unit to transmit the first sub-key value to the diffusion processing unit. The larger the first size, the longer the time used by the preheating processing unit to transmit the first sub-key value to the diffusion processing unit, and the smaller the first size, the shorter the time used by the preheating processing unit to transmit the first sub-key value to the diffusion processing unit. Further, the first size of the first sub-key value can be obtained by querying the first sub-key value in the object interface.

[0310] In step 2320, the first transmission time of the first sub-key value is predicted based on the first size.

[0311] According to an embodiment of the present disclosure, the first transmission time refers to the time used by the preheating processing unit to transmit the first sub-key value to the diffusion processing unit. It can be envisaged that the first size and the first transmission time are directly proportional, and therefore the first transmission time can be predicted based on the first size.

[0312] It can be understood that there are various implementation manners for predicting the first transmission time of the first sub-key value based on the first size.

[0313] In one implementation manner, a lookup table can be used to look up a pre-set correspondence table of the first size and the first transmission time according to the first size, to obtain the first transmission time corresponding to the first size.

[0314] In another implementation manner, the first transmission time of the first sub-key value is predicted based on the first size. A formula can also be used. The first size is substituted into a pre-set formula of the first size and the first transmission time to obtain the first transmission time corresponding to the first size.

[0315] Suppose the pre-set formula is T=k×S, where T represents the first transmission time (unit: ms), S represents the first size (unit: MB), and k is a transmission coefficient, the value of which depends on the bandwidth, delay and other factors of the transmission network, and can be obtained by experimental measurement. Substituting the first size into the formula can obtain the first transmission time corresponding to the first size. For example, if k=2 and the first size S=20MB, then the first transmission time T=2×20=40ms.

[0316] In step 2330, the execution start time of the de-noising sub-step corresponding to the first sub-key value pair is obtained.

[0317] According to an embodiment of the present disclosure, each de-noising step includes a plurality of de-noising sub-steps, and the de-noising sub-step is the minimum time unit of the inverse process of the diffusion processing unit. The first number of model tasks can be allocated to the plurality of de-noising sub-steps for execution, so that the diffusion processing unit can execute the model task corresponding to the de-noising sub-step in each de-noising sub-step. It can be understood that because in each de-noising sub-step, the diffusion processing unit needs to execute the model task corresponding to the de-noising sub-step. In the de-noising sub-step of the first de-noising step, the diffusion processing unit needs to use the first key value of the model task when executing the model task, and the first key value of the model task is integrated by the first sub-key value corresponding to the model task transmitted by the plurality of preheating processing units. The execution start time of the de-noising sub-step corresponding to the first sub-key value is the execution start time of the model task corresponding to the first sub-key value.

[0318] For example, referring to FIG. 24, it is assumed that the de-noising step includes de-noising sub-step 1, de-noising sub-step 2, de-noising sub-step 3, and de-noising sub-step 4, and the diffusion processing unit starts to execute model task 1 in de-noising sub-step 1, starts to execute model task 2 in de-noising sub-step 2, starts to execute model task 3 in de-noising sub-step 3, and starts to execute model task 4 in de-noising sub-step 4. The diffusion processing unit needs to use the first key value A integrated by the first sub-key value A when executing the model task 1, so the start time of the de-noising sub-step 1 is determined as the execution start time of the de-noising sub-step corresponding to the first sub-key value A. Similarly, the diffusion processing unit needs to use the first key value B integrated by the first sub-key value B when executing the model task 2, so the start time of the de-noising sub-step 2 is determined as the execution start time of the de-noising sub-step corresponding to the first sub-key value B. The diffusion processing unit needs to use the first key value C integrated by the first sub-key value C when executing the model task 3, so the start time of the de-noising sub-step 3 is determined as the execution start time of the de-noising sub-step corresponding to the first sub-key value C. The diffusion processing unit needs to use the first key value D integrated by the first sub-key value D when executing the model task 4, so the start time of the de-noising sub-step 4 is determined as the execution start time of the de-noising sub-step corresponding to the first sub-key value D.

[0319] In step 2340, the transmission start time is determined based on the execution start time and the first transmission time.

[0320] According to an embodiment of the present disclosure, the transmission start time refers to the time when the preheating processing unit starts to transmit the first sub-key value. After the execution start time indicating that the first sub-key value is to be used and the first transmission time indicating the time when the preheating processing unit transmits the first sub-key value to the diffusion processing unit are determined, the transmission start time can be determined as the difference between the execution start time and the first transmission time.

[0321] For example, referring to FIG. 24, after the first transmission time of the first sub-key value A and the execution start time of the de-noising sub-step corresponding to the first sub-key value A are determined, the transmission start time of the first sub-key value A can be determined. Similarly, after the first transmission time of the first sub-key value B and the execution start time of the de-noising sub-step corresponding to the first sub-key value B are determined, the transmission start time of the first sub-key value B can be determined. After the first transmission time of the first sub-key value C and the execution start time of the de-noising sub-step corresponding to the first sub-key value C are determined, the transmission start time of the first sub-key value C can be determined. After the first transmission time of the first sub-key value D and the execution start time of the de-noising sub-step corresponding to the first sub-key value D are determined, the transmission start time of the first sub-key value D can be determined.

[0322] In step 2350, at the transmission start time, the first sub-key value corresponding to the model task is transmitted to the diffusion processing unit corresponding to the model task.

[0323] According to an embodiment of the present disclosure, after determining the transmission start time, the first sub-key value corresponding to the model task needs to be transmitted to the diffusion processing unit corresponding to the model task at the transmission start time. For example, referring to FIG. 24, it is assumed that the preheating processing unit 1, the preheating processing unit 2, the diffusion processing unit 1, the diffusion processing unit 2, the diffusion processing unit 3, and the diffusion processing unit 4 are included. The diffusion processing unit 1 is used to execute the first model task, the diffusion processing unit 2 is used to execute the second model task, the diffusion processing unit 3 is used to execute the third model task, and the diffusion processing unit 4 is used to execute the fourth model task. Further, the diffusion processing unit 1 needs to use the first key value A integrated from the first sub-key value A when executing the first model task, the diffusion processing unit 2 needs to use the first key value B integrated from the first sub-key value B when executing the second model task, the diffusion processing unit 3 needs to use the first key value C integrated from the first sub-key value C when executing the third model task, and the diffusion processing unit 4 needs to use the first key value D integrated from the first sub-key value D when executing the fourth model task. In the transmission process, the preheating processing unit 1 and the preheating processing unit 2 transmit the first sub-key value A corresponding to the first model task obtained by themselves to the diffusion processing unit 1 at the transmission start time of the first sub-key value A, so that the diffusion processing unit 1 can receive the first sub-key value A at the beginning of the denoising sub-step 1. The preheating processing unit 1 and the preheating processing unit 2 transmit the first sub-key value B corresponding to the second model task obtained by themselves to the diffusion processing unit 2 at the transmission start time of the first sub-key value B, so that the diffusion processing unit 2 can receive the first sub-key value B at the beginning of the denoising sub-step 2. The preheating processing unit 1 and the preheating processing unit 2 transmit the first sub-key value C corresponding to the third model task obtained by themselves to the diffusion processing unit 3 at the transmission start time of the first sub-key value C, so that the diffusion processing unit 3 can receive the first sub-key value C at the beginning of the denoising sub-step 3. The preheating processing unit 1 and the preheating processing unit 2 transmit the first sub-key value D corresponding to the fourth model task obtained by themselves to the diffusion processing unit 4 at the transmission start time of the first sub-key value D, so that the diffusion processing unit 4 can receive the first sub-key value D at the beginning of the denoising sub-step 4.

[0324] The embodiments of the above steps 2310 to 2350 can determine the transmission start time of the first sub-key value based on the execution start time of the denoising sub-step corresponding to the first sub-key value and the first transmission time of the first sub-key value, so as to transmit the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task at the transmission start time. By transmitting the first sub-key values corresponding to multiple model tasks in batches through the above steps, the situation that the communication pressure increases sharply due to the centralized transmission of the first sub-key values of all model tasks is avoided, and the communication pressure between the preheating processing unit and the diffusion processing unit is effectively relieved.

[0325] Detailed description of step 430

[0326] At step 430, at any other denoising step after the first denoising step in the plurality of denoising steps, the diffusion processing model is executed by the respective diffusion processing unit using the first key value generated by the previous denoising step of the other denoising step.

[0327] In an embodiment, the respective model task is executed starting from a different denoising sub-step of the other denoising step;

[0328] Referring to FIG. 25, in order to improve the accuracy of the diffusion processing when the model tasks are executed in different denoising sub-steps, the related sub-key values of the current denoising step need to be obtained, specifically, step 430 comprises:

[0329] Step 2510, at any other denoising step after the first denoising step in the plurality of denoising steps, for a single diffusion processing unit, obtaining a first current sub-key value generated by a first model task corresponding to the single diffusion processing unit in the other denoising step, and a first previous sub-key value generated by a previous model task before the first model task in the other denoising step;

[0330] Step 2520, for a single diffusion processing unit, extracting a first subsequent sub-key value corresponding to a subsequent model task after the first model task from the first key value generated by the previous denoising step of the other denoising step;

[0331] Step 2530, combining the first previous sub-key value, the first current sub-key value, and the first subsequent sub-key value into a first synthesized key value, and executing the respective model task by the respective diffusion processing unit using the first synthesized key value.

[0332] The following describes step 2510 and step 2530 in detail:

[0333] According to an embodiment of the present disclosure, the “respective model task is executed starting from a different denoising sub-step of the other denoising step” is described in conjunction with FIG. 26: it is assumed that a denoising step includes three denoising sub-steps, and three sequentially executed model tasks are assigned to the three denoising sub-steps respectively. It can be obtained that in denoising sub-step 1-1, diffusion processing unit 1 executes the first model task for image block 1. In denoising sub-step 1-2, diffusion processing unit 2 executes the second model task for image block 1, and diffusion processing unit 1 executes the first model task for image block 2. In denoising sub-step 3, diffusion processing unit 3 executes the third model task for image block 1, diffusion processing unit 2 executes the second model task for image block 2, and diffusion processing unit 1 executes the first model task for image block 3.

[0334] In step 2510, for each diffusion processing unit, the first current sub-key value generated by the first model task corresponding to the diffusion processing unit in the other de-noising step, and the first precedent sub-key value generated by the precedent model task before the first model task in the other de-noising step are obtained.

[0335] According to an embodiment of the present disclosure, the first model task refers to the model task to which the diffusion processing unit is assigned. The precedent model task refers to the model task whose execution sequence is before the first model task. The first current sub-key value refers to the key value generated by the diffusion processing unit when the first model task is executed in the other de-noising step, and the first precedent sub-key value refers to the key value generated by the other diffusion processing unit when the precedent model task is executed in the other de-noising step. The other de-noising step refers to the de-noising step after the first de-noising step.

[0336] For example, the processing procedure of step 2510 is further described in detail in combination with FIG. 26. Assuming that the de-noising step 2 has three de-noising sub-steps, i.e., de-noising sub-step 2-1, de-noising sub-step 2-2, and de-noising sub-step 2-3. For the diffusion processing unit 2, the first current sub-key value generated by the first model task corresponding to the diffusion processing unit 2 in the de-noising sub-step 2-2, and the first precedent sub-key value generated by the precedent model task (i.e., the model task executed by the diffusion processing unit 1) before the first model task in the de-noising sub-step 2-1 are obtained.

[0337] In step 2520, for each diffusion processing unit, the first subsequent sub-key value corresponding to the subsequent model task after the first model task is extracted from the first key value generated in the previous de-noising step of the other de-noising step.

[0338] According to an embodiment of the present disclosure, the subsequent model task refers to the model task whose execution sequence is after the first model task. The first subsequent sub-key value refers to the key value generated by the other diffusion processing unit when the subsequent model task is executed in the previous de-noising step of the other de-noising step.

[0339] For example, the processing procedure of step 2520 is further described in detail in combination with FIG. 26. Assuming that the other de-noising step is the de-noising step 2, for the diffusion processing unit 2, the first subsequent sub-key value generated by the subsequent model task (i.e., the model task executed by the diffusion processing unit 3) after the first model task in the de-noising sub-step 1-3 is extracted from the first key value generated in the previous de-noising step (de-noising step 1) of the de-noising step 2.

[0340] In step 2530, the first precedent sub-key value, the first current sub-key value, and the first subsequent sub-key value are combined into a first synthesized key value, and each model task is executed by each diffusion processing unit using the first synthesized key value.

[0341] According to an embodiment of the present disclosure, the first synthetic key value refers to a key value obtained by integrating the first preceding sub-key value, the first current sub-key value, and the first subsequent sub-key value.

[0342] The synthetic operation can be in the form of splicing, assuming that the first preceding sub-key value is KV pre , the first current sub-key value is KV cur , the first subsequent sub-key value is KV post , and they are all in vector form. If the dimensions of these key values are the same, they can be spliced together in order, i.e., KV 合成 = [KV pre , KV cur , KV post ]. If the dimensions of the key values are different, they can be first adjusted in dimension, such as being converted into the same dimension through linear transformation before being spliced. The synthetic operation can also be in the form of weighted summation, assuming that the weights are w pre , w cur , w post , and w pre +w cur +w post = 1, then KV 合成 = w pre ×KV pre +w cur ×KV cur +w post ×KV post . The values of the weights can be adjusted according to the importance of different model tasks.

[0343] For example, the process of step 2530 is described in further detail in connection with FIG. 26. Assuming that the other de-noising step is de-noising step 2, for diffusion processing unit 2, it has been determined that the first current sub-key value generated by the first model task corresponding to diffusion processing unit 2 in de-noising sub-step 2-2, the first previous sub-key value generated by the previous model task corresponding to diffusion processing unit 1 in de-noising sub-step 2-1, and the first subsequent sub-key value generated by the subsequent model task corresponding to diffusion processing unit 3 in de-noising sub-step 1-3. Based on this, the first current sub-key value, the first previous sub-key value, and the first subsequent sub-key value can be integrated to obtain an integrated key value. Since the first integrated key value integrates the key values corresponding to multiple model tasks, after the first integrated key value is determined, each diffusion processing unit can use the first integrated key value to perform each model task. Further, in conventional techniques, the key value generated by the previous de-noising step is generally used for diffusion processing, which is not conducive to the accuracy of diffusion processing. Since the first current sub-key value and the first previous sub-key value in the embodiments of the present disclosure are generated by the current de-noising step, using the first integrated key value obtained by integrating the first current sub-key value and the first previous sub-key value for diffusion processing can achieve better convergence effect and improve the accuracy of diffusion processing.

[0344] The specific method of “performing each model task by each diffusion processing unit using the first integrated key value” will be described in detail below.

[0345] The embodiments of steps 2510 to 2530 integrate the first current sub-key value, the first previous sub-key value, and the first subsequent sub-key value to obtain an integrated key value, i.e., the first integrated key value. Since the first integrated key value integrates the key values corresponding to multiple model tasks, after the first integrated key value is determined, each diffusion processing unit can use the first integrated key value to perform each model task. Further, since the first current sub-key value and the first previous sub-key value are generated by the current de-noising step, using the first integrated key value obtained by integrating the first current sub-key value and the first previous sub-key value for diffusion processing can achieve better convergence effect and improve the accuracy of diffusion processing compared with using the key value generated by the previous de-noising step for diffusion processing.

[0346] In an embodiment, referring to FIG. 27, step 2530 includes:

[0347] Step 2710, dividing the image into a fourth number of image patches, the fourth number being equal to the number of de-noising sub-steps in the de-noising step, and the fourth number being greater than or equal to the first number;

[0348] Step 2720, the first number of denoising sub-steps in the fourth number of denoising sub-steps are respectively taken as denoising sub-steps of the first number of diffusion processing units for processing the first image block, wherein the diffusion processing units respectively utilize the first synthesis key value to process the second image block to the last image block of the fourth number after processing the first image block by the fifth number of denoising sub-steps, to perform each model task, wherein the fifth number is the fourth number minus 1.

[0349] The steps 2710 and 2720 are described in detail as follows:

[0350] In step 2710, the image is divided into a fourth number of image blocks, the fourth number being equal to the number of denoising sub-steps in the denoising step, and the fourth number being greater than or equal to the first number.

[0351] According to an embodiment of the present disclosure, the fourth number is the number of image blocks obtained after the image is divided. The number of denoising sub-steps in the denoising step is equal to the fourth number.

[0352] The division of the image blocks can be similar to the division of the preheating blocks, that is, uniform division or non-uniform division according to the texture, color and other characteristics of the image. For example, for an image with obvious texture changes, the image can be divided along the direction of the texture to ensure that the texture in each image block is relatively consistent, which helps to improve the accuracy of subsequent processing.

[0353] For example, referring to FIG. 28A, the image can be divided into 4 image blocks, the fourth number is 4, and the number of denoising sub-steps in the denoising step is also 4. Assuming that the first number is 3, that is, 3 model tasks are assigned to 3 diffusion processing units for execution. Referring to FIG. 28B, for image block 1, diffusion processing unit 1 performs the first model task at denoising sub-step 1-1, diffusion processing unit 2 performs the second model task at denoising sub-step 1-2, and diffusion processing unit 3 performs the third model task at denoising sub-step 1-3. For image block 2, diffusion processing unit 1 performs the first model task at denoising sub-step 1-2, diffusion processing unit 2 performs the second model task at denoising sub-step 1-3, and diffusion processing unit 3 performs the third model task at denoising sub-step 1-4. For image block 3, diffusion processing unit 1 performs the first model task at denoising sub-step 1-3, diffusion processing unit 2 performs the second model task at denoising sub-step 1-4, and diffusion processing unit 3 performs the third model task at denoising sub-step 2-1.

[0354] In step 2720, the first number of denoising sub-steps in the fourth number of denoising sub-steps are respectively used as the first number of diffusion processing units to process the denoising sub-steps of the first image block. In the fifth number of denoising sub-steps after processing the first image block, the diffusion processing unit uses the first synthesis key value to process the second image block to the last image block of the fourth number, so as to execute each model task. The fifth number is the fourth number minus 1.

[0355] The fifth number here refers to the number of denoising sub-steps remaining after the diffusion processing unit has processed the first image block, which can be used to process subsequent image blocks. Its value is the difference between the fourth number and 1. That is, after processing the first image block, the diffusion processing unit uses the first synthesis key value to process the second image block to the last image block of the fourth number in the fifth number of denoising sub-steps to perform each model task.

[0356] For example, referring to Figures 28A and 28B, step 2720 is illustrated here: Assume the image can be divided into four image blocks, so the fourth number is four, and the number of denoising sub-steps in the denoising step is also four. When the first number is three, the three model tasks are assigned to three diffusion processing units for execution. In step 2720, for denoising step 1, the first three of the four denoising sub-steps of denoising step 1 are used as denoising sub-steps for the first image block by the three diffusion processing units. Furthermore, referring to Figure 28, for diffusion processing unit 1, after processing image block 1 (i.e., performing the first model task on image block 1), the three denoising sub-steps can respectively use the first synthetic key value to process image block 2, image block 3, and image block 4. Similarly, for diffusion processing unit 2, after processing image block 1 (i.e., performing the second model task on image block 1), the three denoising sub-steps can respectively utilize the first synthetic key value to process image block 2, image block 3, and image block 4. For diffusion processing unit 3, after processing image block 1 (i.e., performing the third model task on image block 1), the three denoising sub-steps can respectively utilize the first synthetic key value to process image block 2, image block 3, and image block 4. For diffusion processing unit 4, after processing image block 1 (i.e., performing the fourth model task on image block 1), the three denoising sub-steps can respectively utilize the first synthetic key value to process image block 2, image block 3, and image block 4.

[0357] The specific method of “processing the second image block to the last image block of the fourth number using the first synthetic key value respectively” will be described in detail below.

[0358] In the embodiments described in steps 2710 to 2720, the image is divided into a fourth number of image blocks. The first first number of denoising sub-steps in the fourth number of denoising sub-steps are used as the first number of diffusion processing units to process the denoising sub-steps of the first image block. Through this step, each diffusion processing unit only needs to store the parameters of the part of the model task it is responsible for processing when processing the first image block, which greatly reduces the storage space occupied compared to the scheme of storing all parameters of the diffusion model. Furthermore, in the fifth number of denoising sub-steps after processing the first image block, the diffusion processing unit uses the first synthesized key value to process the second to the last image blocks of the fourth number of denoising sub-steps. By using the first synthesized key value to process the image blocks, compared with using the key value generated by the previous denoising step for diffusion processing, better convergence effect can be achieved and the accuracy of diffusion processing can be improved.

[0359] In one embodiment, referring to FIG29, step 2620 includes:

[0360] Step 2910: Obtain diffusion target description information;

[0361] Step 2920: Generate a diffusion guidance vector based on the diffusion target description information;

[0362] Step 2930: Guided by the diffusion guiding vector, process the second image block to the last image block of the fourth number using the first synthetic key value.

[0363] Steps 2910 to 2930 are described in detail below:

[0364] In step 2910, the diffusion target description information is obtained.

[0365] According to the embodiments of this disclosure, the "diffusion target description information" has been explained in step 1510, and will not be repeated here.

[0366] In step 2920, a diffusion guidance vector is generated based on the diffusion target description information.

[0367] According to embodiments of this disclosure, there are multiple ways to generate a diffusion guidance vector based on diffusion target description information.

[0368] In one implementation, the diffusion target description information is input into an embedding layer, which then outputs a diffusion guidance vector. For example, the embedding layer can be a fully connected neural network with 128 neurons and a ReLU activation function. Specifically, the diffusion target description information is first converted into a format suitable for the embedding layer, such as converting text into a sequence of word indices. This sequence is then input into the embedding layer, which maps each word index to a corresponding embedding vector. This vector is then processed through a linear transformation and the ReLU activation function of the fully connected layer, ultimately outputting the diffusion guidance vector.

[0369] In another implementation, the diffusion target description information is segmented into words. Each word's corresponding word vector is obtained by searching for its corresponding word vector. These word vectors are then concatenated in the order of the segmented words to obtain the diffusion guidance vector. The word segmentation of the diffusion target description information can be performed using a Word2Vec pre-trained model as the word vector library. Specifically, first, a suitable word segmentation tool is used to segment the diffusion target description information, resulting in a word list. Then, for each word in the word list, its corresponding word vector is searched in the Word2Vec pre-trained model. If a word does not exist in the word vector library, a default vector or an approximate vector can be generated using interpolation or other methods. Finally, the found word vectors are concatenated in the order of the segmented words to obtain the diffusion guidance vector.

[0370] By converting the diffusion target description information into vector form, it can be quickly identified and processed by the diffusion processing model, and can be quickly adapted to the diffusion processing model at a relatively low cost, thereby improving the diffusion processing efficiency of the diffusion processing model.

[0371] According to another embodiment of this disclosure, in molecular graph modeling applications, various properties of molecules and their chemical reactions with other substances can be obtained, thereby generating diffusion steering vectors. In time series modeling applications, the constraints of time series modeling can be obtained, thereby generating diffusion steering vectors. These are just a few examples; further details are omitted here.

[0372] In step 2930, guided by the diffusion guide vector, the second image block to the last image block of the fourth number are processed using the first synthetic key value.

[0373] According to embodiments of this disclosure, after determining the diffusion guidance vector, the diffusion guidance vector can guide each diffusion processing unit to process multiple image blocks sequentially. For example, referring to FIG28B, a diffusion guidance vector is first generated based on the diffusion target description information. Guided by the diffusion guidance vector, in denoising sub-step 1-1, diffusion processing unit 1, after processing image block 1, can process image block 2, image block 3, and image block 4 respectively using the first synthetic key value in the three denoising sub-steps (i.e., denoising sub-steps 1-2, 1-3, and 1-4). Similarly, guided by the diffusion guidance vector, in denoising sub-step 1-2, diffusion processing unit 2, after processing image block 1, can process image block 2, image block 3, and image block 4 respectively using the first synthetic key value in the three denoising sub-steps (i.e., denoising sub-steps 1-3, 1-4, and 2-1). Guided by the diffusion guiding vector, in the denoising sub-steps 1-3, the diffusion processing unit 3 can process image block 2, image block 3 and image block 4 respectively using the first synthetic key value in the three denoising sub-steps (i.e., denoising sub-steps 1-4, denoising sub-steps 2-1 and denoising sub-steps 2-2) after processing image block 1.

[0374] The embodiments of steps 2910 to 2930 above convert the diffusion target description information into vector form, which can be quickly identified and processed by the diffusion processing model. This allows for rapid adaptation to the diffusion processing model at a relatively low cost, thereby improving the diffusion processing efficiency of the diffusion processing model.

[0375] Detailed implementation diagrams of the diffusion processing method according to embodiments of this disclosure.

[0376] The implementation details of the diffusion processing method according to embodiments of this disclosure will now be described in detail with reference to FIG30.

[0377] As described above, in a processor architecture with a CPU and a GPU, the CPU is responsible for instruction scheduling, and each GPU can be the smallest hardware unit that executes the model task, i.e., a processing unit. The scheduler in Figure 30 can refer to the CPU. The preheating processing unit 1 and the diffusion processing unit 2 can be GPUs.

[0378] In step 3010, the image and diffusion target description information are received.

[0379] In step 3021, a diffusion guidance vector is generated based on the diffusion target description information.

[0380] In step 3022, in the preheating step before multiple denoising steps, the image is divided into a second number of preheating blocks and assigned to a second number of preheating processing units for processing.

[0381] In one embodiment, the number of preheating steps is determined by: obtaining the model size of the diffusion processing model; obtaining the diffusion processing complexity; and determining the number of preheating steps based on the model size and the diffusion processing complexity.

[0382] In one embodiment, obtaining the diffusion processing complexity includes: obtaining diffusion target description information; obtaining the number of image elements after diffusion and the relationship information between the image elements after diffusion based on the diffusion target description information; determining a first sub-complexity based on the number of image elements after diffusion; determining a second sub-complexity based on the relationship information between the image elements after diffusion; and determining the diffusion processing complexity based on the first sub-complexity and the second sub-complexity.

[0383] In one embodiment, determining the number of preheating steps based on model size and diffusion processing complexity includes: determining a first score based on model size; determining a second score based on diffusion processing complexity; and determining the number of preheating steps based on the first score and the second score.

[0384] In one embodiment, the second number is determined by: acquiring the image size of the image; acquiring the service type used for diffusion processing; acquiring the target diffusion processing efficiency; and determining the second number based on the image size, service type, and target diffusion processing efficiency.

[0385] In step 3023, the diffusion processing model is divided into a first number of model tasks and assigned to a first number of diffusion processing units for execution. When the diffusion processing unit executes the assigned model task in each denoising step, it uses the first key value generated during the execution of the diffusion processing model in the previous denoising step.

[0386] In one embodiment, the first number of model tasks are ordered and are distributed to multiple warm-up steps in the following manner: obtaining the task size of each model task; merging a portion of sequentially adjacent model tasks using multiple first candidate merging schemes, such that the number of generated merged task blocks is equal to the number of warm-up steps; calculating the total task size of each merged task block in each first candidate merging scheme based on the task size, and calculating the first variance of the total task size of each merged task block; determining the first candidate merging scheme with the smallest first variance as the target merging scheme, and merging a portion of sequentially adjacent model tasks into one warm-up step based on the target merging scheme.

[0387] Taking the multi-head attention model layer in the diffusion processing model as an example, the specific operations of the diffusion processing unit in performing the model task are as follows: First, the input image vector and the first key value are linearly transformed to obtain the query vector Q, the key vector K, and the value vector V. Assume that the weight matrices of the linear transformation are W, respectively. Q W K W VThen Q = X × W Q K = KV × W K V = KV × W V Here, X is the input image vector, and KV is the first key value. Then, the similarity between the query vector and the key vector is calculated, typically using a dot product, to obtain the attention score S = Q × K. T Next, the attention scores are normalized, for example, using the softmax function, to obtain the attention weights A = softmax(S). Finally, the attention weights are multiplied by the value vector and summed to obtain the multi-head attention output O = A × V. During execution, the model parameters are also updated according to the loss function, for example, using gradient descent to update the weight matrix W. Q W K W V The value of is adjusted to improve the model's performance.

[0388] In step 3031, each preheating unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task.

[0389] In one embodiment, the preheating step is a plurality of preheating steps, wherein a first number of model tasks are assigned to the plurality of preheating steps, such that the number of model tasks assigned to each preheating step is a third number, and the third number in different preheating steps is not exactly the same; the first number of model tasks are executed sequentially in the preheating steps, including: in each preheating step, executing the third number of model tasks corresponding to the preheating step, thereby executing the first number of model tasks in the plurality of preheating steps.

[0390] In step 3032, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

[0391] In one embodiment, the preheating step consists of multiple preheating steps; passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task includes: in the last preheating step of the multiple preheating steps, passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task in sequence according to the order of the model tasks.

[0392] In one embodiment, each model task begins execution in a different denoising sub-step of the first denoising step; passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task includes: obtaining a first size of the first subkey value; predicting a first transmission time of the first subkey value based on the first size; obtaining the execution start time of the denoising sub-step corresponding to the first subkey value; determining the transmission start time based on the execution start time and the first transmission time; and passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task at the transmission start time.

[0393] In step 3041, the first key value used in the first denoising step of the plurality of denoising steps is synthesized using the second number of first sub-key values ​​generated by the second number of preheating processing units in the preheating step.

[0394] In one embodiment, the first sub-key value is represented as a first sub-matrix corresponding to the preheating block, and the first key value is represented as a first matrix corresponding to the image; by using the second number of first sub-key values ​​generated by the second number of preheating processing units in the preheating step, the first key value used in the first denoising step of the multiple denoising steps is synthesized, including: integrating the first sub-matrices generated by the second number of preheating processing units respectively and corresponding to each preheating block into a first matrix according to the order of the preheating blocks.

[0395] In step 3042, the diffusion processing model is executed in the first denoising step.

[0396] In step 3050, in any other denoising step after the first denoising step among the multiple denoising steps, the diffusion processing model is executed through each diffusion processing unit using the first key value generated in the previous denoising step of the other denoising step.

[0397] In one embodiment, each model task begins execution in a different denoising sub-step of other denoising steps; in any other denoising step after the first denoising step in multiple denoising steps, the diffusion processing model is executed through each diffusion processing unit using the first key value generated by the previous denoising step of the other denoising step, including: in any other denoising step after the first denoising step in multiple denoising steps, for a single diffusion processing unit, obtaining the first current subkey value generated by the first model task corresponding to the single diffusion processing unit in other denoising steps, and the first preceding subkey value generated by the preceding model task before the first model task in other denoising steps; for a single diffusion processing unit, extracting the first subsequent subkey value corresponding to the subsequent model task after the first model task from the first key value generated by the previous denoising step of the other denoising steps; synthesizing the first preceding subkey value, the first current subkey value, and the first subsequent subkey value into a first synthesized key value, and executing each model task through each diffusion processing unit using the first synthesized key value.

[0398] In one embodiment, each diffusion processing unit performs each model task using a first synthetic key value, including: dividing the image into a fourth number of image blocks, where the fourth number is equal to the number of denoising sub-steps in the denoising step, and the fourth number is greater than or equal to a first number; using the first first number of denoising sub-steps in the fourth number of denoising sub-steps as the first number of diffusion processing units to process the denoising sub-steps of the first image block, wherein, after processing the first image block, the diffusion processing unit processes the second image block to the last image block of the fourth number using the first synthetic key value in the fifth number of denoising sub-steps to perform each model task, wherein the fifth number is the fourth number minus 1.

[0399] In one embodiment, each diffusion processing unit performs the steps of each model task using the first synthetic key value. Taking the denoising encoder in the diffusion processing model as an example, the specific operation of the diffusion processing unit in performing the model task using the first synthetic key value is as follows: First, the input image vector and the first synthetic key value are input into the first layer of the mutual attention model of the denoising encoder. In the mutual attention model, the query, key, and value vectors and attention weights are calculated, similar to the operation of the multi-head attention model layer described above. Then, the output of the mutual attention model is non-linearly transformed through an activation function, such as using the ReLU function. Next, the transformed output is input into the next layer of the mutual attention model, and the above operation is repeated until all layers of the denoising encoder are completed. During the calculation of each layer, the input of the model is adjusted according to the first synthetic key value to improve the accuracy of denoising. At the same time, the loss function is calculated based on the difference between the model output and the target image, and the parameters of the model are updated using an optimization algorithm (such as the Adam optimizer) to continuously optimize the performance of the model.

[0400] Description of apparatus and devices according to embodiments of this disclosure

[0401] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0402] It should be noted that in various specific embodiments of this application, when processing data related to object characteristics, such as object attribute information or sets of attribute information, is required, the object's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining object attribute information, separate permission or consent from the object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the object's separate permission or consent will the necessary object-related data for the proper functioning of these embodiments be acquired.

[0403] Figure 31 is a schematic diagram of the diffusion processing apparatus provided in an embodiment of this disclosure. The method applied by the diffusion processing apparatus 3100 is executed in multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into a first number of model tasks, which are assigned to a first number of diffusion processing units for execution. When the diffusion processing unit executes the assigned model task in each denoising step, it uses the first key value generated during the execution of the diffusion processing model in the previous denoising step. The diffusion processing apparatus 3100 includes:

[0404] The allocation unit 3110 is used to divide the image into a second number of preheating blocks in the preheating step before multiple denoising steps, and allocate them to a second number of preheating processing units for processing. Each preheating processing unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task.

[0405] The transmission unit 3120 is used to transmit the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can use the second number of first subkey values ​​generated by the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step among multiple denoising steps, so that the first denoising step can execute the diffusion processing model.

[0406] The execution unit 3130 is used to execute the diffusion processing model in any other denoising step after the first denoising step in a plurality of denoising steps, by means of the diffusion processing units, using the first key value generated in the previous denoising step of the other denoising step.

[0407] Optionally, there are multiple warm-up steps, wherein a first number of model tasks are assigned to multiple warm-up steps, such that the number of model tasks assigned to each warm-up step is a third number, and the third number in different warm-up steps is not exactly the same.

[0408] The allocation unit 3110 is specifically used for:

[0409] In each warm-up step, the third number of model tasks corresponding to the warm-up step are executed, so that the first number of model tasks are executed in multiple warm-up steps.

[0410] Optionally, the first number of model tasks are ordered, and the allocation unit 3110 is specifically used for:

[0411] Get the task size for each model task;

[0412] By using multiple first-candidate merging schemes, a portion of sequentially adjacent model tasks are merged, so that the number of generated merged task blocks is equal to the number of warm-up steps.

[0413] Based on the task size, calculate the total task size of each merged task block in each first candidate merge scheme, and calculate the first variance of the total task size of each merged task block;

[0414] The first candidate merging scheme with the smallest variance is determined as the target merging scheme, and based on the target merging scheme, a portion of sequentially adjacent model tasks are merged into a warm-up step.

[0415] Optionally, the allocation unit 3110 is specifically used for:

[0416] Obtain the model size of the diffusion processing model;

[0417] Obtain the complexity of the diffusion process;

[0418] The number of preheating steps is determined based on the model size and diffusion processing complexity.

[0419] Optionally, the allocation unit 3110 is specifically used for:

[0420] Obtain description information of the diffusion target;

[0421] Based on the diffusion target description information, obtain the number of elements in the diffused image and the relationship information between the elements in the diffused image;

[0422] The first sub-complexity is determined based on the number of elements in the diffused image;

[0423] The second sub-complexity is determined based on the relationship information of the elements in the diffused image;

[0424] The diffusion processing complexity is determined based on the first and second sub-complexities.

[0425] Optionally, the allocation unit 3110 is specifically used for:

[0426] The first score is determined based on the model size;

[0427] The second score is determined based on the complexity of the diffusion process.

[0428] The number of preheating steps is determined based on the first and second scores.

[0429] Optionally, the transfer unit 3120 is specifically used for:

[0430] Get the image size;

[0431] Obtain the business type used for diffusion processing;

[0432] Obtain the target diffusion processing efficiency;

[0433] The second number is determined based on image size, service type, and target diffusion processing efficiency.

[0434] Optionally, the first subkey value is represented as the first submatrix corresponding to the preheating block, and the first key value is represented as the first matrix corresponding to the image;

[0435] The transfer unit 3120 is specifically used for:

[0436] The first sub-matrices generated by the second number of preheating units, each corresponding to a preheating block, are integrated into a first matrix according to the order of the preheating blocks.

[0437] Optionally, the preheating step can be multiple preheating steps;

[0438] The transfer unit 3120 is specifically used for:

[0439] In the last preheating step of multiple preheating steps, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task in the order of the model task.

[0440] Optionally, each model task begins execution in a different denoising substep of the first denoising step;

[0441] The transfer unit 3120 is specifically used for:

[0442] Get the first size of the first subkey value;

[0443] Based on the first size, predict the first transmission time of the first subkey value;

[0444] Get the start time of the denoising sub-step corresponding to the first subkey value;

[0445] The transmission start time is determined based on the execution start time and the first transmission time;

[0446] At the start of transmission, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

[0447] Optionally, each model task can begin execution in a different denoising substep of another denoising step;

[0448] Execution unit 3130 is specifically used for:

[0449] In any other denoising step after the first denoising step in a plurality of denoising steps, for a single diffusion processing unit, the first current subkey value generated by the first model task corresponding to the single diffusion processing unit in the other denoising step, and the first preceding subkey value generated by the preceding model task before the first model task in the other denoising step are obtained.

[0450] For a single diffusion processing unit, the first subsequent sub-key value corresponding to the subsequent model task after the first model task is extracted from the first key value generated in the previous denoising step of other denoising steps.

[0451] The first preceding subkey value, the first current subkey value, and the first following subkey value are combined into a first synthesized key value, and each model task is executed using the first synthesized key value through each diffusion processing unit.

[0452] Optionally, the execution unit 3130 is specifically used for:

[0453] The image is divided into a fourth number of image blocks, which is equal to the number of denoising sub-steps in the denoising step, and the fourth number is greater than or equal to the first number.

[0454] The first number of denoising sub-steps in the fourth number of denoising sub-steps are respectively used as the first number of diffusion processing units to process the denoising sub-steps of the first image block. In the fifth number of denoising sub-steps after processing the first image block, the diffusion processing unit uses the first synthesis key value to process the second image block to the last image block of the fourth number, so as to execute each model task. The fifth number is the fourth number minus 1.

[0455] Optionally, the execution unit 3130 is specifically used for:

[0456] Obtain description information of the diffusion target;

[0457] Based on the diffusion target description information, a diffusion guidance vector is generated;

[0458] Guided by the diffusion guiding vector, the second image block to the last image block of the fourth number are processed using the first synthetic key value.

[0459] Referring to Figure 32, which is a partial structural block diagram of a terminal implementing the diffusion processing method of this disclosure embodiment, the terminal includes: a radio frequency (RF) circuit 3210, a memory 3215, an input unit 3230, a display unit 3240, a sensor 3250, an audio circuit 3260, a wireless fidelity (WiFi) module 3270, a processor 3280, and a power supply 3290, etc. Those skilled in the art will understand that the terminal structure shown in Figure 32 does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0460] The RF circuit 3210 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 3280; in addition, it transmits uplink data to the base station.

[0461] The memory 3215 can be used to store software programs and modules, and the processor 3280 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 3215.

[0462] The input unit 3230 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the content terminal. Specifically, the input unit 3230 may include a touch panel 3231 and other input devices 3232.

[0463] Display unit 3240 can be used to display input or provided information, as well as various menus of the content terminal. Display unit 3240 may include display panel 3241.

[0464] Audio circuitry 3260, speaker 3261, and microphone 3262 provide an audio interface.

[0465] In this embodiment, the processor 3280 included in the terminal can execute the diffusion processing method of the previous embodiment.

[0466] The terminals disclosed in this embodiment include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to textural graph applications and molecular graph modeling.

[0467] Figure 33 is a partial structural block diagram of a server implementing the diffusion processing method of this disclosure. The server can vary significantly due to different configurations or performance, and may include one or more Central Processing Units (CPUs) 3322 (e.g., one or more processors) and a memory 3332, and one or more storage media 3330 (e.g., one or more mass storage devices) for storing application programs 3342 or data 3344. The memory 3332 and storage media 3330 may be temporary or persistent storage. The program stored in the storage media 3330 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the server. Furthermore, the CPU 3322 may be configured to communicate with the storage media 3330 and execute the series of instruction operations in the storage media 3330 on the server.

[0468] The server may also include one or more power supplies 3326, one or more wired or wireless network interfaces 3350, one or more input / output interfaces 3358, and / or one or more operating systems 3341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0469] The central processing unit 3322 in the server can be used to execute the diffusion processing method of the present disclosure embodiments.

[0470] This disclosure also provides a computer-readable storage medium for storing program code for executing the diffusion processing methods of the foregoing embodiments.

[0471] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the diffusion processing method described above.

[0472] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar terms and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0473] It should be understood that in this disclosure, "at least one item" refers to one or more items, and "more than one item" refers to two or more items. "And / or" is used to describe the relationship between related content, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related content are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0474] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0475] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0476] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0477] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0478] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server 130, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0479] In summary, this application provides a diffusion processing method, apparatus, electronic device, computer-readable storage medium, and computer program product. Before executing the diffusion processing model in multiple denoising steps, the image is first divided into a second number of preheating blocks in a preheating step, and these blocks are assigned to a second number of preheating processing units for processing. Each preheating processing unit sequentially executes a first number of model tasks to obtain a first sub-key value corresponding to each model task. In this process, since each preheating processing unit can execute multiple model tasks for different preheating blocks in the same preheating step, compared to the traditional method where only one diffusion processing unit obtains parameters in a single preheating step, the resource utilization of the preheating step is greatly improved. In traditional methods, the number of preheating steps is equal to the number of diffusion processing units, resulting in a large amount of idle resources. This method reduces this idle situation by processing multiple model tasks in multiple preheating blocks in parallel. The first sub-key value is then passed to the corresponding diffusion processing unit, which synthesizes the first key value used in the first denoising step. Subsequent denoising steps then use the first key value generated in the previous denoising step to execute the diffusion processing model. This approach reuses the previously calculated first key value, reduces redundant calculations, improves the processing speed of the diffusion processing model, and thus improves the overall efficiency of the diffusion processing.

[0480] Furthermore, when there are multiple preheating steps, the first number of model tasks are distributed across multiple preheating steps. The number of model tasks assigned to each preheating step is the third number, and this third number is not identical across different preheating steps. Each preheating step executes its corresponding third number of model tasks, thus completing the first number of model tasks across multiple preheating steps. Different model tasks have varying complexity and computational load. Executing a large number of complex model tasks in a single preheating step would overload the preheating processing unit, potentially leading to performance degradation or even system failure. Distributing the model tasks across multiple preheating steps allows for a more even distribution of the processing load across all steps, reducing the risk of excessive load due to executing too many model tasks in a short period, improving system stability and reliability, and ultimately ensuring the smooth progress of the diffusion process.

[0481] Given an ordered set of model tasks, the task size of each model task is first determined. Then, several first-candidate merging schemes are used to merge some sequentially adjacent model tasks, ensuring the number of merged task blocks equals the number of warm-up steps. Based on the task size, the total task size and first variance of each merged task block are calculated. The scheme with the smallest first variance is selected as the target merging scheme, and sequentially adjacent model tasks are merged into a single warm-up step according to this scheme. Different model tasks vary in size due to differences in the number of layer parameters, computational complexity, etc. Randomly assigning model tasks to different warm-up steps can lead to an uneven processing load, with some warm-up steps being overloaded while others are underutilized. By calculating the first variance and selecting the scheme with the smallest variance, the computational load of model tasks in each warm-up step is relatively balanced, the processing load of each warm-up unit is more distributed, performance bottlenecks are avoided due to excessive workload in any particular warm-up unit, resource utilization is improved, and the entire warm-up process is made more efficient.

[0482] When determining the number of warm-up steps, the model size and complexity of the diffusion processing model are first obtained, and then the number of warm-up steps is determined based on these two factors. The model size reflects the complexity of the model; the larger the model, the more denoising steps, mutual attention models, etc., it contains, and the more complex the model task to be performed. The diffusion processing complexity reflects the complexity of the target image to be generated; the higher the complexity, the greater the computational cost required to generate the target image. If these two factors are not considered and the number of warm-up steps is determined arbitrarily, it may result in too many or too few warm-up steps. Too many warm-up steps will waste resources and increase processing time; too few warm-up steps will not provide a sufficiently accurate first key value for the first denoising step, affecting the accuracy of the diffusion processing. By comprehensively considering the model size and diffusion processing complexity to determine the number of warm-up steps, the processing difficulty of the diffusion processing model is quantified, improving the accuracy of determining the number of warm-up steps and making the setting of warm-up steps more reasonable. This ensures that an accurate first key value is provided for the first denoising step while avoiding resource waste.

[0483] When determining the complexity of diffusion processing, the target description information is first obtained. Based on this, the number of elements in the diffused image and the relationship information between these elements are obtained. The first and second sub-complexities are then determined, leading to the final diffusion processing complexity. The number of elements in the diffused image reflects the quantity of elements in the target image; more elements mean a more complex image. The relationship information between the elements reflects the interactions between them; more complex relationships mean greater difficulty in image generation. Considering only a single factor cannot comprehensively and accurately assess the complexity of diffusion processing. By determining the first and second sub-complexities separately and considering them comprehensively, the diffusion processing complexity is determined from multiple dimensions, resulting in a more accurate assessment than a single-dimensional determination. Accurate diffusion processing complexity provides a more reliable basis for determining the number of preheating steps, thereby improving the efficiency and accuracy of diffusion processing.

[0484] When determining the number of preheating steps based on model size and diffusion processing complexity, a first score and a second score are first determined based on both, and then the number of preheating steps is determined based on these two scores. Converting model size and diffusion processing complexity into fractional form more intuitively reflects their influence on the determination of the number of preheating steps. By comprehensively calculating these two scores to determine the number of preheating steps, multiple factors influencing the determination are taken into account, avoiding the inaccuracies that may arise from a single factor determining the number of preheating steps. This improves the accuracy of the determination results and makes the setting of preheating steps more consistent with actual diffusion processing requirements.

[0485] When determining the second number, the image size, the service type used for diffusion processing, and the target diffusion processing efficiency are first obtained, and then the second number is determined based on these factors. Image size affects the number of preheating blocks obtained after image partitioning; the larger the image, the greater the computational load when executing the diffusion processing method, requiring more preheating blocks to balance the computational load of each preheating processing unit. Different service types have different requirements for the accuracy and complexity of diffusion processing. For example, molecular detection scenarios have very high accuracy requirements for generating molecular maps, requiring more preheating blocks to meet these accuracy requirements. The target diffusion processing efficiency reflects the user's expectation of processing time; the higher the efficiency requirement, the more preheating blocks are needed to achieve parallel processing. Considering these three factors comprehensively allows for a more complete assessment of image partitioning needs, improves the accuracy of determining the second number, makes image partitioning more reasonable, and is conducive to improving the efficiency and quality of diffusion processing.

[0486] When the first sub-key value is represented as the first sub-matrix corresponding to the preheating block, and the first key value is represented as the first matrix corresponding to the image, the first sub-matrices generated by the second number of preheating processing units, each corresponding to a preheating block, are integrated into the first matrix according to the order of the preheating blocks. When dividing the image into preheating blocks, each block has its specific position and order, which is crucial for accurately synthesizing the first matrix. If the integration is not performed according to the order of the preheating blocks, the elements in the first matrix may be misplaced, affecting the accuracy of the first key value. Integrating in order ensures the integrity and accuracy of the first matrix, allowing the diffusion processing unit to obtain more accurate results when using the first key value to execute the diffusion processing model, thus improving the accuracy of the diffusion processing.

[0487] When there are multiple preheating steps, in the last preheating step, the first subkey value corresponding to the model task is sequentially passed to the diffusion processing unit corresponding to the model task in the order of the model tasks. In the last preheating step, the preheating processing unit has completed the calculation of all model tasks and obtained the first subkey values ​​corresponding to the multiple model tasks preceding the diffusion processing model. The diffusion processing unit executes the model tasks sequentially, executing the preceding model tasks first. Passing the first subkey value in the order of the model tasks ensures that the diffusion processing unit synthesizes the first key value corresponding to the preceding model task in a timely manner and uses the first key value to execute the model task. This method allows the preheating step and the operation of passing the first subkey value to be executed concurrently within the same time period, without waiting for the last preheating step to end, reducing unnecessary waiting time, reducing the image processing time of the diffusion processing model, and improving execution efficiency.

[0488] When each model task begins execution in a different denoising sub-step of the first denoising step, the first size of the first subkey value is first obtained. Based on this, the first transmission time is predicted, and the execution start time of the denoising sub-step corresponding to the first subkey value is obtained. Based on these two times, the transmission start time is determined. At the transmission start time, the first subkey value corresponding to the model task is transmitted to the diffusion processing unit corresponding to the model task. The first size of the first subkey value determines the time required for its transmission; the larger the first size, the longer the transmission time. If the transmission time is not considered, sending the first subkey values ​​of all model tasks at a single point in time will lead to a surge in communication pressure, potentially resulting in transmission delays or even data loss. By predicting the first transmission time and combining it with the execution start time of the denoising sub-step to determine the transmission start time, and transmitting the first subkey values ​​corresponding to multiple model tasks in batches, the transmission process can be more evenly distributed along the timeline, avoiding the surge in communication pressure caused by concentrated transmission. This effectively alleviates the communication pressure between the preheating processing unit and the diffusion processing unit, ensuring the stability and reliability of data transmission.

[0489] When various model tasks begin execution in different denoising sub-steps of other denoising steps, in other denoising steps, for a single diffusion processing unit, the first current subkey value generated by the first model task in that denoising step and the first preceding subkey value generated by the preceding model task in that denoising step are obtained. From the first key value generated in the previous denoising step of other denoising steps, the first subsequent subkey value corresponding to the subsequent model task after the first model task is extracted. These three subkey values ​​are combined into a first composite key value. Each diffusion processing unit uses the first composite key value to execute its respective model task. Traditional methods generally use the key value generated in the previous denoising step for diffusion processing, without considering the key values ​​generated by the preceding and current model tasks in the current denoising step. However, in this method, the first current subkey value and the first preceding subkey value are generated in the current denoising step, better reflecting the actual situation of the current denoising step. Combining these three subkey values ​​into the first composite key value integrates the key values ​​generated by multiple model tasks at different stages, providing more comprehensive and accurate parameters for diffusion processing. Using the first synthetic key value for diffusion processing can achieve better convergence results, improve the accuracy of diffusion processing, and make the generated target image more in line with expectations.

[0490] When each diffusion processing unit executes its model task using the first synthesized key value, the image is divided into a fourth number of image blocks. This fourth number is equal to and greater than or equal to the number of denoising sub-steps in the denoising step. The first first number of denoising sub-steps in the fourth number of denoising sub-steps are used as the first number of diffusion processing units to process the denoising sub-steps of the first image block. After processing the first image block, the diffusion processing unit, in its fifth number of denoising sub-steps, uses the first synthesized key value to process the second to the last image blocks of the fourth number of denoising sub-steps to execute each model task. When processing the first image block, each diffusion processing unit only needs to store the parameters of the model task it is responsible for processing, rather than storing all the parameters of the diffusion model, greatly reducing storage space usage. Simultaneously, using the first synthesized key value to process subsequent image blocks, since it integrates key values ​​from multiple model tasks, can more accurately guide the diffusion processing process. Compared to using the key value generated by the previous denoising step for diffusion processing, it achieves better convergence results and improves the accuracy of the diffusion processing.

[0491] When processing the second to the last image blocks (from the fourth block to the first) using the first synthetic key value, the diffusion target description information is first obtained. Based on this, a diffusion guidance vector is generated, and processing is performed under the guidance of the diffusion guidance vector. The diffusion target description information is a textual description of the target image to be generated. Converting it into a diffusion guidance vector transforms textual information into a vector form that can be recognized and processed by a computer. The diffusion processing model can quickly recognize and process information in vector form. Through this transformation, the diffusion processing model can be quickly adapted at a relatively low cost, avoiding complex information conversion processes and improving the diffusion processing efficiency of the model, making the diffusion processing process more efficient.

[0492] In a processor architecture with both CPU and GPU, the CPU is responsible for instruction scheduling, while each GPU can be the smallest hardware unit executing model tasks. This division of labor fully leverages the advantages of both CPU and GPU. The CPU possesses powerful logic control and instruction scheduling capabilities, enabling it to rationally arrange the execution order and timing of various model tasks; the GPU, on the other hand, has highly parallel computing capabilities, suitable for handling large-scale matrix operations and other model tasks. This division of labor improves the overall system's computational efficiency, making the diffusion processing model more efficient. Simultaneously, during model task execution, the model input is adjusted based on the first synthetic key value to improve denoising accuracy. The first synthetic key value integrates key values ​​from multiple model tasks, more accurately reflecting the current denoising step. Adjusting the model input based on this value allows the model to better adapt to actual denoising requirements, improving denoising accuracy and thus enhancing the quality of diffusion processing, resulting in clearer and more accurate target images.

[0493] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0494] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A diffusion processing method performed by an electronic device, the diffusion processing method being performed in a plurality of denoising steps, a diffusion processing model being performed in each of the denoising steps, the diffusion processing model being divided into a first number of model tasks, being assigned to a first number of diffusion processing units for performing, in each denoising step, the diffusion processing units performing the assigned model tasks using a first key value generated in a performing process of the diffusion processing model of a previous denoising step, the diffusion processing method comprising: in a pre-warming step before the plurality of denoising steps, dividing an image into a second number of pre-warming partitions, being assigned to a second number of pre-warming processing units for processing, each of the pre-warming processing units sequentially performing the first number of model tasks in the pre-warming step to obtain a first sub-key value corresponding to each of the model tasks; passing the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task for synthesizing the first key value used in a first denoising step of the plurality of denoising steps by the second number of the first sub-key values generated by the second number of pre-warming processing units in the pre-warming step, for the first denoising step to perform the diffusion processing model; and in any other denoising step after the first denoising step of the plurality of denoising steps, performing the diffusion processing model by each of the diffusion processing units using the first key value generated in a previous denoising step of the other denoising step.

2. The diffusion treatment method according to claim 1, the preheating step is a plurality of preheating steps, wherein, the first number of model tasks are assigned to the plurality of pre-warming steps such that a number of model tasks assigned to each of the pre-warming steps is a third number, the third number in different pre-warming steps being not completely the same; the sequentially performing the first number of model tasks in the pre-warming step comprises: in each of the pre-warming steps, performing the third number of model tasks corresponding to the pre-warming step, so that the first number of model tasks are performed in the plurality of pre-warming steps.

3. The diffusion processing method of claim 2, the first number of model tasks being in order, the first number of model tasks being assigned to the plurality of pre-warming steps by: obtaining a task size of each of the model tasks; merging a part of sequentially adjacent model tasks in each of a plurality of first candidate merging schemes, so that a number of merged task blocks generated is equal to a number of the pre-warming steps; calculating a total task size of each of the merged task blocks in each of the first candidate merging schemes based on the task size, and calculating a first variance of the total task size of each of the merged task blocks; determining the first candidate merging scheme with the smallest first variance as a target merging scheme, and merging a part of sequentially adjacent model tasks into one of the pre-warming steps based on the target merging scheme.

4. The diffusion processing method of claim 2 or 3, the number of pre-warming steps being determined by: obtaining a model size of the diffusion processing model; obtaining a diffusion processing complexity; determining the number of pre-warming steps based on the model size and the diffusion processing complexity.

5. The diffusion processing method of claim 4, wherein the obtaining the diffusion processing complexity comprises: obtaining diffusion target description information; obtaining a number of diffusion image elements and diffusion image element relationship information based on the diffusion target description information; determining a first sub-complexity based on the number of diffusion image elements; determining a second sub-complexity based on the diffusion image element relationship information; and determining the diffusion processing complexity based on the first sub-complexity and the second sub-complexity.

6. The diffusion processing method of claim 4 or 5, wherein the determining the number of pre-warming steps based on the model size and the diffusion processing complexity comprises: determining a first score based on the model size; determining a second score based on the diffusion processing complexity; and determining the number of pre-warming steps based on the first score and the second score.

7. The diffusion processing method of any one of claims 1 to 6, wherein the second number is determined by: obtaining an image size of the image; obtaining a service type for the diffusion processing; obtaining a target diffusion processing efficiency; and determining the second number based on the image size, the service type, and the target diffusion processing efficiency.

8. The diffusion processing method of any one of claims 1 to 7, wherein the first sub-key value is embodied as a first sub-matrix corresponding to the pre-warming block, and the first key value is embodied as a first matrix corresponding to the image; and the synthesizing the first key value used in a first denoising step of the plurality of denoising steps by the second number of first sub-key values generated by the second number of pre-warming processing units in the pre-warming step comprises: integrating the second number of first sub-matrices respectively generated by the second number of pre-warming processing units and corresponding to the pre-warming blocks into the first matrix in the order of the pre-warming blocks.

9. The diffusion processing method of any one of claims 1 to 8, wherein the pre-warming step is a plurality of pre-warming steps; and the passing the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task comprises: sequentially passing the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task in the order of the model task in a last pre-warming step of the plurality of pre-warming steps.

10. The diffusion processing method of any one of claims 1 to 9, wherein each of the model tasks starts to be executed in a different denoising sub-step of the first denoising step; and the passing the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task comprises: obtaining a first size of the first sub-key value; predicting a first transmission time of the first sub-key value based on the first size; obtaining an execution start time of the denoising sub-step corresponding to the first sub-key value; determining a transmission start time based on the execution start time and the first transmission time; and passing the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task at the transmission start time. ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ ​ 11. The diffusion processing method of any one of claims 1-10, each of the model tasks is executed in a different denoising sub-step of the other denoising steps; any other denoising step after the first denoising step of the plurality of denoising steps is executed by each of the diffusion processing units using the first key value generated from a previous denoising step of the other denoising step, the diffusion processing model is executed including: for a single diffusion processing unit, any other denoising step after the first denoising step of the plurality of denoising steps, a first current sub-key value generated by a first model task corresponding to the single diffusion processing unit in the other denoising step, and a first previous sub-key value generated by a previous model task before the first model task in the other denoising step are obtained; for the single diffusion processing unit, a first subsequent sub-key value corresponding to a subsequent model task after the first model task is extracted from the first key value generated from a previous denoising step of the other denoising step; the first previous sub-key value, the first current sub-key value, and the first subsequent sub-key value are combined into a first combined key value, and each of the model tasks is executed by each of the diffusion processing units using the first combined key value.

12. The diffusion processing method of claim 11, the executing each of the model tasks by each of the diffusion processing units using the first combined key value includes: dividing an image into a fourth number of image patches, the fourth number being equal to a number of denoising sub-steps in the denoising steps, the fourth number being greater than or equal to the first number; a first number of denoising sub-steps of the fourth number of denoising sub-steps are respectively used as denoising sub-steps for a first image patch by the first number of diffusion processing units, wherein a fifth number of denoising sub-steps after the first image patch are respectively used by the diffusion processing units to process a second image patch to a last image patch of the fourth number using the first combined key value to execute each of the model tasks, wherein the fifth number is the fourth number minus 1.

13. The diffusion processing method of claim 12, the respectively using the first combined key value to process the second image patch to the last image patch of the fourth number includes: obtaining diffusion target description information; generating a diffusion guide vector based on the diffusion target description information; respectively using the first combined key value to process the second image patch to the last image patch of the fourth number under guidance of the diffusion guide vector.

14. A diffusion processing apparatus, a method applied by the diffusion processing apparatus is executed in a plurality of denoising steps, a diffusion processing model is executed in each of the denoising steps, the diffusion processing model is divided into a first number of model tasks, and is assigned to a first number of diffusion processing units for execution, when the diffusion processing units execute the assigned model tasks in each denoising step, a first key value generated in a previous denoising step of the diffusion processing model execution process is used, the diffusion processing apparatus includes: an allocation unit configured to, before the plurality of denoising steps, divide the image into a second number of preheating blocks, and allocate the preheating blocks to a second number of preheating processing units for processing, each of the preheating processing units sequentially performing the first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each of the model tasks; a passing unit configured to pass the first sub-key value corresponding to the model task to a diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes the first key value used in a first denoising step of the plurality of denoising steps by using a second number of the first sub-key values generated by the second number of preheating processing units in the preheating step, so that the first denoising step performs the diffusion processing model; and an execution unit configured to, after the first denoising step of the plurality of denoising steps, execute the diffusion processing model by each of the diffusion processing units by using the first key value generated by a previous denoising step of any other denoising step of the plurality of denoising steps.

15. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the diffusion processing method according to any one of claims 1 to 13.

16. A computer readable storage medium, wherein the storage medium stores a computer program, and the computer program is executed by a processor to implement the diffusion processing method according to any one of claims 1 to 13.

17. A computer program product, comprising a computer program, wherein the computer program is read and executed by a processor of a computer device, so that the computer device executes the diffusion processing method according to any one of claims 1 to 13. ​

Images