Diffusion processing method and device, electronic equipment and medium

Abstract

The invention provides a diffusion processing method and device, electronic equipment and a medium. The diffusion processing method comprises the steps that in a preheating step before a plurality of denoising steps, an image to be diffused is divided into a second number of preheating blocks, the preheating blocks are allocated to a second number of preheating processing units for processing, each preheating processing unit sequentially executes a first number of model tasks in the preheating step, and a first sub-key value corresponding to each model task is obtained; and transmitting the first sub-key value corresponding to the model task to a diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes a first key value used in the first denoising step by using a second number of first sub-key values generated by a second number of preheating processing units in the preheating step, and the first denoising step executes a diffusion processing model. According to the embodiment of the invention, idle resources in the preheating process can be reduced, and the efficiency of the whole diffusion treatment is improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, and in particular, to a diffusion processing method and device, electronic equipment and medium. BACKGROUND

[0002] A diffusion model can generate a high-resolution image by simulating the diffusion process of a substance in a medium. For example, in a text-to-image application, a diffusion guide vector is generated according to the text description of an image in a text-to-image. A to-be-diffused image is input into the diffusion model, and the to-be-diffused image is gradually denoised under the guidance of the diffusion guide vector, that is, the diffusion process is divided into a plurality of denoising steps, and each denoising step removes a part of noise from the to-be-diffused image. In this way, after all the denoising steps are completed, a target image that meets the text description of the image is obtained.

[0003] Therefore, when the diffusion model is executed, the diffusion model is actually executed at each denoising step. In each denoising step, in order to promote parallelism and improve execution efficiency, the diffusion model is divided into a plurality of model tasks, and each model task is assigned to a diffusion processing unit (such as a GPU) for execution. For example, the diffusion model has 10 layers and is divided into 5 model tasks, and each model task has 2 layers. The 2 layers are executed by one diffusion processing unit, and a total of 5 diffusion processing units are required. When each model task is executed, it cannot rely only on the parameters of the model task itself, and sometimes it needs parameters obtained by multiple model tasks. For example, the diffusion transformer network (Di T), which is a mainstream diffusion model, has an important multi-head attention model layer, which needs key-value (KV) parameters of a plurality of model tasks as a whole, rather than KV parameters in a certain model task. However, when a certain model task is executed, the model tasks that precede the model task in the plurality of model tasks have obtained the KV parameters in the denoising step, and the model tasks that follow the model task in the plurality of model tasks have not obtained the KV parameters in the denoising step, but the KV parameters have been obtained in the previous denoising step. Considering that the KV parameters in consecutive denoising steps are not very different, the KV parameters in the previous denoising step can be used.

[0004] However, for the first denoising step, there is no previous denoising step. Therefore, before the first denoising step, a plurality of warm-up steps are added. Each diffusion processing unit is provided with a warm-up step. In the warm-up step, the diffusion processing unit processes a plurality of to-be-diffused image blocks to obtain KV parameters of the to-be-diffused image as a whole. After a number of warm-up steps equal to the number of diffusion processing units, the preliminary KV parameters of each diffusion processing unit are obtained. However, the more diffusion processing units there are, the more warm-up steps are set in advance, and the resource utilization rate in the warm-up step is insufficient, and the overall diffusion processing efficiency is low. SUMMARY

[0005] The disclosure provides a diffusion processing method and device, electronic equipment and medium, which can reduce resource idling during preheating and improve the efficiency of the entire diffusion processing.

[0006] According to an aspect of the disclosure, a diffusion processing method is provided, which is executed in multiple denoising steps, a diffusion processing model is executed in each denoising step, the diffusion processing model is divided into a first number of model tasks and is executed by a first number of diffusion processing units, in each denoising step, the diffusion processing units need to use the first key value generated during the execution of the diffusion processing model in the last denoising step, the diffusion processing method comprises:

[0007] In a preheating step before the multiple denoising steps, a to-be-diffused image is divided into a second number of preheating blocks, which are processed by a second number of preheating processing units, each preheating processing unit sequentially executes the first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each model task;

[0008] The first sub-key value corresponding to the model task is transmitted to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes the first key value used in the first denoising step by using the second number of first sub-key values generated by the second number of preheating processing units in the preheating step, so that the diffusion processing model is executed in the first denoising step;

[0009] In other denoising steps after the first denoising step, each diffusion processing unit executes the diffusion processing model by using the first key value generated in the previous denoising step of the other denoising step.

[0010] According to an aspect of the disclosure, a diffusion processing device is provided, the method applied by the diffusion processing device is executed in multiple denoising steps, a diffusion processing model is executed in each denoising step, the diffusion processing model is divided into a first number of model tasks and is executed by a first number of diffusion processing units, in each denoising step, the diffusion processing units need to use the first key value generated during the execution of the diffusion processing model in the last denoising step, the diffusion processing device comprises:

[0011] The allocation unit is configured to divide, in a preheating step before the multiple denoising steps, a to-be-diffused image into a second number of preheating blocks, which are processed by a second number of preheating processing units, each preheating processing unit sequentially executes the first number of model tasks in the preheating step to obtain a first sub-key value corresponding to each model task;

[0012] a delivery unit configured to deliver the first sub-key value corresponding to the model task to a diffusion processing unit corresponding to the model task, so that the diffusion processing unit synthesizes the first key value used in a first denoising step by using a second number of first sub-key values generated by a second number of preheating processing units in the preheating step, so that the diffusion processing model is executed in the first denoising step;

[0013] a performing unit configured to execute the diffusion processing model by using the first key value generated by a previous denoising step in a subsequent denoising step through each diffusion processing unit after the first denoising step.

[0014] Optionally, the preheating step is a plurality of preheating steps, wherein the first number of model tasks are distributed into the plurality of preheating steps, so that the number of model tasks distributed to each preheating step is a third number, and the third number in different preheating steps is not completely the same.

[0015] The distribution unit is specifically configured to:

[0016] In each preheating step, the third number of model tasks corresponding to the preheating step are executed, so that the first number of model tasks are executed in the plurality of preheating steps.

[0017] Optionally, the first number of model tasks is in order, and the distribution unit is specifically configured to:

[0018] obtain a task size of each model task;

[0019] merge a part of sequentially adjacent model tasks by using a plurality of first candidate merging schemes, so that the number of generated merging task blocks is equal to the number of preheating steps;

[0020] calculate a total task size of each merging task block in each first candidate merging scheme based on the task size, and calculate a first variance of the total task size of each merging task block;

[0021] determine the first candidate merging scheme with the smallest first variance as a target merging scheme, and merge a part of sequentially adjacent model tasks into one preheating step based on the target merging scheme.

[0022] Optionally, the distribution unit is specifically configured to:

[0023] obtain a model size of the diffusion processing model;

[0024] obtain a diffusion processing complexity;

[0025] The number of preheating steps is determined based on the model size and the diffusion processing complexity.

[0026] Optionally, the allocation unit is specifically used for:

[0027] Obtain description information of the diffusion target;

[0028] Based on the diffusion target description information, the number of image elements after diffusion and the relationship information between image elements after diffusion are obtained.

[0029] The first sub-complexity is determined based on the number of elements in the diffused image;

[0030] Based on the element relationship information of the diffused image, the second sub-complexity is determined;

[0031] The diffusion processing complexity is determined based on the first sub-complexity and the second sub-complexity.

[0032] Optionally, the allocation unit is specifically used for:

[0033] Based on the model size, determine the first score;

[0034] The second score is determined based on the aforementioned diffusion processing complexity;

[0035] The number of preheating steps is determined based on the first score and the second score.

[0036] Optionally, the transmission unit is specifically used for:

[0037] Get the size of the image to be diffused;

[0038] Obtain the service type used for the diffusion process;

[0039] Obtain the target diffusion processing efficiency;

[0040] The second number is determined based on the size of the image to be diffused, the service type, and the target diffusion processing efficiency.

[0041] Optionally, the first subkey value is represented as a first submatrix corresponding to the preheating block, and the first key value is represented as a first matrix corresponding to the image to be diffused;

[0042] The transmission unit is specifically used for:

[0043] The first sub-matrices generated by the second number of preheating units, each corresponding to a preheating block, are integrated into the first matrix according to the order of the preheating blocks.

[0044] Optionally, the preheating step may be a plurality of preheating steps;

[0045] The transmission unit is specifically used for:

[0046] In the last preheating step of the plurality of preheating steps, the first subkey value corresponding to the model task is sequentially passed to the diffusion processing unit corresponding to the model task according to the order of the model tasks.

[0047] Optionally, each of the model tasks begins execution in a different denoising substep of the first denoising step;

[0048] The transmission unit is specifically used for:

[0049] Get the first size of the first subkey value;

[0050] Based on the first size, predict the first transmission time of the first subkey value;

[0051] Obtain the start time of the denoising sub-step corresponding to the first subkey value;

[0052] The transmission start time is determined based on the execution start time and the first transmission time;

[0053] At the start time of the transmission, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

[0054] Optionally, each of the model tasks begins execution in a different denoising substep of the other denoising steps;

[0055] The execution unit is specifically used for:

[0056] In the other denoising steps after the first denoising step, for a single diffusion processing unit, the first current subkey value generated by the first model task corresponding to the single diffusion processing unit in the other denoising steps, and the first preceding subkey value generated by the preceding model task before the first model task in the other denoising steps are obtained.

[0057] For the single diffusion processing unit, the first subsequent sub-key value corresponding to the subsequent model task after the first model task is extracted from the first key value generated in the previous denoising step of the other denoising steps.

[0058] The first preceding subkey value, the first current subkey value, and the first following subkey value are combined into a first synthesized key value, and each of the diffusion processing units uses the first synthesized key value to execute each of the model tasks.

[0059] Optionally, the execution unit is specifically used for:

[0060] The image to be diffused is divided into a fourth number of image blocks, the fourth number being equal to the number of denoising sub-steps in the denoising step, and the fourth number being greater than or equal to the first number.

[0061] The first few denoising sub-steps in the fourth number of denoising sub-steps are respectively used as the first number of diffusion processing units to process the first image block to be diffused. The diffusion processing unit processes the second image block to the last image block to be diffused from the first number of denoising sub-steps after processing the first image block to be diffused in the fifth number of denoising sub-steps, respectively, using the first synthesis key value to process the second image block to the last image block to be diffused in the fourth number of denoising sub-steps, so as to execute each of the model tasks. The fifth number is the fourth number minus 1.

[0062] Optionally, the execution unit is specifically used for:

[0063] Obtain description information of the diffusion target;

[0064] Based on the diffusion target description information, a diffusion guidance vector is generated;

[0065] Guided by the diffusion guiding vector, the second image block to be diffused and the last image block to be diffused of the fourth number are processed using the first synthetic key value.

[0066] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the diffusion processing method as described above.

[0067] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the diffusion processing method as described above.

[0068] According to one aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the diffusion processing method as described above.

[0069] Unlike existing technologies that use a number of preheating steps equal to the number of diffusion processing units and obtain the overall KV parameters for each diffusion processing unit in each preheating step, this embodiment divides the image to be diffused into a second number of preheating blocks. These blocks are then assigned to a second number of preheating processing units. Each preheating processing unit may sequentially execute multiple model tasks within the same preheating step (however, these model tasks are performed on a single preheating block, not the entire image to be diffused, as directly executing multiple model tasks on the entire image in the same preheating step would be too computationally demanding). This allows for the acquisition of KV parameters for multiple model tasks within a single preheating step, which are then passed to the corresponding diffusion processing units. Each diffusion processing unit can obtain the KV parameters for each of the second number of preheating blocks from the second number of preheating processing units and integrate them to obtain the overall KV parameters for the image to be diffused. Compared to existing technologies where a preheating step can only provide KV parameters to one diffusion processing unit, the embodiments of this disclosure may provide preliminary overall KV parameters to multiple diffusion processing units in one preheating step, reducing resource idleness during the preheating process and improving the efficiency of the entire diffusion process.

[0070] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0071] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0072] Figure 1 This is a system architecture diagram of the diffusion processing method applied according to embodiments of the present disclosure;

[0073] Figures 2A-2C This is an application scenario interface diagram of the embodiments of this disclosure applied in the context of text-based image processing.

[0074] Figure 3 This is a general structural diagram of the diffusion model;

[0075] Figure 4 This is a main flowchart of a diffusion processing method according to an embodiment of the present disclosure;

[0076] Figure 5This is a schematic diagram illustrating how the diffusion processing model according to an embodiment of the present disclosure is divided into a first number of model tasks and assigned to a first number of diffusion processing units for execution.

[0077] Figure 6 This is a schematic diagram of dividing an image to be diffused into a second number of preheating blocks according to an embodiment of the present disclosure;

[0078] Figure 7 This is a schematic diagram showing that each preheating processing unit according to an embodiment of the present disclosure sequentially executes a first number of model tasks in the preheating step to obtain a first subkey value corresponding to each model task;

[0079] Figure 8 This is a schematic diagram illustrating how each preheating processing unit, according to an embodiment of the present disclosure, transmits the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task;

[0080] Figure 9 This is a schematic diagram of the steps performed by the preheating treatment unit and the diffusion treatment unit according to embodiments of the present disclosure;

[0081] Figure 10 This is a flowchart illustrating the sequential execution of the first number of model tasks in the preheating step according to an embodiment of the present disclosure;

[0082] Figure 11A This is a schematic diagram illustrating the uneven distribution of a first number of model tasks among the plurality of preheating steps according to an embodiment of the present disclosure.

[0083] Figure 11B This is a schematic diagram illustrating the equal distribution of a first number of model tasks to the plurality of preheating steps according to an embodiment of the present disclosure;

[0084] Figure 12 This is a flowchart illustrating the allocation of a first number of model tasks to multiple warm-up steps according to embodiments of the present disclosure;

[0085] Figure 13 This is a schematic diagram illustrating the allocation of a first number of model tasks to multiple warm-up steps according to an embodiment of the present disclosure;

[0086] Figure 14 This is a flowchart illustrating the determination of the number of preheating steps according to embodiments of the present disclosure;

[0087] Figure 15 This is a flowchart illustrating the process of obtaining diffusion complexity according to embodiments of this disclosure;

[0088] Figure 16A This is a schematic diagram of diffusion target description information and target image according to an embodiment of the present disclosure;

[0089] Figure 16BThis is a schematic diagram of another diffusion target description information and target image according to an embodiment of the present disclosure;

[0090] Figure 17 This is a flowchart illustrating the determination of the number of preheating steps based on model size and diffusion processing complexity according to embodiments of this disclosure.

[0091] Figure 18 This is a flowchart illustrating the determination of a second number according to embodiments of the present disclosure;

[0092] Figure 19 This is a flowchart illustrating the synthesis of a first key value used in a first denoising step using a second number of first sub-key values ​​generated in a second number of preheating processing units according to an embodiment of the present disclosure.

[0093] Figure 20A This is a schematic diagram of dividing the image to be diffused into a second number of preheating blocks according to an embodiment of the present disclosure;

[0094] Figure 20B This is a schematic diagram of integrating the first sub-matrices generated by the second number of preheating processing units, each corresponding to a preheating block, into a first matrix according to the order of the preheating blocks, according to an embodiment of the present disclosure.

[0095] Figure 21 This is a flowchart illustrating how a first subkey value corresponding to a model task is passed to a diffusion processing unit corresponding to a model task, according to an embodiment of this disclosure.

[0096] Figure 22 This is a schematic diagram illustrating how, in the last preheating step of a plurality of preheating steps, the first subkey value corresponding to the model task is sequentially passed to the diffusion processing unit corresponding to the model task according to the order of the model tasks, according to an embodiment of the present disclosure.

[0097] Figure 23 This is a flowchart illustrating how, according to an embodiment of the present disclosure, the first subkey value corresponding to the model task is transmitted to the diffusion processing unit corresponding to the model task at the start time of transmission.

[0098] Figure 24 This is a schematic diagram illustrating the transmission of the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task at the start time of transmission, according to an embodiment of the present disclosure.

[0099] Figure 25 This is a flowchart of an embodiment of the present disclosure, in which subsequent denoising steps after the first denoising step are executed by each diffusion processing unit using the first key value generated in the previous denoising step of the other denoising steps.

[0100] Figure 26This is a schematic diagram illustrating how various model tasks according to embodiments of the present disclosure begin execution in different denoising substeps of other denoising steps;

[0101] Figure 27 This is a flowchart illustrating how various model tasks are performed by various diffusion processing units using a first synthetic key value, according to embodiments of the present disclosure.

[0102] Figure 28A This is a schematic diagram illustrating the division of an image to be diffused into a fourth number of image blocks according to an embodiment of the present disclosure;

[0103] Figure 28B This is a schematic diagram illustrating how each model task is performed using a first synthetic key value through each diffusion processing unit according to an embodiment of the present disclosure.

[0104] Figure 29 This is a flowchart illustrating the process of processing the second image block to be diffused to the last image block to be diffused using the first synthetic key value according to embodiments of the present disclosure.

[0105] Figure 30 This is a detailed flowchart of an application example of a diffusion processing method according to an embodiment of the present disclosure;

[0106] Figure 31 This is a block diagram of a diffusion processing apparatus according to an embodiment of the present disclosure;

[0107] Figure 32 It is performed according to the embodiments of this disclosure. Figure 4 The terminal structure diagram of the diffusion treatment method shown;

[0108] Figure 33 It is performed according to the embodiments of this disclosure. Figure 4 The server architecture diagram for the diffusion processing method is shown. Detailed Implementation

[0109] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0110] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0111] Diffusion models are generative models that generate high-resolution images by simulating the diffusion process of matter in a medium. Diffusion models are inspired by non-equilibrium thermodynamics. They define a Markov chain of diffusion steps to slowly add random noise to the data and then learn to reverse the diffusion process, constructing the desired data sample from the noise. Unlike VAEs or flow-based models, diffusion models are learned with a fixed procedure, and the latent variables have high dimensionality (the same as the original data). Diffusion models include forward diffusion and backward diffusion.

[0112] Forward diffusion: Noise is gradually added to the original image until it becomes completely noisy. The original image can be any image. Adding noise to the original image is done in noisy steps. In each noisy step, the image before and after adding noise is recorded for use in back diffusion.

[0113] Backdiffusion: Starting from a completely noisy state, noise is gradually removed to restore a clear image. This process is achieved through a series of denoising steps, which correspond to the respective noise-adding steps. The noise-adding and denoising steps are collectively called diffusion steps. For example, in forward diffusion, the noise-adding steps are 1, 2, ..., T, while in backdiffusion, the denoising steps are T, T-1, ..., 1. Denoising step T corresponds to noise-adding step T, denoising step T-1 corresponds to noise-adding step T-1...

[0114] Multi-head attention (MBO) is an improved method based on self-attention. Self-attention is a method that computes weights for each position in an input sequence, effectively handling long-distance dependencies. However, in practical applications, multiple different concerns may exist, requiring multiple self-attention mechanisms to address these differences. MBO uses multiple self-attention mechanisms on a single input sequence to obtain multiple attention results, which are then concatenated and linearly projected to obtain the final output. The advantage of MBO is its ability to handle multiple concerns and its superior handling of complex semantic relationships.

[0115] Diffusion models can generate high-resolution images by simulating the diffusion process of matter in a medium. For example, in text-based image processing, a diffusion guidance vector is generated based on the text description in the text-based image. The image to be diffused is input into the diffusion model, and under the guidance of the diffusion guidance vector, progressive denoising is performed. That is, the diffusion process is divided into several denoising steps, and each denoising step removes a portion of noise from the image to be diffused. In this way, after all the denoising steps, the target image that matches the text description is obtained.

[0116] Therefore, during the execution of the diffusion model, it is actually executed once in each denoising step. In each denoising step, to promote parallelism and improve efficiency, the diffusion model is divided into multiple model tasks, each assigned to a diffusion processing unit (such as a GPU). For example, a diffusion module with 10 layers is divided into 5 model tasks, each with 2 layers. These 2 layers are then assigned to a diffusion processing unit, requiring a total of 5 diffusion processing units. Each model task, during execution, cannot rely solely on its own parameters; sometimes it requires parameters obtained from multiple model tasks. For example, the Diffusion Transform Network (DiT), a mainstream diffusion model, has an important multi-head attention model layer, which requires the overall key-value (KV) parameters from multiple model tasks, not just the KV parameters from a single model task. However, when a certain model task is executed, the model task preceding it in the denoising step may have already obtained its KV parameters, while the model task following it may not have obtained their KV parameters in the previous denoising step. Considering that the KV parameters are not significantly different in consecutive denoising steps, the KV parameters of the previous denoising step can be used.

[0117] However, the first denoising step doesn't have a preceding denoising step. Therefore, several warm-up steps are added before the first denoising step. Each diffusion processing unit has one warm-up step. In this warm-up step, the diffusion processing unit processes multiple image blocks to be diffused, obtaining the KV parameters of the overall image to be diffused. After a number of warm-up steps equal to the number of diffusion processing units, the preliminary KV parameters for each diffusion processing unit are obtained. However, the number of warm-up steps is equal to the number of diffusion processing units, resulting in insufficient resource utilization in the warm-up steps and low overall diffusion processing efficiency.

[0118] Based on this, embodiments of this disclosure provide a diffusion processing method, apparatus, electronic device, and medium. The diffusion processing method provided by embodiments of this disclosure can provide preliminary overall KV parameters to multiple diffusion processing units in a preheating step, reducing resource idleness during the preheating process and improving the efficiency of the entire diffusion processing.

[0119] System architecture and scenario description applied in the embodiments of this disclosure

[0120] Figure 1 This is a system architecture diagram of the diffusion processing method applied according to embodiments of the present disclosure. It includes: object terminal 110, Internet 120, gateway 130, and server 140.

[0121] The object terminal 110 is a device used by the object to input the requirements of the diffusion processing for a specific application and to display the image finally generated by the diffusion processing. Furthermore, the actual diffusion processing, including multiple warm-up steps and multiple denoising steps in the diffusion model of this embodiment, can also be performed by the object terminal 110. In this case, the object terminal 110 can independently execute the diffusion processing method of this embodiment. The object terminal 110 includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Additionally, it can be a single device or a collection of multiple devices. For example, multiple devices can be connected via a local area network, sharing a single display device to work collaboratively, collectively forming an object terminal. The object terminal 110 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0122] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages sent from target terminal 110 to server 140 are forwarded to the corresponding server 140 via gateway 130. Messages sent from server 140 to target terminal 110 are also forwarded to the corresponding target terminal 110 via gateway 130.

[0123] Server 140 is a device that provides overall services to all networked object terminals 110. When the diffusion model is installed on server 140, server 140 is responsible for the actual diffusion processing of multiple warm-up steps and multiple denoising steps of the diffusion model in this embodiment. Compared to object terminals 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0124] The embodiments disclosed herein can be applied in various scenarios, such as Figures 2A-2CThe examples shown include textural graph scenarios, molecular graph modeling, and time series modeling. The following will use the textural graph scenario as an example to explain in detail the application of the diffusion processing procedure in this embodiment. Textural graph application refers to the application of generating an image corresponding to the input descriptive text of the generated image. For example, when the input descriptive text is "A crescent moon hangs in the sky, and a person is walking on the road carrying a bag," the output is as follows: Figure 2C The diagram shown is an example of a text-based graph. Currently, text-based graphs are widely used in fields such as chatbots, large language model processing, engineering drawing, and games.

[0125] As mentioned above, when the diffusion model is used in a specific application, it can generate a diffusion guidance vector based on the specific requirements of that application. In texturing image applications, the image to be generated is the target image, and the specific requirements of the application are the description of the target image, such as "a crescent moon hangs in the sky, and a person is walking on the road carrying a bag." Based on the target image description, a diffusion guidance vector is generated. The diffusion guidance vector is a vector specifically used to guide the diffusion model in generating the target image. The image to be diffused is the image before diffusion processing; in texturing image applications, it can be any image. Although the image to be diffused is arbitrary, under the guidance of the diffusion guidance vector, denoising is gradually performed, resulting in an image like... Figure 2C The target image shown.

[0126] like Figure 2A As shown, on the object terminal 110, the object opens the Wenshengtu application. A prompt appears on the interface: "Please enter the text description of the image to be generated:". The object enters in the dialog box: "A crescent moon hangs in the sky, and a person is walking on the road carrying a bag."

[0127] like Figure 2B As shown, after entering the target image description, click the "OK" control next to it. Then, as... Figure 2C The image shown represents the final generated target image. This target image is obtained by diffusion processing using the diffusion processing method of this disclosure embodiment. The specific processing procedure of this disclosure embodiment will be described in detail below.

[0128] It should be understood that the above description only illustrates some application scenarios of this disclosure. The business scenarios to which this disclosure can be applied may include, but are not limited to, the specific embodiments described above.

[0129] General Description of Embodiments in this Disclosure

[0130] It is important to emphasize that the embodiments of this disclosure are applicable to various application scenarios, such as textural image scenarios and molecular image modeling scenarios. In related technologies, several preheating steps are added before the first denoising step. Each diffusion processing unit sets one preheating step. In this preheating step, the diffusion processing unit processes multiple image blocks to be diffused to obtain the KV parameters of the overall image to be diffused. After a number of preheating steps equal to the number of diffusion processing units, the preliminary KV parameters of each diffusion processing unit are obtained. However, the number of preheating steps is set in advance as many as there are diffusion processing units, resulting in insufficient resource utilization in the preheating steps and low efficiency of the entire diffusion processing. One embodiment of this disclosure provides a diffusion processing method, apparatus, electronic device, and medium that can divide the image to be diffused into a second number of preheating blocks and assign them to a second number of preheating processing units for processing. Each preheating processing unit may sequentially execute multiple model tasks in the same preheating step. Thus, the KV parameters of multiple model tasks are obtained through one preheating step, providing preliminary overall KV parameters to multiple diffusion processing units, reducing resource idleness during the preheating process, and improving the efficiency of the entire diffusion processing.

[0131] According to embodiments of this disclosure, diffusion processing refers to the specific process in the diffusion model that comprehensively schedules various computing resources based on the target image description. The diffusion processing method of this disclosure can be executed by the target terminal 110, or by the server 140, or partially by the target terminal 110 and partially by the server 140.

[0132] Before describing the diffusion processing method of the embodiments of this disclosure in detail, please refer to... Figure 3 Let me introduce the basic structure of the diffusion model.

[0133] A diffusion model is a generative model that generates high-resolution images by simulating the diffusion process of matter in a medium. The diffusion space of matter in the medium is called the latent space. The space in which image processing is performed is called the image space. To process image data using the diffusion process, the data in the image space needs to be transformed into data in the latent space; therefore, an encoder and a decoder are required. The encoder transforms the basic image data into a vector z for diffusion processing, and the decoder transforms the target image vector generated by the diffusion model into the diffused target image.

[0134] The entire diffusion process is divided into forward diffusion and backward diffusion. Forward diffusion consists of several noise-adding steps. Noise is added in each noise-adding step until the vector z corresponding to the original image is completely transformed into the vector z corresponding to the noisy image. t The vector z corresponding to the original image undergoes the first noise-adding step to obtain vector z1, then the second noise-adding step to obtain vector z2, and so on, until t noise-adding steps are completed, finally yielding vector z. tVector z t The corresponding image is entirely a noisy image. In each noise-adding step, the image vectors with added noise and without added noise are recorded for use in training the corresponding denoising step in back-diffusion.

[0135] In backdiffusion, the vector z corresponding to the completely noisy image... t Initially, noise is gradually removed through a series of denoising steps to recover a clear image vector. There is a one-to-one correspondence between denoising and noise-adding steps. For example, in forward diffusion, the noise-adding steps are 1, 2, ..., T, while in backward diffusion, the denoising steps are T, T-1, ..., 1. Denoising step T corresponds to noise-adding step T, denoising step T-1 corresponds to noise-adding step T-1, and so on. It starts from the input vector of the denoising step, and each time, using the trained denoising step, the noise for the current denoising step is predicted. The predicted noise is subtracted from the input vector of the denoising step to obtain the denoised vector. Since the noise added in the corresponding noise-adding step and the image vector before adding noise are recorded, in the corresponding denoising step, the image vector before adding noise in the corresponding noise-adding step can be compared with the denoised vector. The difference between the two is used to train the denoising process of each denoising step, making the diffusion model more accurate.

[0136] In practical use of the diffusion model, the inputs are a target image description and the image to be diffused. The target image description is a description of the target image to be generated, such as "a crescent moon hangs in the sky, and a person is walking on the road carrying a bag." The image to be diffused is the image that the diffusion model targets. Since the diffusion model has been trained to be able to progressively denoise any image to obtain the desired target image, in principle, the image to be diffused can be any image. Because the diffusion model can only process vectorized data, the target image description needs to be converted into a diffusion guidance vector. The diffusion guidance vector guides each denoising step of the diffusion model, resulting in the vector of the final generated target image. In practical use of the diffusion model, only the denoising step is performed; the noise-adding step is not performed. The noise-adding step is used in the training process.

[0137] like Figure 3 As shown, each denoising step includes a denoising encoder and a denoising decoder. Each denoising encoder and decoder includes two or more mutual attention models. The input image vector is encoded for denoising using the two or more mutual attention models in the denoising encoder. The result encoded by the denoising encoder is decoded for denoising using the two or more mutual attention models in the denoising decoder. After the denoising encoder and decoder, one denoising step is completed. After T denoising steps, the target image vector corresponding to the diffused target image is recovered.

[0138] likeFigure 4 As shown, according to one embodiment of this disclosure, the diffusion processing method is executed in multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into a first number of model tasks, which are assigned to a first number of diffusion processing units for execution. When each diffusion processing unit executes its assigned model task in each denoising step, it needs to use the first key value generated during the execution of the diffusion processing model in the previous denoising step. The diffusion processing method includes:

[0139] Step 410: In the preheating step before multiple denoising steps, the image to be diffused is divided into a second number of preheating blocks and assigned to a second number of preheating processing units for processing. Each preheating processing unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task.

[0140] Step 420: Pass the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can use the second number of first subkey values ​​generated by the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step, so that the diffusion processing model can be executed in the first denoising step.

[0141] Step 430: In the subsequent denoising steps after the first denoising step, the diffusion processing model is executed through each diffusion processing unit using the first key value generated in the previous denoising step.

[0142] The following is a brief description of steps 410-430 above.

[0143] In step 410, in the preheating step before multiple denoising steps, the image to be diffused is divided into a second number of preheating blocks and assigned to a second number of preheating processing units for processing. Each preheating processing unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task.

[0144] According to embodiments of this disclosure, the diffusion processing model is the model mentioned above that generates high-resolution images by simulating the diffusion process of matter in a medium; further details on the diffusion processing model are omitted here. A denoising step refers to the step in the diffusion processing model that progressively removes noise. The reverse process of the diffusion processing model includes multiple denoising steps, and each denoising step includes a denoising encoder and a denoising decoder. Each denoising encoder and denoising decoder contains two or more mutual attention models.

[0145] Understandably, the diffusion processing model is divided into a first number of model tasks, which are then assigned to a first number of diffusion processing units for execution. A model task is a subset of tasks that need to be executed within the diffusion processing model. A diffusion processing model can be divided into multiple model tasks, where the first number is the total number of model tasks in the diffusion processing model. In one partitioning method, each denoising step can be considered as a model task. If there are N denoising steps, there are N model tasks. In this case, the first number is N. In another partitioning method, all denoising encoders in all denoising steps can be considered as one model task, and all denoising decoders in all denoising steps can be considered as another model task. In this case, the first number is 2. In yet another partitioning method, all mutual attention models with a certain attribute in all denoising steps can be integrated into one model task. In this case, the first number equals the number of attributes. The method of partitioning model tasks is not limited here.

[0146] The first number of model tasks are ordered. When each denoising step is considered a single model task, the order of the model tasks is the same as the order of the denoising steps themselves. For example, if the image to be diffused is input as a concatenated denoising step N, denoising step N-1... denoising step 1, then denoising step N, denoising step N-1... denoising step 1 is the order of the model tasks. When all denoising encoders in all denoising steps are considered as one model task, and all denoising decoders in all denoising steps are considered as another model task, the model tasks of the denoising encoders and decoders are the order of the model tasks. When all mutual attention models with certain attributes in all denoising steps are integrated into one model task, all attributes can be sorted, and the order of the attributes is the order of the model tasks.

[0147] A diffusion processing unit (DPMU) is a processing unit used to execute the model task of diffusion processing during the denoising step, while a warm-up processing unit (WMU) is used to execute the model task of diffusion processing during the warm-up step. Furthermore, the number of DPMUs and WMUs can be predetermined empirically. A processing unit is the hardware structure that executes the model task. For example, in a processor architecture with a CPU and a GPU, the CPU is responsible for instruction scheduling, and each GPU can be the smallest hardware unit for executing the model task, i.e., a processing unit. Figure 3 The denoising encoder and decoder shown are not hardware structures; they can be deployed on the same hardware structure or on different hardware structures. For example, when each denoising step is deployed on a processing unit, the denoising encoder and decoder are deployed on the same processing unit. When all denoising encoders in all denoising steps are treated as one model task, and all denoising decoders in all denoising steps are treated as another model task, the denoising encoder and decoder are deployed on different processing units. Each model task corresponds to one processing unit.

[0148] According to embodiments of this disclosure, the first key value refers to the preliminary KV parameters of the diffusion processing unit. The first key value can also be understood as a caching technique. When the diffusion processing unit performs its assigned model task in each denoising step, the first key value can be used to reuse previous calculation results, thereby reducing redundant calculations and improving the processing speed of the diffusion processing model. For example, this is combined with… Figure 5 The statement "When each diffusion processing unit performs its assigned model task in a denoising step, it needs to use the first key value generated during the execution of the diffusion processing model in the previous denoising step" is explained below. Assume the diffusion processing model is divided into three model tasks, assigned to three diffusion processing units, and the forward process of the diffusion processing model includes ordered denoising steps 1, 2, and 3. In denoising step 1, diffusion processing unit 1 performs the first model task of the diffusion processing model on the image to be diffused, diffusion processing unit 2 performs the second model task, and diffusion processing unit 3 performs the third model task. Similarly, in denoising step 2 following denoising step 1, diffusion processing unit 1 performs the first model task, diffusion processing unit 2 performs the second model task, and diffusion processing unit 3 performs the third model task. Furthermore, taking diffusion processing unit 1 as an example: Assume that during denoising step 1, diffusion processing unit 1 generates a first key value A while performing the first model task of the diffusion processing model. During denoising step 2, diffusion processing unit 1 still needs to use this first key value A while performing the first model task of the diffusion processing model. Assume that during denoising step 2, diffusion processing unit 1 generates a first key value B while performing the first model task of the diffusion processing model. During denoising step 3, diffusion processing unit 1 needs to use this first key value B while performing the first model task of the diffusion processing model.

[0149] According to embodiments of this disclosure, since the first key value generated during the execution of the diffusion processing model in the previous denoising step is required when each diffusion processing unit performs its assigned model task in each denoising step, the first denoising step of the forward process of the diffusion processing model does not have a previous denoising step. Therefore, several warm-up steps can be added before the first denoising step. The warm-up steps serve to provide the first key value for the first denoising step of the forward process of the diffusion processing model. In the warm-up steps, the image to be diffused needs to be divided into a second number of warm-up blocks and assigned to a second number of warm-up processing units for processing. Here, the image to be diffused refers to the input image of the diffusion processing model. As mentioned above, it can be any image because the diffusion processing model has the ability to transform any image into the target image through progressive denoising. In some applications, the image to be diffused can be a specified image. And the warm-up block is a part of the image to be diffused. In embodiments of this disclosure, the image to be diffused is divided into a second number of warm-up blocks. Figure 6 As shown, the second number is 2. The image to be diffused is divided into preheating block 1 and preheating block 2. For example, a 480*800 image to be diffused can be divided into 4 preheating blocks of 120*800 each.

[0150] Understandably, to provide the first key value for the first denoising step, it's necessary to ensure that the number of model tasks sequentially executed by the preheating processing units in the preheating step is consistent with the number of model tasks sequentially executed by the diffusion processing units in the denoising step, so as to obtain the first key value corresponding to each model task. However, since each preheating processing unit only processes a portion of the multiple preheating blocks, the key value corresponding to the model task obtained by each preheating processing unit is only a part of the first key value, i.e., the first sub-key value. For example, here, combined with... Figure 7 The following explanation addresses the statement "Each preheating unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task": Assume the image to be diffused is divided into two preheating blocks, assigned to preheating processing unit 1 and preheating processing unit 2 for processing. For preheating block 1, preheating processing unit 1 sequentially executes model tasks 1, 2, 3, and 4 in the preheating step, obtaining the first subkey values ​​corresponding to model tasks 1, 2, 3, and 4. For preheating block 2, preheating processing unit 2 sequentially executes model tasks 1, 2, 3, and 4 in the preheating step, obtaining the first subkey values ​​corresponding to model tasks 1, 2, 3, and 4.

[0151] The specific process of step 410 will be described in detail later.

[0152] In step 420, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can use the second number of first subkey values ​​generated by the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step, so that the diffusion processing model can be executed in the first denoising step.

[0153] According to embodiments of this disclosure, since multiple diffusion processing units need to use the first key value corresponding to the model task calculated in the preheating step when performing the model task in the first denoising step, each preheating processing unit, after obtaining the first subkey value corresponding to each model task, also needs to pass the first subkey value corresponding to each model task to the diffusion processing unit of the model task. For example, refer to... Figure 7 After obtaining the first subkey value corresponding to model task 1, preheating unit 1 and preheating unit 2 respectively pass the first subkey value corresponding to model task 1 to diffusion processing unit 1 corresponding to model task 1. Similarly, after obtaining the first subkey value corresponding to model task 2, preheating unit 1 and preheating unit 2 respectively pass the first subkey value corresponding to model task 2 to diffusion processing unit 2 corresponding to model task 2. After obtaining the first subkey value corresponding to model task 3, preheating unit 1 and preheating unit 2 respectively pass the first subkey value corresponding to model task 4 to diffusion processing unit 4 corresponding to model task 4.

[0154] Furthermore, the preheating unit can broadcast the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task. Alternatively, the preheating unit can also use P2P (peer-to-peer) communication to transmit the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task.

[0155] According to embodiments of this disclosure, since the first sub-key value corresponding to the model task obtained by each preheating processing unit is only a part of the first key value, each diffusion processing unit, after receiving the first sub-key value corresponding to the model task sent by the preheating processing unit, needs to synthesize the first sub-key value corresponding to the model task sent by the preheating processing unit to obtain the complete first key value. In the first denoising step of the diffusion processing model, the diffusion processing unit can use this complete first key value to execute the corresponding model task. For example, see... Figure 8After receiving the first sub-key value corresponding to the first model task sent by the preheating processing units 1 and 2, diffusion processing unit 1 synthesizes the first key value A used in denoising step 1 to execute the first model task of the diffusion processing model. After receiving the first sub-key value corresponding to the second model task sent by the preheating processing units 1 and 2, diffusion processing unit 2 synthesizes the first key value B used in denoising step 1 to execute the second model task of the diffusion processing model. After receiving the first sub-key value corresponding to the third model task sent by the preheating processing units 1 and 2, diffusion processing unit 3 synthesizes the first key value C used in denoising step 1 to execute the third model task of the diffusion processing model. After receiving the first sub-key value corresponding to the fourth model task sent by the preheating processing units 1 and 2, diffusion processing unit 4 synthesizes the first key value D used in denoising step 1 to execute the fourth model task of the diffusion processing model.

[0156] Understandably, after determining the first sub-key value corresponding to the model task, the preheating unit passes it to the diffusion processing unit corresponding to the model task. This allows the diffusion processing unit to synthesize the first key value corresponding to the model task using multiple first sub-key values, and then use this first key value to perform the denoising step of the diffusion processing model. (Refer to...) Figure 9 The embodiments of this disclosure hide the communication between the preheating unit and the diffusion unit (i.e., the preheating unit passes the first subkey value corresponding to the model task to the diffusion unit corresponding to the model task) in the preheating step and the denoising step, so that the denoising step can be immediately connected after the preheating step, which greatly improves the efficiency of diffusion processing.

[0157] According to embodiments of this disclosure, in order to verify that the processing efficiency of the diffusion processing model will be improved when using the diffusion processing method of this disclosure, embodiments of this disclosure also measure the processing efficiency of the diffusion processing model through multiple experiments to obtain the processing efficiency of the diffusion processing model for different images to be diffused. See Table 1 below:

[0158]

[0159] Table 1

[0160] Referring to Table 1, taking an image of 1024 pixels as an example, the latency of the diffusion processing model without a preheating step is used as a reference. It can be seen that the latency of the diffusion processing model using the prior art is 67% higher than that without a preheating step, while the latency of the diffusion processing model using the method of this disclosure is 60% higher. Therefore, the latency of the diffusion processing model using the method of this disclosure is less than that using the prior art. Furthermore, as the number of pixels in the image to be diffused increases, the latency of the diffusion processing model using the method of this disclosure will be significantly reduced compared to the prior art. Therefore, it can be determined that, compared to the existing setting of "setting as many preheating steps as there are diffusion processing units," the embodiments of this disclosure can provide preliminary overall KV parameters to multiple diffusion processing units in one preheating step, reducing resource idleness during the preheating process and improving the efficiency of the entire diffusion process.

[0161] In step 430, in the subsequent denoising steps after the first denoising step, the diffusion processing model is executed by each diffusion processing unit using the first key value generated in the previous denoising step.

[0162] According to embodiments of this disclosure, in each denoising step after the first denoising step, the diffusion processing unit needs to use the first key value generated in the previous denoising step when executing the diffusion processing model, so as to reuse the previous calculation results and reduce the amount of redundant calculation. For example, assuming that in denoising step 1, for the image to be diffused, diffusion processing unit 1 generates the first key value 1 when executing the first model task of the diffusion processing model, diffusion processing unit 2 generates the first key value 2 when executing the second model task of the diffusion processing model, diffusion processing unit 3 generates the first key value 3 when executing the third model task of the diffusion processing model, and diffusion processing unit 4 generates the first key value 4 when executing the fourth model task of the diffusion processing model. It can be determined that in denoising step 2 after denoising step 1, diffusion processing unit 1 needs to use the first key value 1 when performing the first model task of the diffusion processing model, diffusion processing unit 2 needs to use the first key value 2 when performing the second model task of the diffusion processing model, diffusion processing unit 3 needs to use the first key value 3 when performing the third model task of the diffusion processing model, and diffusion processing unit 4 needs to use the first key value 4 when performing the fourth model task of the diffusion processing model.

[0163] Unlike existing technologies, the embodiments of steps 410 to 430 do not have a number of preheating steps equal to the number of diffusion processing units, and do not obtain the preliminary overall KV parameters for a corresponding diffusion processing unit in each preheating step. Instead, they divide the image to be diffused into a second number of preheating blocks. These blocks are then assigned to a second number of preheating processing units for processing. Each preheating processing unit may sequentially execute multiple model tasks in the same preheating step (however, these model tasks are performed on a single preheating block, not on the entire image to be diffused; directly executing multiple model tasks on the entire image to be diffused in the same preheating step would be too computationally demanding). This allows for the acquisition of KV parameters for multiple model tasks within a single preheating step, which are then passed to the corresponding diffusion processing units. The diffusion processing units can obtain the KV parameters for each of the second number of preheating blocks from the second number of preheating processing units, and integrate them to obtain the overall KV parameters for the image to be diffused. Compared to existing technologies where a preheating step can only provide KV parameters to one diffusion processing unit, the embodiments of this disclosure may provide preliminary overall KV parameters to multiple diffusion processing units in one preheating step, reducing resource idleness during the preheating process and improving the efficiency of the entire diffusion process.

[0164] The above is a general description of steps 410 to 430. The specific implementation process of steps 410 to 430 is described in detail below.

[0165] Detailed description of step 410

[0166] Step 410: In the preheating step before multiple denoising steps, the image to be diffused is divided into a second number of preheating blocks and assigned to a second number of preheating processing units for processing. Each preheating processing unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task.

[0167] In one embodiment, there is only one preheating step. In this case, each preheating processing unit sequentially executes a first number of model tasks in a single preheating step, achieving the goal of obtaining the first sub-key value corresponding to multiple model tasks in one preheating step. For example, suppose the image to be diffused is divided into two preheating blocks, each assigned to a separate preheating processing unit. Preheating processing unit 1 and preheating processing unit 2 sequentially execute three model tasks in a single preheating step. For preheating block 1, preheating processing unit 1 can sequentially obtain the first sub-key value corresponding to the first model task, the first sub-key value corresponding to the second model task, and the first sub-key value corresponding to the third model task in the preheating step. For preheating block 2, preheating processing unit 2 can sequentially obtain the first sub-key value corresponding to the first model task, the first sub-key value corresponding to the second model task, and the first sub-key value corresponding to the third model task in the preheating step. In the denoising step following the single preheating step, diffusion processing unit 1 corresponding to the first model task can synthesize a first key value using the first sub-key value corresponding to the first model task, and then use this first key value to execute the first model task of the diffusion processing model. The diffusion processing unit 2 corresponding to the second model task can synthesize a first key value using the first sub-key value corresponding to the second model task, and then use the first key value to execute the second model task of the diffusion processing model. The diffusion processing unit 3 corresponding to the third model task can synthesize a first key value using the first sub-key value corresponding to the third model task, and then use the first key value to execute the third model task of the diffusion processing model.

[0168] In one embodiment, the preheating step is a plurality of preheating steps, wherein a first number of model tasks are assigned to the plurality of preheating steps, such that the number of model tasks assigned to each preheating step is a third number, and the third number in different preheating steps is not exactly the same.

[0169] Reference Figure 10 Step 410 includes:

[0170] Step 1010: In each preheating step, execute the third number of model tasks corresponding to the preheating step, so that the first number of model tasks are executed in multiple preheating steps.

[0171] Step 1010 is described in detail below:

[0172] According to embodiments of this disclosure, the third number refers to the number of model tasks assigned to each preheating step. The third number in different preheating steps may not be exactly the same, and the sum of the number of model tasks in multiple preheating steps is the first number.

[0173] For example, refer to Figure 11AAssuming there are two warm-up steps before the denoising step, and the diffusion processing model is divided into four model tasks, these four tasks are assigned to the two warm-up steps, such that model task 1 is assigned to warm-up step 1, and model tasks 2, 3, and 4 are assigned to warm-up step 2. It can be determined that in warm-up step 1, each warm-up processing unit executes model task 1, and in warm-up step 2, which follows warm-up step 1, each warm-up processing unit sequentially executes model task 2, model task 3, and model task 4.

[0174] According to another embodiment of this disclosure, when multiple warm-up steps are included before the denoising step, the third number in different warm-up steps can also be exactly the same. In this case, the sum of the number of model tasks in multiple warm-up steps is still the first number.

[0175] For example, refer to Figure 11B Assuming there are two warm-up steps before the denoising step, and the diffusion processing model is divided into four model tasks, these four tasks are distributed across the two warm-up steps, such that warm-up step 1 is assigned model tasks 1 and 2, and warm-up step 2 is assigned model tasks 3 and 4. It can be determined that in warm-up step 1, each warm-up processing unit executes model tasks 1 and 2; in warm-up step 2, following warm-up step 1, each warm-up processing unit sequentially executes model tasks 3 and 4.

[0176] The embodiment of step 1010 above distributes a first number of model tasks into multiple preheating steps, such that the number of model tasks allocated to each preheating step is a third number, thereby executing the third number of model tasks corresponding to each preheating step. By having the preheating processing unit execute model tasks in batches, the risk of excessive load caused by executing too many model tasks in a short period of time is reduced.

[0177] In one embodiment, the first number of model tasks are ordered, referring to... Figure 12 The first number of model tasks in step 410 are distributed across multiple warm-up steps in the following manner:

[0178] Step 1210: Obtain the task size for each model task;

[0179] Step 1220: Using multiple first candidate merging schemes, merge a portion of sequentially adjacent model tasks so that the number of generated merged task blocks is equal to the number of warm-up steps.

[0180] Step 1230: Based on the task size, calculate the total task size of each merged task block in each first candidate merge scheme, and calculate the first variance of the total task size of each merged task block;

[0181] Step 1240: Determine the first candidate merging scheme with the smallest variance as the target merging scheme, and merge a portion of sequentially adjacent model tasks into a warm-up step based on the target merging scheme.

[0182] Steps 1210 to 1240 are described in detail below:

[0183] In step 1210, the task size of each model task is obtained.

[0184] According to embodiments of this disclosure, the diffusion model includes several denoising steps, and each denoising step includes at least two mutual attention models in each of the denoising decoders. Each mutual attention model includes several layers. To reasonably allocate model tasks and balance the processing burden of each preheating processing unit, the model tasks need to be divided precisely at the layer level. The diffusion processing model can be divided into layers first, and then some adjacent layers can be merged to aggregate model tasks. Furthermore, the task size of each model task refers to the size of the layers aggregated by the model task, and the size of the layers aggregated by the model task can be the size of the layer parameters of the layers aggregated by the model task.

[0185] For example, suppose the diffusion processing model includes layers 1, 2, 3, 4, and 5. Layers 1 and 2 can be aggregated into model task 1, and layers 3, 4, and 5 can be aggregated into model task 2. The task size of model task 1 depends on the number of layer parameters in layers 1 and 2, and the task size of model task 2 depends on the number of layer parameters in layers 3, 4, and 5.

[0186] In step 1220, a portion of sequentially adjacent model tasks are merged using multiple first candidate merging schemes, such that the number of generated merged task blocks is equal to the number of preheating steps.

[0187] According to embodiments of this disclosure, the diffusion processing model may contain many layers, and correspondingly, the number of model tasks in the diffusion processing model is also numerous, and in most cases, the number of model tasks is greater than the number of warm-up steps. In this case, it is necessary to merge some sequentially adjacent model tasks so that the number of generated merged task blocks equals the number of warm-up steps. There are multiple merging schemes for this. Each scheme that merges sequentially adjacent model tasks so that the number of generated merged tasks equals the number of warm-up steps is called a first candidate merging scheme.

[0188] For example, refer to Figure 13Assume the diffusion processing model includes model task 1, model task 2, model task 3, and model task 4, with task size A for model task 1, task size B for model task 2, task size C for model task 3, and task size D for model task 4. Assume there are 2 preheating steps. Several sequentially adjacent model tasks can be merged to obtain multiple first candidate merging schemes, where the number of merged task blocks in each first candidate merging scheme equals the number of preheating steps. Specifically, in first candidate merging scheme A, model tasks are assigned to preheating step 1, and sequentially adjacent model tasks 2, 3, and 4 are merged and assigned to preheating step 2. In first candidate merging scheme B, sequentially adjacent model tasks 1 and 2 are merged and assigned to preheating step 1, and sequentially adjacent model tasks 3 and 4 are merged and assigned to preheating step 2. In first candidate merging scheme C, sequentially adjacent model tasks 1, 2, and 3 are merged and assigned to preheating step 1, and model task 4 is assigned to preheating step 2.

[0189] In step 1230, based on the task size, the total task size of each merged task block in each first candidate merge scheme is calculated, and the first variance of the total task size of each merged task block is calculated.

[0190] According to embodiments of this disclosure, the total task size of a merged task block refers to the sum of the sizes of all model tasks merged within that merged task block. The first variance of the total task size of each merged task block refers to the difference in the total task size of different merged task blocks, and the first variance can reflect whether the total task sizes of each merged task block in each first candidate merging scheme are balanced.

[0191] For example, suppose Figure 13 Among the multiple first-candidate merge schemes, the first variance of the total task size of each merge task block in first-candidate merge scheme A is 400Kb. 2 The first variance of the total task size of each merge task block in the first candidate merge scheme B is 1296Kb. 2 The first variance of the total task size of each merge task block in the first candidate merge scheme C is 225Kb. 2 Therefore, it can be determined that the balance of each merged task block in the first candidate merge scheme C is greater than that in the first candidate merge scheme A, which is greater than that in the first candidate merge scheme B.

[0192] In step 1240, the first candidate merging scheme with the smallest variance is determined as the target merging scheme, and based on the target merging scheme, a portion of sequentially adjacent model tasks are merged into a warm-up step.

[0193] According to embodiments of this disclosure, the target merging scheme refers to the final merging scheme for sequentially adjacent model tasks used to divide the model tasks. Minimizing the first variance means that the total task size of each merged task block in the first candidate merging scheme is most balanced. Determining the first candidate merging scheme with the smallest first variance as the target merging scheme can ensure a relatively balanced computational load for the model tasks in each preheating step, which is beneficial for distributing the processing load of each preheating processing unit.

[0194] For example, suppose Figure 13 The first variance of the total task size of each merge task block in the first candidate merge scheme A is 400Kb. 2 The first variance of the total task size of each merge task block in the first candidate merge scheme B is 1296Kb. 2 The first variance of the total task size of each merge task block in the first candidate merge scheme C is 225Kb. 2 Therefore, the first candidate merging scheme C can be determined as the target merging scheme, merging the sequentially adjacent model tasks 1, 2 and 3 and assigning them to warm-up step 1, and assigning model task 4 to warm-up step 2.

[0195] The embodiments of steps 1210 to 1240 above can select the target merging scheme according to the principle of minimizing the first variance of the total task size of each merging task block in the first candidate merging scheme, so that the computational load of the model tasks in each preheating step is relatively balanced, which is conducive to distributing the processing load of each preheating processing unit and improving resource utilization.

[0196] In one embodiment, the specific implementation of allocating the first number of model tasks to multiple warm-up steps in step 410 has already been described above; here, the method for determining the number of warm-up steps will be described. (Refer to...) Figure 14 The number of preheating steps is determined as follows:

[0197] Step 1410: Obtain the model size of the diffusion processing model;

[0198] Step 1420: Obtain the diffusion processing complexity;

[0199] Step 1430: Determine the number of preheating steps based on the model size and diffusion processing complexity.

[0200] Steps 1410 to 1430 are described in detail below:

[0201] In step 1410, the model size of the diffusion processing model is obtained.

[0202] According to embodiments of this disclosure, the diffusion processing model includes multiple denoising steps, and each denoising step includes multiple mutual attention models, etc. Each mutual attention model may have multiple layers, and each layer has some layer parameters. The model size of the diffusion processing model can be understood as the sum of the sizes of the layer parameters of the mutual attention models in the denoising steps included in the diffusion processing model. The model size of the diffusion processing model can be obtained by querying the size of the diffusion processing model in the object interface.

[0203] Moreover, the model size of the diffusion processing model is directly proportional to the number of preheating steps. The larger the model size of the diffusion processing model, the more complex the diffusion processing model is, and the more model tasks may be in the diffusion processing model, requiring more preheating steps.

[0204] In step 1420, the diffusion processing complexity is obtained.

[0205] According to embodiments of this disclosure, diffusion processing complexity refers to the complexity of the target image to be generated. Although the target image has not yet been generated, its complexity can be largely determined based on the target image description. Moreover, diffusion processing complexity is directly proportional to the number of preheating steps; the greater the diffusion processing complexity, the more complex the target image to be generated, and the more preheating steps are required.

[0206] Furthermore, the complexity of diffusion processing can be determined by multiple factors, and the specific methods for "obtaining the complexity of diffusion processing" will be described in detail below.

[0207] In step 1430, the number of preheating steps is determined based on the model size and diffusion processing complexity.

[0208] According to embodiments of this disclosure, both the model size and the diffusion processing complexity are proportional to the number of preheating steps. A larger model size results in greater diffusion processing complexity, indicating a more complex target image to be generated and a higher computational load on the preheating processing unit. To minimize the computational load on the preheating processing unit and prevent it from malfunctioning due to excessive computational load in a short period, a larger number of preheating steps is required when the number of preheating processing units remains constant.

[0209] For example, suppose there exists a diffusion processing model A and a diffusion processing model B, where the model size of diffusion processing model A is larger than that of diffusion processing model B, and the diffusion processing complexity of diffusion processing model A is greater than that of diffusion processing model B. Therefore, it can be determined that the number of preheating steps for diffusion processing model A is greater than the number of preheating steps for diffusion processing model B. By determining a larger number of preheating steps for diffusion processing model A, the computational load on the preheating processing units used to execute the preheating steps of diffusion processing model A is reduced, avoiding the situation where too much computational load is concentrated in a single preheating step.

[0210] The specific method for "determining the number of preheating steps based on model size and diffusion processing complexity" will be described in detail below.

[0211] The embodiments of steps 1410 to 1430 above quantify the processing difficulty of the diffusion processing model by obtaining the model size and diffusion processing complexity of the diffusion processing model, and determining the number of preheating steps based on the model size and diffusion processing complexity of the diffusion processing model, thereby improving the accuracy of determining the number of preheating steps.

[0212] In one embodiment, reference is made to Figure 15 Step 1420 includes:

[0213] Step 1510: Obtain diffusion target description information;

[0214] Step 1520: Based on the diffusion target description information, obtain the number of elements in the diffused image and the relationship information between the elements in the diffused image;

[0215] Step 1530: Determine the first sub-complexity based on the number of elements in the diffused image;

[0216] Step 1540: Determine the second sub-complexity based on the element relationship information of the diffused image;

[0217] Step 1550: Determine the diffusion processing complexity based on the first and second sub-complexities.

[0218] Steps 1510 to 1550 are described in detail below:

[0219] In step 1510, the diffusion target description information is obtained.

[0220] According to embodiments of this disclosure, diffusion target description information refers to a textual description of the target image to be generated. For example, see... Figure 16A The diffusion target description information could be something like, "There are white clouds in the blue sky, a house below, and a tree next to the house." Furthermore, this diffusion target description information can be obtained through input via an object's interface; for example, the object can use voice input, causing the terminal to convert the object's speech into the diffusion target description information. Moreover, the diffusion target description information can also be obtained from a specified portion of the corpus, such as generating illustrations for a novel based on a specified paragraph.

[0221] In step 1520, based on the diffusion target description information, the number of image elements after diffusion and the relationship information between the image elements after diffusion are obtained.

[0222] According to embodiments of this disclosure, the number of elements in the diffused image refers to the number of elements in the description information of the diffused target. (Refer to...)Figure 16A The description of the diffusion target is "blue sky with white clouds, a house below, and a tree next to the house." This indicates that the elements in the diffusion target description include "blue sky," "white clouds," "house," and "tree," thus determining that the number of elements in the diffused image is four. (Refer to...) Figure 16B The description of the diffusion target is "white clouds in the blue sky, a house below, a tree next to the house, a person under the tree, and a bag in the person's hand". It can be determined that the elements in the description of the diffusion target include "blue sky", "white clouds", "house", "tree", "person" and "bag", and thus the number of elements in the diffused image is determined to be 6.

[0223] Post-diffusion image element relationship information refers to the information about the interaction relationships between various elements in the description information of the diffusion target. (See reference...) Figure 16A The target description information for the diffusion is "white clouds in a blue sky, a house below it, and a tree next to the house." Therefore, the diffused image elements include "white clouds in a blue sky," "a house under the white clouds," and "a tree next to the house." (Refer to...) Figure 16B The description of the diffusion target is "white clouds in a blue sky, a house below the house, a tree next to the house, a person under the tree, and the person holding a bag." This determines that the relationship between the elements in the diffused image includes "white clouds in a blue sky," "a house under the clouds," "a tree next to the house," and "the person holding a bag." Therefore, we can conclude that... Figure 16B The information about the relationship between elements in the diffused image is greater than that in the previous image. Figure 16A The relationships between image elements become complex after diffusion.

[0224] In step 1530, the first sub-complexity is determined based on the number of elements in the diffused image.

[0225] According to embodiments of this disclosure, the number of image elements after diffusion is proportional to the first sub-complexity. Various implementation methods are possible based on the number of image elements after diffusion.

[0226] In one implementation, a lookup table approach can be used. The table matches the number of elements in the diffused image with the first sub-complexity, obtained by consulting a pre-defined table that maps the number of elements to the first sub-complexity. Table 2 below shows a practical example of this mapping.

[0227] Number of image elements after diffusion First sub-complexity 0-10 5 11-20 8 21-30 12 ...... ......

[0228] Table 2

[0229] For example, if the number of image elements after diffusion is 15, looking up Table 2 yields a first sub-complexity of 8. Using table lookup is simple and easy, improving the efficiency of determining the complexity of the diffusion process.

[0230] In another implementation, the first sub-complexity can be determined based on the number of elements in the diffused image, or by substituting into a formula. Substituting the number of elements in the diffused image into a pre-defined formula for the number of elements in the diffused image and the first sub-complexity yields the first sub-complexity corresponding to the number of elements in the diffused image.

[0231] For example, suppose the pre-set formula for the number of elements in the diffused image and the first sub-complexity is: First sub-complexity = Number of elements in the diffused image / 2. If the number of elements in the diffused image is 10, the first sub-complexity can be determined to be 5 using the pre-set formula for the number of elements in the diffused image and the first sub-complexity.

[0232] In step 1540, the second sub-complexity is determined based on the element relationship information of the diffused image.

[0233] According to embodiments of this disclosure, to more accurately measure the complexity of the relationship information of the elements in the diffused image, the relationship information can be quantified first. Assuming the relationship information includes "white clouds in a blue sky," "a house under the white clouds," and "a tree next to the house," the relationship information can be quantized as 3. Assuming the relationship information includes "white clouds in a blue sky," "a house under the white clouds," "a tree next to the house," and "a person holding a bag," the relationship information can be quantized as 4.

[0234] It is understandable that the relationship information of the image elements after diffusion is proportional to the second sub-complexity. There are multiple ways to determine the second sub-complexity based on the relationship information of the image elements after diffusion.

[0235] In one implementation, a lookup table approach can be used. Based on the post-diffusion image element relationship information, a pre-set table mapping post-diffusion image element relationship information to the second sub-complexity is consulted to obtain the second sub-complexity corresponding to the post-diffusion image element relationship information. Table 3 below is a practical example of a table mapping post-diffusion image element relationship information to the second sub-complexity:

[0236] Image element relationship information after diffusion Second sub-complexity 0-5 5 6-10 8 11-20 12 ...... ......

[0237] Table 3

[0238] For example, if the element relationship information of the diffused image is 8, looking up Table 3, we obtain the second sub-complexity as 8. Using table lookup is simple and easy, improving the efficiency of determining the complexity of the diffusion process.

[0239] In another implementation, the second sub-complexity can be determined based on the relationship information of the diffused image elements by substituting into a formula. The relationship information of the diffused image elements is substituted into a pre-set formula for the relationship information of the diffused image elements and the second sub-complexity to obtain the second sub-complexity corresponding to the relationship information of the diffused image elements.

[0240] For example, suppose the pre-set formula for the relationship information of the diffused image elements and the second sub-complexity is: Second sub-complexity = Relationship information of the diffused image elements * 5. If the relationship information of the diffused image elements is 4, the second sub-complexity can be determined to be 20 by using the pre-set formula for the relationship information of the diffused image elements and the second sub-complexity.

[0241] In step 1550, the diffusion processing complexity is determined based on the first sub-complexity and the second sub-complexity.

[0242] According to embodiments of this disclosure, the diffusion processing complexity can be determined based on a first sub-complexity and a second sub-complexity in various ways.

[0243] In one implementation, the sum of the first and second sub-complexities can be determined as the diffusion processing complexity. For example, assuming the first sub-complexity is 5 and the second sub-complexity is 10, the diffusion processing complexity can be determined to be 15.

[0244] In another implementation, the average of the first and second sub-complexities can be determined as the diffusion processing complexity. For example, assuming the first sub-complexity is 10 and the second sub-complexity is 20, the diffusion processing complexity can be determined to be 15.

[0245] In another implementation, different weights can be assigned to the first and second sub-complexities, and the weighted sum of the first and second sub-complexities can be determined as the diffusion processing complexity. For example, assuming the first sub-complexity is 5, the second sub-complexity is 10, the weight of the first sub-complexity is 0.2, and the weight of the second sub-complexity is 0.3, the diffusion processing complexity can be determined to be 4.

[0246] The embodiments of steps 1510 to 1550 above determine the first sub-complexity based on the number of elements in the diffused image, determine the second sub-complexity based on the relationship information of the elements in the diffused image, and determine the diffusion processing complexity based on the first and second sub-complexities. This approach considers both the number of elements in the diffusion target description information and the interaction relationships between these elements, determining the diffusion processing complexity from multiple dimensions. This is more accurate than determining it from a single dimension and improves the efficiency of the diffusion processing.

[0247] In one embodiment, reference is made to Figure 17 Step 1430 includes:

[0248] Step 1710: Determine the first score based on the model size;

[0249] Step 1720: Determine the second score based on the diffusion processing complexity;

[0250] Step 1730: Determine the number of preheating steps based on the first score and the second score.

[0251] Steps 1710 to 1730 are described in detail below:

[0252] In step 1710, the first score is determined based on the model size.

[0253] According to embodiments of this disclosure, determining the first score based on the model size can be implemented in various ways.

[0254] In one implementation, a lookup table can be used. The model size is searched against a pre-defined table mapping model size to the first score to obtain the corresponding first score. Table 4 below shows a practical example of this mapping:

[0255] Model size First score 0-5 GB 10 5 GB-10 GB 20 10-20 GB 30 ...... ......

[0256] Table 4

[0257] For example, if the model size is 15GB, looking up Table 4 yields a score of 30. Using table lookups is simple and convenient, greatly reducing processing overhead.

[0258] In another implementation, the first score can be determined based on the model size by substituting into a formula. The model size is then substituted into a pre-defined formula for model size and first score to obtain the first score corresponding to the model size.

[0259] For example, suppose the pre-set formula for the model size and the first score is: First Score = Model Size / 1GB. Assuming the model size is 8GB, the first score can be determined to be 8 using the pre-set formula.

[0260] In step 1720, the second score is determined based on the diffusion processing complexity.

[0261] According to embodiments of this disclosure, there are various ways to determine the second score based on the diffusion processing complexity.

[0262] In one implementation, a lookup table approach can be used. The second score corresponding to the diffusion processing complexity is obtained by looking up a pre-defined table mapping diffusion processing complexity to the second score.

[0263] In another implementation, the second score can be determined based on the diffusion processing complexity by substituting into a formula. Substituting the diffusion processing complexity into a pre-defined formula for the diffusion processing complexity and the second score yields the second score corresponding to the diffusion processing complexity.

[0264] In step 1730, the number of preheating steps is determined based on the first score and the second score.

[0265] It is understandable that determining the number of preheating steps based on the first and second scores includes: determining the third score based on the first and second scores; and determining the number of preheating steps based on the third score.

[0266] According to embodiments of this disclosure, there are various ways to determine a third score based on a first score and a second score.

[0267] In one implementation, the sum of the first and second scores can be used to determine the third score. For example, assuming the first score is 50 and the second score is 30, the third score can be determined as 80.

[0268] In another implementation, the third score can be determined by the average of the first and second scores. For example, assuming the first score is 50 and the second score is 30, the third score can be determined as 40.

[0269] In another implementation, different weights can be assigned to the first and second scores, and the weighted sum of the first and second scores can be used to determine the third score. For example, assuming the first score is 50, the second score is 30, the weight of the first score is 0.2, and the weight of the second score is 0.3, the third score can be determined as 19.

[0270] According to embodiments of this disclosure, there are various ways to determine the number of preheating steps based on a third fraction.

[0271] In one implementation, a lookup table can be used to find the pre-set correspondence between the third score and the number of preheating steps, and obtain the number of preheating steps corresponding to the third score.

[0272] In another implementation, the number of preheating steps is determined based on the third fraction. Alternatively, a substitution formula can be used. Substituting the third fraction into a pre-set formula for the number of preheating steps yields the number of preheating steps corresponding to the third fraction.

[0273] The embodiments of steps 1710 to 1730 above determine a first score based on the model size, a second score based on the diffusion processing complexity, and finally determine the number of preheating steps based on the first and second scores. This method comprehensively considers various factors affecting the number of preheating steps, improving the accuracy of determining the number of preheating steps.

[0274] Detailed description of step 420

[0275] In step 420, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can use the second number of first subkey values ​​generated by the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step, so that the diffusion processing model can be executed in the first denoising step.

[0276] In one embodiment, reference is made to Figure 18 The second number in step 420 is determined in the following way:

[0277] Step 1810: Obtain the size of the image to be diffused;

[0278] Step 1820: Obtain the business type used for diffusion processing;

[0279] Step 1830: Obtain the target diffusion processing efficiency;

[0280] Step 1840: Determine the second number based on the size of the image to be diffused, the service type, and the target diffusion processing efficiency.

[0281] Steps 1810 to 1840 are described in detail below:

[0282] In step 1810, the size of the image to be diffused is obtained.

[0283] According to embodiments of this disclosure, the image to be diffused refers to the input image of the diffusion processing model, and the size of the image to be diffused refers to the size of the input image of the diffusion processing model. The size of the image to be diffused affects the number of preheating blocks obtained after dividing the image to be diffused. The larger the size of the image to be diffused, the greater the computational load when the diffusion processing method is executed subsequently. In this case, it is necessary to divide the image to be diffused into more preheating blocks and distribute the preheating blocks to multiple preheating processing units for execution, so as to balance the computational load of each preheating processing unit and avoid one preheating processing unit bearing too much computational load.

[0284] In step 1820, the business type used for diffusion processing is obtained.

[0285] According to embodiments of this disclosure, the business type of diffusion processing refers to the application scenario of the diffusion processing model. Business types of diffusion processing include: textural image processing scenarios, molecular detection scenarios, and image processing scenarios, etc. The complexity required for diffusion processing varies depending on the business type.

[0286] For example, when diffusion processing is used in molecular detection applications, the accuracy requirements for generating molecular maps are very high, necessitating greater processing complexity than for ordinary textural image processing, and requiring higher pixel counts in the generated molecular maps. Therefore, the diffusion image needs to be divided into more preheating blocks, and these preheating blocks need to be assigned to multiple preheating processing units for execution to meet the accuracy requirements of the application type.

[0287] In step 1830, the target diffusion processing efficiency is obtained.

[0288] According to embodiments of this disclosure, target diffusion processing efficiency refers to the processing efficiency of the desired image to be diffused. The shorter the processing time of the desired image to be diffused, the higher the target diffusion processing efficiency; conversely, the longer the processing time, the lower the target diffusion processing efficiency. Furthermore, if a higher target diffusion processing efficiency is required, the diffusion image can be divided into more preheating blocks, allowing multiple preheating processing units to process multiple preheating blocks in parallel, thereby improving the target diffusion processing efficiency.

[0289] In step 1840, a second number is determined based on the size of the image to be diffused, the service type, and the target diffusion processing efficiency.

[0290] Understandably, the second number is determined based on the size of the image to be diffused, the service type, and the target diffusion processing efficiency, including: determining the fourth score based on the size of the image to be diffused; determining the fifth score based on the service type; determining the sixth score based on the target diffusion processing efficiency; determining the seventh score based on the fourth, fifth, and sixth scores; and determining the second number based on the seventh score.

[0291] According to embodiments of this disclosure, there are various ways to determine the fourth score based on the size of the image to be diffused.

[0292] In one implementation, a lookup table can be used. A pre-defined table mapping image size to fourth score is consulted to obtain the fourth score corresponding to the image size to be diffused.

[0293] In another implementation, the fourth score can be determined based on the size of the image to be diffused by substituting it into a formula. The size of the image to be diffused is then substituted into a pre-defined formula for the fourth score to obtain the fourth score corresponding to that image size.

[0294] According to embodiments of this disclosure, there are multiple ways to determine the fifth score based on the business type.

[0295] In one implementation, a table lookup method can be used. The pre-defined table mapping business type to fifth score is consulted to obtain the fifth score corresponding to the business type.

[0296] In another implementation, the fifth score can be determined based on the business type by substituting into a formula. The business type is then substituted into a pre-defined formula for the business type and the fifth score to obtain the fifth score corresponding to that business type.

[0297] According to embodiments of this disclosure, determining the sixth score based on the target diffusion processing efficiency can be implemented in various ways.

[0298] In one implementation, a lookup table approach can be used. Based on the target diffusion processing efficiency, a pre-set table mapping target diffusion processing efficiency to the sixth score is consulted to obtain the sixth score corresponding to the target diffusion processing efficiency.

[0299] In another implementation, the sixth score can be determined based on the target diffusion processing efficiency by substituting into a formula. The target diffusion processing efficiency is then substituted into a pre-set formula for the target diffusion processing efficiency and the sixth score to obtain the sixth score corresponding to the target diffusion processing efficiency.

[0300] According to embodiments of this disclosure, there are various ways to determine the seventh score based on the fourth, fifth, and sixth scores.

[0301] In one implementation, the sum of the fourth, fifth, and sixth fractions can be used to determine the seventh fraction. For example, assuming the fourth fraction is 10, the fifth fraction is 20, and the sixth fraction is 30, the seventh fraction can be determined as 60.

[0302] In another implementation, the seventh score can be determined by the average of the fourth, fifth, and sixth scores. For example, assuming the fourth score is 50, the fifth score is 30, and the sixth score is 40, the seventh score can be determined as 40.

[0303] In another implementation, different weights can be assigned to the fourth, fifth, and sixth scores, and the weighted sum of the fourth, fifth, and sixth scores can be determined as the seventh score. For example, assuming the fourth score is 50, the fifth score is 30, the sixth score is 40, the weight of the fourth score is 0.2, the weight of the fifth score is 0.3, and the weight of the sixth score is 0.1, the seventh score can be determined as 23.

[0304] According to embodiments of this disclosure, there are various ways to determine the second number based on the seventh fraction.

[0305] In one implementation, a lookup table can be used to find the pre-set correspondence between the seventh score and the second number, and obtain the second number corresponding to the seventh score.

[0306] In another implementation, the second number is determined based on the seventh fraction. Alternatively, a substitution formula can be used. Substituting the seventh fraction into a pre-defined formula for the seventh fraction and the second number yields the second number corresponding to the seventh fraction.

[0307] The embodiments of steps 1810 to 1840 above can determine the second number based on the size of the image to be diffused, the service type, and the target processing efficiency, taking into account a variety of factors that affect the second number, thus improving the accuracy of determining the second number.

[0308] In one embodiment, the first subkey value is represented as a first submatrix corresponding to the preheating block, and the first key value is represented as a first matrix corresponding to the image to be diffused;

[0309] Reference Figure 19 Step 420, "using the second number of first sub-key values ​​generated in the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step," includes:

[0310] Step 1910: The first sub-matrices generated by the second number of preheating units and corresponding to each preheating block are integrated into a first matrix according to the order of the preheating blocks.

[0311] Step 1910 is described in detail below:

[0312] According to embodiments of this disclosure, the first matrix refers to the representation of the first key value. The first key value can be represented as a first matrix corresponding to the image to be diffused, and each element in the first matrix is ​​equivalent to a sampled value of the image to be diffused.

[0313] The first submatrix refers to the representation of the first subkey value. After dividing the image to be diffused into a second number of preheating blocks, the first subkey value obtained by the preheating processing unit after processing the preheating blocks can be represented as the first submatrix corresponding to the preheating block, and each element in the first submatrix is ​​equivalent to the sampled value in the preheating block corresponding to the first submatrix.

[0314] It is understandable that the first key value used in the first denoising step can be synthesized from the second number of first sub-key values ​​generated by the second number of preheating processing units in the preheating step, and naturally, the first sub-matrix can be used to integrate it into the first matrix. It can be imagined that when the image to be diffused is divided into the second number of preheating blocks, the order in which the preheating blocks are divided is the order in which the first sub-matrix is ​​integrated into the first matrix.

[0315] For example, refer to Figure 20A Assuming that when dividing the image to be diffused into the second number of preheating blocks, the image is divided into four equal parts in a top-to-bottom order, resulting in preheating block 1, preheating block 2, preheating block 3, and preheating block 4. Correspondingly, refer to... Figure 20B After obtaining the first sub-matrix 1 corresponding to preheating block 1 generated by preheating unit 1, the first sub-matrix 2 corresponding to preheating block 2 generated by preheating unit 2, the first sub-matrix 3 corresponding to preheating block 3 generated by preheating unit 3, and the first sub-matrix 4 corresponding to preheating block 4 generated by preheating unit 4, it is necessary to proceed according to... Figure 20A The order of preheating blocks 1, 2, 3 and 4 is used to integrate the first submatrix 1, 2, 3 and 4 to obtain the first matrix.

[0316] The embodiment of step 1910 above integrates the first sub-matrices generated by the second number of preheating processing units, which are corresponding to each preheating block, into a first matrix according to the order of the preheating blocks. When integrating the first matrix, the division order when dividing the image to be diffused into preheating blocks is taken into account, which improves the integration accuracy and ensures the integrity of the determined first matrix.

[0317] In one embodiment, the preheating step comprises multiple preheating steps;

[0318] Reference Figure 21 Step 420, "passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task," includes:

[0319] Step 2110: In the last preheating step of multiple preheating steps, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task in the order of the model tasks.

[0320] Step 2110 will be described in detail below:

[0321] According to embodiments of this disclosure, in the presence of multiple preheating steps, for each preheating block, the preheating processing unit will sequentially obtain the first subkey value corresponding to each model task and pass the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task. Furthermore, there are multiple implementation methods for the preheating processing unit to pass the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task. A first implementation method is: in the last preheating step of the multiple preheating steps, the first subkey value corresponding to the model task is passed sequentially to the diffusion processing unit corresponding to the model task according to the order of the model tasks.

[0322] Understandably, in the final preheating step of multiple preheating steps, the preheating unit has already determined the first subkey values ​​corresponding to the multiple model tasks preceding the diffusion processing model. Since the diffusion processing unit first executes the preceding model tasks and then sequentially executes the other model tasks, to ensure that the diffusion processing unit can synthesize the first key values ​​corresponding to the preceding model tasks in a timely manner and use these first key values ​​to execute the model tasks, the preheating unit needs to pass the first subkey values ​​corresponding to the model tasks to the corresponding diffusion processing units in the order of the model tasks. Based on this, the diffusion processing unit can synthesize the first key values ​​corresponding to the model tasks sequentially in the order of the model tasks and use these first key values ​​to execute the model tasks.

[0323] Furthermore, refer to Figure 22 It can be determined that at the start of the last preheating step of multiple preheating steps, the preheating processing unit has not yet obtained the first subkey values ​​corresponding to all model tasks of the diffusion processing model. However, when the last preheating step ends, the preheating processing unit has obtained the first subkey values ​​corresponding to all model tasks of the diffusion processing model. After obtaining the first subkey values ​​corresponding to all model tasks of the diffusion processing model, the preheating processing unit sends the first subkey values ​​corresponding to the model tasks that have not yet been sent to the diffusion processing unit, so that the diffusion processing unit can synthesize the first key value using the first subkey values ​​before executing the denoising step of the diffusion processing model. It is understood that the embodiments of this disclosure, by enabling the preheating step and the transmission of the first subkey values ​​corresponding to the model tasks to the diffusion processing unit corresponding to the model tasks to be transmitted concurrently within the same time period, without waiting for the end of the last preheating step, reduce the time for the diffusion processing model to process the image to be diffused and improve the execution efficiency of the diffusion processing model.

[0324] According to another embodiment of this disclosure, a second implementation of "the preheating processing unit transmits the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task" is: after each of the multiple preheating steps is completed, the first subkey value corresponding to the model task allocated in the preheating step is transmitted to the diffusion processing unit corresponding to the model task.

[0325] Understandably, given multiple warm-up steps, each step will be assigned a portion of the model tasks. Therefore, at the end of each warm-up step, the warm-up processing unit will receive the first subkey value corresponding to the assigned model task. To avoid a surge in communication pressure caused by sending all the first subkey values ​​of all model tasks at once, the warm-up processing unit can transmit the first subkey value corresponding to the assigned model task to the corresponding diffusion processing unit at the end of each warm-up step. By transmitting the first subkey values ​​of multiple model tasks in batches, the communication pressure between the warm-up processing unit and the diffusion processing unit can be alleviated.

[0326] For example, refer to Figure 7 Assuming there are three preheating steps: in preheating step 1, the preheating unit executes model task 1; in preheating step 2, the preheating unit executes model tasks 2 and 3 sequentially; and in preheating step 3, the preheating unit executes model task 4. It can be seen that after preheating step 1, the preheating unit passes the first subkey value corresponding to model task 1 to diffusion processing unit 1. After preheating step 2, the preheating unit passes the first subkey value corresponding to model task 2 to diffusion processing unit 2 and the first subkey value corresponding to model task 3 to diffusion processing unit 3. After preheating step 3, the preheating unit passes the first subkey value corresponding to model task 4 to diffusion processing unit 4.

[0327] The embodiment of step 2110 above, by passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task sequentially according to the order of the model tasks in the last preheating step of multiple preheating steps, allows the preheating step and the passing of the first subkey value corresponding to the model task to the diffusion processing unit to the model task to be executed concurrently within the same time period. This eliminates the need to wait until the last preheating step ends before passing the first subkey value corresponding to the model task to the diffusion processing unit, reducing the time the diffusion processing model spends processing the image to be diffused and improving the execution efficiency of the diffusion processing model.

[0328] It is understood that the first two implementations of "the preheating processing unit passes the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task" have been described above. Here, the third implementation of "the preheating processing unit passes the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task" will be described:

[0329] In one embodiment, each model task begins execution in a different denoising substep of the first denoising step;

[0330] Reference Figure 23 Step 420, "passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task," includes:

[0331] Step 2310: Obtain the first size of the first subkey value;

[0332] Step 2320: Based on the first size, predict the first transmission time of the first subkey value;

[0333] Step 2330: Obtain the start time of the denoising sub-step corresponding to the first subkey value;

[0334] Step 2340: Determine the transmission start time based on the execution start time and the first transmission time;

[0335] Step 2350: At the start of transmission, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

[0336] Steps 2310 to 2350 are described in detail below:

[0337] In step 2310, the first size of the first subkey value is obtained.

[0338] According to embodiments of this disclosure, the first size refers to the size of the first subkey value, and the first size affects the time taken by the preheating processing unit to transmit the first subkey value to the diffusion processing unit. The larger the first size, the longer the time taken by the preheating processing unit to transmit the first subkey value to the diffusion processing unit; conversely, the smaller the first size, the shorter the time taken by the preheating processing unit to transmit the first subkey value to the diffusion processing unit. Furthermore, the first size of the first subkey value can be obtained by querying the first subkey value in the object interface.

[0339] In step 2320, the first transmission time of the first subkey value is predicted based on the first size.

[0340] According to embodiments of this disclosure, the first transmission time refers to the time taken by the preheating processing unit to transfer the first sub-bond value to the diffusion processing unit. It is conceivable that the first size and the first transmission time are proportional, and therefore the first transmission time can be predicted based on the first size.

[0341] Understandably, there are multiple ways to predict the first transmission time of the first subkey value based on the first size.

[0342] In one implementation, a lookup table can be used to find the correspondence between the first size and the first transmission time in a pre-set table, and obtain the first transmission time corresponding to the first size.

[0343] In another implementation, the first transmission time of the first subkey value is predicted based on the first size. Alternatively, a substitution method can be used. Substituting the first size into a pre-set formula for the first size and first transmission time yields the first transmission time corresponding to the first size.

[0344] In step 2330, the execution start time of the denoising sub-step corresponding to the first subkey value is obtained.

[0345] According to embodiments of this disclosure, each denoising step includes multiple denoising substeps, where each substep is the smallest time unit of the reverse process of the diffusion processing unit. A first number of model tasks can be assigned to multiple denoising substeps for execution, so that in each denoising substep, the diffusion processing unit can execute the model task corresponding to that substep. It is understood that in each denoising substep, the diffusion processing unit needs to execute the model task corresponding to that substep. In the denoising substep of the first denoising step, the diffusion processing unit needs to use the first key value of the model task when executing it, and the first key value of the model task is obtained by integrating the first subkey values ​​corresponding to the model task passed by multiple preheating processing units. The execution start time of the denoising substep corresponding to the first subkey value refers to the execution start time of the model task corresponding to the first subkey value.

[0346] For example, refer to Figure 24Assume the denoising process includes denoising sub-steps 1, 2, 3, and 4, and the diffusion processing unit starts executing model task 1 at denoising sub-step 1, model task 2 at denoising sub-step 2, model task 3 at denoising sub-step 3, and model task 4 at denoising sub-step 4. When executing model task 1, the diffusion processing unit needs to use the first key value A formed by integrating the first subkey value A; therefore, the start time of denoising sub-step 1 is determined as the execution start time of the denoising sub-step corresponding to the first subkey value A. Similarly, when executing model task 2, the diffusion processing unit needs to use the first key value B formed by integrating the first subkey value B; therefore, the start time of denoising sub-step 2 is determined as the execution start time of the denoising sub-step corresponding to the first subkey value B. When executing model task 3, the diffusion processing unit needs to use the first key value C formed by integrating the first subkey value C; therefore, the start time of denoising sub-step 3 is determined as the execution start time of the denoising sub-step corresponding to the first subkey value C. When the diffusion processing unit performs model task 4, it needs to use the first key value D integrated from the first subkey value D. Therefore, the start time of the denoising sub-step 4 is determined to be the start time of the execution of the denoising sub-step corresponding to the first subkey value D.

[0347] In step 2340, the transmission start time is determined based on the execution start time and the first transmission time.

[0348] According to embodiments of this disclosure, the transmission start time refers to the time when the preheating processing unit begins transmitting the first subkey value. After determining the execution start time, which means that the first subkey value will be used, and the first transmission time, which means the time taken for the preheating processing unit to transmit the first subkey value to the diffusion processing unit, the transmission start time can be determined as the difference between the execution start time and the first transmission time.

[0349] For example, refer to Figure 24 Once the first transmission time of the first subkey value A and the start time of the execution of the denoising sub-step corresponding to the first subkey value A are determined, the transmission start time of the first subkey value A can be determined. Similarly, once the first transmission time of the first subkey value B and the start time of the execution of the denoising sub-step corresponding to the first subkey value B are determined, the transmission start time of the first subkey value B can be determined. Once the first transmission time of the first subkey value C and the start time of the execution of the denoising sub-step corresponding to the first subkey value C are determined, the transmission start time of the first subkey value C can be determined. Once the first transmission time of the first subkey value D and the start time of the execution of the denoising sub-step corresponding to the first subkey value D are determined, the transmission start time of the first subkey value D can be determined.

[0350] In step 2350, at the start of transmission, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

[0351] According to embodiments of this disclosure, after determining the transmission start time, the first subkey value corresponding to the model task needs to be passed to the diffusion processing unit corresponding to the model task from the transmission start time. For example, refer to... Figure 24 Assume the system includes a preheating unit 1, a preheating unit 2, a diffusion unit 1, a diffusion unit 2, a diffusion unit 3, and a diffusion unit 4. Diffusion unit 1 executes the first model task, diffusion unit 2 executes the second model task, diffusion unit 3 executes the third model task, and diffusion unit 4 executes the fourth model task. Furthermore, diffusion unit 1, when executing the first model task, utilizes the first key value A formed by integrating the first subkey value A; diffusion unit 2, when executing the second model task, utilizes the first key value B formed by integrating the first subkey value B; diffusion unit 3, when executing the third model task, utilizes the first key value C formed by integrating the first subkey value C; and diffusion unit 4, when executing the fourth model task, utilizes the first key value D formed by integrating the first subkey value D. In the transmission process, at the start time of the transmission of the first subkey value A, preheating processing unit 1 and preheating processing unit 2 transmit the first subkey value A corresponding to the first model task they obtained to diffusion processing unit 1, so that diffusion processing unit 1 can receive the first subkey value A at the start of denoising substep 1. At the start time of the transmission of the first subkey value B, preheating processing unit 1 and preheating processing unit 2 transmit the first subkey value B corresponding to the second model task they obtained to diffusion processing unit 2, so that diffusion processing unit 2 can receive the first subkey value B at the start of denoising substep 2. At the start time of the transmission of the first subkey value C, preheating processing unit 1 and preheating processing unit 2 transmit the first subkey value C corresponding to the third model task they obtained to diffusion processing unit 3, so that diffusion processing unit 3 can receive the first subkey value C at the start of denoising substep 3. At the start time of the transmission of the first sub-key value D, the preheating processing unit 1 and the preheating processing unit 2 transmit the first sub-key value D corresponding to the fourth model task obtained by each of them to the diffusion processing unit 4, so that the diffusion processing unit 4 can receive the first sub-key value D at the start of the denoising sub-step 4.

[0352] The embodiments of steps 2310 to 2350 described above can determine the transmission start time of the first subkey value based on the execution start time of the denoising sub-step corresponding to the first subkey value and the first transmission time of the first subkey value, so as to transmit the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task at the transmission start time. By transmitting the first subkey values ​​corresponding to multiple model tasks in batches through the above steps, the situation of communication pressure surge caused by sending the first subkey values ​​of all model tasks at once is avoided, effectively alleviating the communication pressure between the preheating processing unit and the diffusion processing unit.

[0353] Detailed description of step 430

[0354] In step 430, in the subsequent denoising steps after the first denoising step, the diffusion processing model is executed by each diffusion processing unit using the first key value generated in the previous denoising step.

[0355] In one embodiment, each model task begins execution in a different denoising substep of other denoising steps;

[0356] Reference Figure 25 Step 430 includes:

[0357] Step 2510: In the other denoising steps after the first denoising step, for a single diffusion processing unit, obtain the first current subkey value generated by the first model task corresponding to the single diffusion processing unit in the other denoising steps, and the first preceding subkey value generated by the preceding model task before the first model task in the other denoising steps.

[0358] Step 2520: For a single diffusion processing unit, extract the first subsequent sub-key value corresponding to the subsequent model task after the first model task from the first key value generated in the previous denoising step of other denoising steps.

[0359] Step 2530: Combine the first preceding subkey value, the first current subkey value, and the first subsequent subkey value into a first composite key value, and use the first composite key value to execute each model task through each diffusion processing unit.

[0360] Steps 2510 and 2530 are described in detail below:

[0361] According to embodiments of this disclosure, incorporated herein by reference Figure 26 The statement "each model task begins execution in a different denoising sub-step within another denoising step" is explained as follows: Assume a denoising step comprises three denoising sub-steps, and the three sequentially executed model tasks are assigned to these three sub-steps. Specifically, in denoising sub-step 1-1, for image block 1 to be diffused, diffusion processing unit 1 executes the first model task. In denoising sub-step 1-2, for image block 1 to be diffused, diffusion processing unit 2 executes the second model task, and for image block 2 to be diffused, diffusion processing unit 1 executes the first model task. In denoising sub-step 3, for image block 1 to be diffused, diffusion processing unit 3 executes the third model task, for image block 2 to be diffused, diffusion processing unit 2 executes the second model task, and for image block 3 to be diffused, diffusion processing unit 1 executes the first model task.

[0362] In step 2510, in the other denoising steps after the first denoising step, for a single diffusion processing unit, the first current subkey value generated by the first model task corresponding to the single diffusion processing unit in the other denoising steps, and the first preceding subkey value generated by the preceding model task before the first model task in the other denoising steps are obtained.

[0363] According to embodiments of this disclosure, the first model task refers to the model task assigned to the diffusion processing unit. The preceding model task refers to the model task executed before the first model task. The first current subkey value refers to the key value generated by the diffusion processing unit when executing the first model task in other denoising steps, and the first preceding subkey value refers to the key value generated by other diffusion processing units when executing the preceding model task in other denoising steps. Wherein, other denoising steps refer to denoising steps following the first denoising step.

[0364] For example, this is combined with Figure 26 Let's describe the processing in step 2510 in more detail. Assume that denoising step 2 has three sub-steps: denoising sub-step 2-1, denoising sub-step 2-2, and denoising sub-step 2-3. For diffusion processing unit 2, obtain the first current subkey value generated by the first model task corresponding to diffusion processing unit 2 in denoising sub-step 2-2, and the first preceding subkey value generated by the preceding model task (i.e., the model task executed by diffusion processing unit 1) in denoising sub-step 2-1.

[0365] In step 2520, for a single diffusion processing unit, the first subsequent sub-key value corresponding to the subsequent model task after the first model task is extracted from the first key value generated in the previous denoising step of other denoising steps.

[0366] According to embodiments of this disclosure, a subsequent model task refers to a model task whose execution order follows that of the first model task. A first subsequent sub-key value refers to a key value generated by other diffusion processing units when executing a subsequent model task in a denoising step preceding other denoising steps.

[0367] For example, this is combined with Figure 26 Let's describe the process of step 2520 in more detail. Assuming the other denoising step is denoising step 2, for diffusion processing unit 2, the first subsequent sub-key value generated by the subsequent model task (i.e. the model task executed by diffusion processing unit 3) after the first model task is extracted from the first key value generated by the previous denoising step (denoising step 1) of denoising step 2.

[0368] In step 2530, the first preceding subkey value, the first current subkey value, and the first subsequent subkey value are combined into a first composite key value, and each model task is executed using the first composite key value through each diffusion processing unit.

[0369] According to embodiments of this disclosure, the first synthesized key value refers to the integrated key value obtained by integrating the first preceding subkey value, the first current subkey value, and the first following subkey value.

[0370] For example, this is combined with Figure 26 The processing procedure of step 2530 will be described in further detail. Assuming the other denoising steps are denoising step 2, for diffusion processing unit 2, the first current sub-key value generated by the first model task corresponding to diffusion processing unit 2 in denoising sub-step 2-2, the first preceding sub-key value generated by the preceding model task corresponding to diffusion processing unit 1 in denoising sub-step 2-1, and the first subsequent sub-key value generated by the subsequent model task corresponding to diffusion processing unit 3 in denoising sub-steps 1-3 have been determined. Based on this, the first current sub-key value, the first preceding sub-key value, and the first subsequent sub-key value can be integrated to obtain the integrated key value. Since the first synthesized key value integrates the key values ​​corresponding to multiple model tasks, after determining the first synthesized key value, each diffusion processing unit can use the first synthesized key value to execute its respective model task. Furthermore, conventional techniques typically use the key value generated in the previous denoising step for diffusion processing, which is not conducive to the accuracy of diffusion processing. Since the first current sub-key value and the first preceding sub-key value in this embodiment are generated in the current denoising step, using the first synthesized key value obtained by integrating them for diffusion processing can achieve better convergence effect and improve the accuracy of diffusion processing.

[0371] The specific method of “performing each model task using the first synthetic key value through each diffusion processing unit” will be described in detail below.

[0372] The embodiments of steps 2510 to 2530 above integrate the first current sub-key value, the first preceding sub-key value, and the first following sub-key value to obtain an integrated key value, i.e., the first synthesized key value. Since the first synthesized key value integrates key values ​​corresponding to multiple model tasks, after determining the first synthesized key value, each diffusion processing unit can use the first synthesized key value to execute its respective model task. Furthermore, since the first current sub-key value and the first preceding sub-key value are generated in this denoising step, using the first synthesized key value obtained by integrating them for diffusion processing achieves better convergence results and improves the accuracy of diffusion processing compared to using the key value generated in the previous denoising step.

[0373] In one embodiment, reference is made to Figure 27 Step 2530 includes:

[0374] Step 2710: Divide the image to be diffused into a fourth number of image blocks, where the fourth number is equal to the number of denoising sub-steps in the denoising step, and the fourth number is greater than or equal to the first number.

[0375] Step 2720: The first number of denoising sub-steps in the fourth number of denoising sub-steps are respectively used as the first number of diffusion processing units to process the denoising sub-steps of the first image block to be diffused. In the fifth number of denoising sub-steps after processing the first image block to be diffused, the diffusion processing unit uses the first synthesis key value to process the second image block to the last image block to be diffused in the fourth number of sub-steps to execute each model task. The fifth number is the fourth number minus 1.

[0376] Steps 2710 and 2720 are described in detail below:

[0377] In step 2710, the image to be diffused is divided into a fourth number of image blocks, the fourth number being equal to the number of denoising sub-steps in the denoising step, and the fourth number being greater than or equal to the first number.

[0378] According to embodiments of this disclosure, the fourth number is the number of image blocks obtained after dividing the image to be diffused. The number of the fourth number corresponds to the number of denoising sub-steps in the denoising step.

[0379] For example, refer to Figure 28A The image to be diffused can be divided into four blocks, resulting in a fourth number of blocks and four denoising sub-steps within the denoising step. Assuming the first number is three, meaning three model tasks are assigned to three diffusion processing units for execution. (Refer to...) Figure 28B For image block 1 to be diffused, diffusion processing unit 1 executes the first model task in denoising sub-step 1-1, diffusion processing unit 2 executes the second model task in denoising sub-step 1-2, and diffusion processing unit 3 executes the third model task in denoising sub-step 1-3. For image block 2 to be diffused, diffusion processing unit 1 executes the first model task in denoising sub-step 1-2, diffusion processing unit 2 executes the second model task in denoising sub-step 1-3, and diffusion processing unit 3 executes the third model task in denoising sub-step 1-4. For image block 3 to be diffused, diffusion processing unit 1 executes the first model task in denoising sub-step 1-3, diffusion processing unit 2 executes the second model task in denoising sub-step 1-4, and diffusion processing unit 3 executes the third model task in denoising sub-step 2-1.

[0380] In step 2720, the first first number of denoising sub-steps in the fourth number of denoising sub-steps are respectively used as the first number of diffusion processing units to process the denoising sub-steps of the first image block to be diffused. In the fifth number of denoising sub-steps after processing the first image block to be diffused, the diffusion processing unit uses the first synthesis key value to process the second image block to the last image block to be diffused in the fourth number of sub-steps to execute each model task. The fifth number is the fourth number minus 1.

[0381] According to embodiments of this disclosure, the fifth number refers to the difference between the fourth number and one.

[0382] For example, refer to Figure 28A and Figure 28B Here, we will illustrate step 2720 with an example: Assume the image to be diffused can be divided into four image blocks, so the fourth number is 4, and the number of denoising sub-steps in the denoising step is also 4. With a first number of 3, the three model tasks are assigned to three diffusion processing units for execution. In step 2720, for denoising step 1, the first three of the four denoising sub-steps of denoising step 1 are used as denoising sub-steps for the first image block to be diffused by the three diffusion processing units. Furthermore, referring to Figure 28, for diffusion processing unit 1, after processing image block 1 (i.e., executing the first model task for image block 1), the three denoising sub-steps can respectively use the first synthetic key value to process image blocks 2, 3, and 4 to be diffused. Similarly, for diffusion processing unit 2, after processing the image block 1 to be diffused (i.e., performing the second model task on image block 1), the three denoising sub-steps of diffusion processing unit 2 can respectively use the first synthetic key value to process image block 2, image block 3, and image block 4. For diffusion processing unit 3, after processing the image block 1 to be diffused (i.e., performing the third model task on image block 1), the three denoising sub-steps of diffusion processing unit 3 can respectively use the first synthetic key value to process image block 2, image block 3, and image block 4. For diffusion processing unit 4, after processing the image block 1 to be diffused (i.e., performing the fourth model task on image block 1), the three denoising sub-steps of diffusion processing unit 4 can respectively use the first synthetic key value to process image block 2, image block 3, and image block 4.

[0383] The specific method of “processing the second image block to be diffused to the last image block of the fourth number using the first synthetic key value” will be described in detail below.

[0384] In the embodiments described in steps 2710 to 2720, the image to be diffused is divided into a fourth number of image blocks. The first first number of denoising sub-steps in the fourth number of denoising sub-steps are used as the first number of diffusion processing units to process the denoising sub-steps of the first image block to be diffused. Through this step, each diffusion processing unit only needs to store the parameters of the part of the model task it is responsible for processing when processing the first image block to be diffused, which greatly reduces the storage space occupied compared to the scheme of storing all parameters of the diffusion model. Furthermore, in the fifth number of denoising sub-steps after processing the first image block to be diffused, the diffusion processing unit uses the first synthesized key value to process the second to the last image block to be diffused from the fourth number of denoising sub-steps. By using the first synthesized key value to process the image blocks to be diffused, compared with using the key value generated by the previous denoising step for diffusion processing, better convergence effect can be achieved and diffusion processing accuracy can be improved.

[0385] In one embodiment, reference is made to Figure 29 Step 2620 includes:

[0386] Step 2910: Obtain diffusion target description information;

[0387] Step 2920: Generate a diffusion guidance vector based on the diffusion target description information;

[0388] Step 2930: Under the guidance of the diffusion guiding vector, process the second image block to be diffused up to the last image block to be diffused using the first synthetic key value.

[0389] Steps 2910 to 2930 are described in detail below:

[0390] In step 2910, the diffusion target description information is obtained.

[0391] According to the embodiments of this disclosure, the "diffusion target description information" has been explained in step 1510, and will not be repeated here.

[0392] In step 2920, a diffusion guidance vector is generated based on the diffusion target description information.

[0393] According to embodiments of this disclosure, there are multiple ways to generate a diffusion guidance vector based on diffusion target description information. In one implementation, the diffusion target description information is input into an embedding layer, and the embedding layer outputs the diffusion guidance vector. In another implementation, the diffusion target description information is segmented into words, and word vectors corresponding to each word are obtained by searching for word vectors. The searched word vectors are then concatenated in the order of the segmented words to obtain the diffusion guidance vector. By converting the diffusion target description information into vector form, it can be quickly recognized and processed by the diffusion processing model, adapting to the diffusion processing model quickly with relatively low cost, thus improving the diffusion processing efficiency of the diffusion processing model.

[0394] According to another embodiment of this disclosure, in molecular graph modeling applications, various properties of molecules and their chemical reactions with other substances can be obtained, thereby generating diffusion steering vectors. In time series modeling applications, the constraints of time series modeling can be obtained, thereby generating diffusion steering vectors. These are just a few examples; further details are omitted here.

[0395] In step 2930, guided by the diffusion guiding vector, the second image block to be diffused and the last image block to be diffused are processed using the first synthetic key value respectively.

[0396] According to embodiments of this disclosure, after determining the diffusion guidance vector, the diffusion guidance vector can guide each diffusion processing unit to sequentially process multiple image blocks to be diffused. For example, refer to... Figure 28B First, a diffusion guidance vector is generated based on the diffusion target description information. Guided by the diffusion guidance vector, in denoising sub-step 1-1, after processing image block 1, diffusion processing unit 1 can process image blocks 2, 3, and 4 respectively using the first synthetic key value in the three denoising sub-steps (i.e., denoising sub-steps 1-2, 1-3, and 1-4). Similarly, guided by the diffusion guidance vector, in denoising sub-step 1-2, after processing image block 1, diffusion processing unit 2 can process image blocks 2, 3, and 4 respectively using the first synthetic key value in the three denoising sub-steps (i.e., denoising sub-steps 1-3, 1-4, and 2-1). Guided by the diffusion guiding vector, in the denoising sub-steps 1-3, the diffusion processing unit 3 can process the image blocks 2, 3 and 4 to be diffused respectively using the first synthetic key value in the three denoising sub-steps (i.e., denoising sub-steps 1-4, denoising sub-steps 2-1 and denoising sub-steps 2-2) after processing the image block 1 to be diffused.

[0397] The embodiments of steps 2910 to 2930 above convert the diffusion target description information into vector form, which can be quickly identified and processed by the diffusion processing model. This allows for rapid adaptation to the diffusion processing model at a relatively low cost, thereby improving the diffusion processing efficiency of the diffusion processing model.

[0398] Detailed implementation diagrams of the diffusion processing method according to embodiments of this disclosure.

[0399] The following reference Figure 30 The following provides a detailed and exemplary description of the implementation details of the diffusion processing method according to embodiments of the present disclosure.

[0400] As mentioned above, in a processor architecture with CPU and GPU, the CPU is responsible for instruction scheduling, and each GPU can be the smallest hardware unit that executes the model task, i.e., the processing unit. Figure 30 The scheduler can refer to the CPU. The preheating processing unit 1 and the diffusion processing unit 2 can be GPUs.

[0401] In step 3010, the image to be diffused and the description information of the diffusion target are received.

[0402] In step 3021, a diffusion guidance vector is generated based on the diffusion target description information.

[0403] In step 3022, in the preheating step before multiple denoising steps, the image to be diffused is divided into a second number of preheating blocks and assigned to a second number of preheating processing units for processing.

[0404] In one embodiment, the number of preheating steps is determined by: obtaining the model size of the diffusion processing model; obtaining the diffusion processing complexity; and determining the number of preheating steps based on the model size and the diffusion processing complexity.

[0405] In one embodiment, obtaining the diffusion processing complexity includes: obtaining diffusion target description information; obtaining the number of image elements after diffusion and the relationship information between the image elements after diffusion based on the diffusion target description information; determining a first sub-complexity based on the number of image elements after diffusion; determining a second sub-complexity based on the relationship information between the image elements after diffusion; and determining the diffusion processing complexity based on the first sub-complexity and the second sub-complexity.

[0406] In one embodiment, determining the number of preheating steps based on model size and diffusion processing complexity includes: determining a first score based on model size; determining a second score based on diffusion processing complexity; and determining the number of preheating steps based on the first score and the second score.

[0407] In one embodiment, the second number is determined by: obtaining the size of the image to be diffused; obtaining the service type used for diffusion processing; obtaining the target diffusion processing efficiency; and determining the second number based on the size of the image to be diffused, the service type, and the target diffusion processing efficiency.

[0408] In step 3023, the diffusion processing model is divided into a first number of model tasks and assigned to a first number of diffusion processing units for execution. When each diffusion processing unit executes its assigned model task in each denoising step, it needs to use the first key value generated during the execution of the diffusion processing model in the previous denoising step.

[0409] In one embodiment, the first number of model tasks are ordered and are distributed to multiple warm-up steps in the following manner: obtaining the task size of each model task; merging a portion of sequentially adjacent model tasks using multiple first candidate merging schemes, such that the number of generated merged task blocks is equal to the number of warm-up steps; calculating the total task size of each merged task block in each first candidate merging scheme based on the task size, and calculating the first variance of the total task size of each merged task block; determining the first candidate merging scheme with the smallest first variance as the target merging scheme, and merging a portion of sequentially adjacent model tasks into one warm-up step based on the target merging scheme.

[0410] In step 3031, each preheating unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task.

[0411] In one embodiment, the preheating step is a plurality of preheating steps, wherein a first number of model tasks are assigned to the plurality of preheating steps, such that the number of model tasks assigned to each preheating step is a third number, and the third number in different preheating steps is not exactly the same; the first number of model tasks are executed sequentially in the preheating steps, including: in each preheating step, executing the third number of model tasks corresponding to the preheating step, thereby executing the first number of model tasks in the plurality of preheating steps.

[0412] In step 3032, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

[0413] In one embodiment, the preheating step consists of multiple preheating steps; passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task includes: in the last preheating step of the multiple preheating steps, passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task in sequence according to the order of the model tasks.

[0414] In one embodiment, each model task begins execution in a different denoising sub-step of the first denoising step; passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task includes: obtaining a first size of the first subkey value; predicting a first transmission time of the first subkey value based on the first size; obtaining the execution start time of the denoising sub-step corresponding to the first subkey value; determining the transmission start time based on the execution start time and the first transmission time; and passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task at the transmission start time.

[0415] In step 3041, the first key value used in the first denoising step is synthesized using the second number of first sub-key values ​​generated in the second number of preheating processing units in the preheating step.

[0416] In one embodiment, the first sub-key value is represented as a first sub-matrix corresponding to the preheating block, and the first key value is represented as a first matrix corresponding to the image to be diffused; the first key value used in the first denoising step is synthesized using the second number of first sub-key values ​​generated in the second number of preheating processing units, including: integrating the first sub-matrices generated by the second number of preheating processing units respectively, corresponding to each preheating block, into a first matrix according to the order of the preheating blocks.

[0417] In step 3042, the diffusion processing model is executed in the first denoising step.

[0418] In step 3050, in the subsequent denoising steps after the first denoising step, the diffusion processing model is executed by each diffusion processing unit using the first key value generated in the previous denoising step.

[0419] In one embodiment, each model task begins execution in a different denoising sub-step of other denoising steps; in other denoising steps after the first denoising step, the diffusion processing model is executed by each diffusion processing unit using the first key value generated in the previous denoising step of other denoising steps, including: in other denoising steps after the first denoising step, for a single diffusion processing unit, obtaining the first current subkey value generated in other denoising steps by the first model task corresponding to the single diffusion processing unit, and the first preceding subkey value generated in other denoising steps by the preceding model task before the first model task; for a single diffusion processing unit, extracting the first subsequent subkey value corresponding to the subsequent model task after the first model task from the first key value generated in the previous denoising step of other denoising steps; synthesizing the first preceding subkey value, the first current subkey value, and the first subsequent subkey value into a first synthesized key value, and executing each model task by each diffusion processing unit using the first synthesized key value.

[0420] In one embodiment, each diffusion processing unit performs each model task using a first synthetic key value, including: dividing the image to be diffused into a fourth number of image blocks to be diffused, the fourth number being equal to the number of denoising sub-steps in the denoising step, and the fourth number being greater than or equal to a first number; using the first first number of denoising sub-steps in the fourth number of denoising sub-steps as the first number of diffusion processing units to process the denoising sub-steps of the first image block to be diffused, wherein, after processing the first image block to be diffused, the diffusion processing unit processes the second image block to the last image block to be diffused from the fourth number of the fifth number of denoising sub-steps using the first synthetic key value to perform each model task, wherein the fifth number is the fourth number minus 1.

[0421] Description of apparatus and devices according to embodiments of this disclosure

[0422] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0423] It should be noted that in various specific embodiments of this application, when processing data related to object characteristics, such as object attribute information or sets of attribute information, is required, the object's permission or consent will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining object attribute information, separate permission or consent from the object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the object's separate permission or consent will the necessary object-related data for the proper functioning of these embodiments be acquired.

[0424] Figure 31 This is a schematic diagram of the diffusion processing apparatus provided in an embodiment of the present disclosure. The method applied by the diffusion processing apparatus 3100 is executed in multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into a first number of model tasks, which are assigned to a first number of diffusion processing units for execution. When each diffusion processing unit executes its assigned model task in each denoising step, it needs to use the first key value generated during the execution of the diffusion processing model in the previous denoising step. The diffusion processing apparatus 3100 includes:

[0425] The allocation unit 3110 is used in the preheating step before multiple denoising steps to divide the image to be diffused into a second number of preheating blocks and allocate them to a second number of preheating processing units for processing. Each preheating processing unit sequentially executes a first number of model tasks in the preheating step to obtain the first subkey value corresponding to each model task.

[0426] The transmission unit 3120 is used to transmit the first sub-key value corresponding to the model task to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can use the second number of first sub-key values ​​generated by the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step, so that the diffusion processing model can be executed in the first denoising step.

[0427] The execution unit 3130 is used to execute the diffusion processing model in the other denoising steps after the first denoising step, through each diffusion processing unit, using the first key value generated in the previous denoising step of the other denoising steps.

[0428] Optionally, there are multiple warm-up steps, wherein a first number of model tasks are assigned to multiple warm-up steps, such that the number of model tasks assigned to each warm-up step is a third number, and the third number in different warm-up steps is not exactly the same.

[0429] The allocation unit 3110 is specifically used for:

[0430] In each warm-up step, the third number of model tasks corresponding to the warm-up step are executed, so that the first number of model tasks are executed in multiple warm-up steps.

[0431] Optionally, the first number of model tasks are ordered, and the allocation unit 3110 is specifically used for:

[0432] Get the task size for each model task;

[0433] By using multiple first-candidate merging schemes, a portion of sequentially adjacent model tasks are merged, so that the number of generated merged task blocks is equal to the number of warm-up steps.

[0434] Based on the task size, calculate the total task size of each merged task block in each first candidate merge scheme, and calculate the first variance of the total task size of each merged task block;

[0435] The first candidate merging scheme with the smallest variance is determined as the target merging scheme, and based on the target merging scheme, a portion of sequentially adjacent model tasks are merged into a warm-up step.

[0436] Optionally, the allocation unit 3110 is specifically used for:

[0437] Obtain the model size of the diffusion processing model;

[0438] Obtain the complexity of the diffusion process;

[0439] The number of preheating steps is determined based on the model size and diffusion processing complexity.

[0440] Optionally, the allocation unit 3110 is specifically used for:

[0441] Obtain description information of the diffusion target;

[0442] Based on the diffusion target description information, obtain the number of elements in the diffused image and the relationship information between the elements in the diffused image;

[0443] The first sub-complexity is determined based on the number of elements in the diffused image;

[0444] The second sub-complexity is determined based on the relationship information of the elements in the diffused image;

[0445] The diffusion processing complexity is determined based on the first and second sub-complexities.

[0446] Optionally, the allocation unit 3110 is specifically used for:

[0447] Determine the first score based on the model size;

[0448] The second score is determined based on the complexity of the diffusion process.

[0449] The number of preheating steps is determined based on the first and second scores.

[0450] Optionally, the transfer unit 3120 is specifically used for:

[0451] Get the size of the image to be diffused;

[0452] Obtain the business type used for diffusion processing;

[0453] Obtain the target diffusion processing efficiency;

[0454] The second number is determined based on the size of the image to be diffused, the type of service, and the target diffusion processing efficiency.

[0455] Optionally, the first subkey value is represented as the first submatrix corresponding to the preheating block, and the first key value is represented as the first matrix corresponding to the image to be diffused;

[0456] The transfer unit 3120 is specifically used for:

[0457] The first sub-matrices generated by the second number of preheating units, each corresponding to a preheating block, are integrated into a first matrix according to the order of the preheating blocks.

[0458] Optionally, the preheating step can be multiple preheating steps;

[0459] The transfer unit 3120 is specifically used for:

[0460] In the last preheating step of multiple preheating steps, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task in the order of the model task.

[0461] Optionally, each model task begins execution in a different denoising substep of the first denoising step;

[0462] The transfer unit 3120 is specifically used for:

[0463] Get the first size of the first subkey value;

[0464] Based on the first size, predict the first transmission time of the first subkey value;

[0465] Get the start time of the denoising sub-step corresponding to the first subkey value;

[0466] The transmission start time is determined based on the execution start time and the first transmission time;

[0467] At the start of transmission, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

[0468] Optionally, each model task can begin execution in a different denoising substep of another denoising step;

[0469] Execution unit 3130 is specifically used for:

[0470] In the subsequent denoising steps after the first denoising step, for a single diffusion processing unit, the first current subkey value generated by the first model task corresponding to the single diffusion processing unit in the other denoising steps, and the first preceding subkey value generated by the preceding model task before the first model task in the other denoising steps are obtained.

[0471] For a single diffusion processing unit, the first subsequent sub-key value corresponding to the subsequent model task after the first model task is extracted from the first key value generated in the previous denoising step of other denoising steps.

[0472] The first preceding subkey value, the first current subkey value, and the first following subkey value are combined into a first synthesized key value, and each model task is executed using the first synthesized key value through each diffusion processing unit.

[0473] Optionally, the execution unit 3130 is specifically used for:

[0474] The image to be diffused is divided into a fourth number of image blocks, the fourth number being equal to the number of denoising sub-steps in the denoising step, and the fourth number being greater than or equal to the first number.

[0475] The first few denoising sub-steps in the fourth number of denoising sub-steps are respectively used as the first number of diffusion processing units to process the denoising sub-steps of the first image block to be diffused. In the fifth number of denoising sub-steps after processing the first image block to be diffused, the diffusion processing unit uses the first synthesis key value to process the second image block to the last image block to be diffused in the fourth number of denoising sub-steps to execute each model task. The fifth number is the fourth number minus 1.

[0476] Optionally, the execution unit 3130 is specifically used for:

[0477] Obtain description information of the diffusion target;

[0478] Based on the diffusion target description information, a diffusion guidance vector is generated;

[0479] Guided by the diffusion guiding vector, the second image block to be diffused and the last image block to be diffused are processed using the first synthetic key value respectively.

[0480] Reference Figure 32 , Figure 32 To implement the structural block diagram of the terminal portion of the diffusion processing method according to the embodiments of this disclosure, the terminal includes: a radio frequency (RF) circuit 3210, a memory 3215, an input unit 3230, a display unit 3240, a sensor 3250, an audio circuit 3260, a wireless fidelity (WiFi) module 3270, a processor 3280, and a power supply 3290, etc. Those skilled in the art will understand that... Figure 32 The terminal structure shown does not constitute a limitation on mobile phones or computers and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0481] The RF circuit 3210 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 3280; in addition, it transmits uplink data to the base station.

[0482] The memory 3215 can be used to store software programs and modules, and the processor 3280 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 3215.

[0483] The input unit 3230 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the content terminal. Specifically, the input unit 3230 may include a touch panel 3231 and other input devices 3232.

[0484] Display unit 3240 can be used to display input or provided information, as well as various menus of the content terminal. Display unit 3240 may include display panel 3241.

[0485] Audio circuitry 3260, speaker 3261, and microphone 3262 provide an audio interface.

[0486] In this embodiment, the processor 3280 included in the terminal can execute the diffusion processing method of the previous embodiment.

[0487] The terminals disclosed in this embodiment include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft. The embodiments of this invention can be applied to various scenarios, including but not limited to textural graph applications and molecular graph modeling.

[0488] Figure 33 This is a partial structural block diagram of a server for implementing the diffusion processing method of this disclosure. The server can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 3322 (e.g., one or more processors) and memory 3332, and one or more storage media 3330 (e.g., one or more mass storage devices) for storing application programs 3342 or data 3344. The memory 3332 and storage media 3330 may be temporary or persistent storage. The program stored in the storage media 3330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 3322 may be configured to communicate with the storage media 3330 and execute the series of instruction operations in the storage media 3330 on the server.

[0489] The server may also include one or more power supplies 3326, one or more wired or wireless network interfaces 3350, one or more input / output interfaces 3358, and / or one or more operating systems 3341, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0490] The central processing unit 3322 in the server can be used to execute the diffusion processing method of the present disclosure embodiments.

[0491] This disclosure also provides a computer-readable storage medium for storing program code for executing the diffusion processing methods of the foregoing embodiments.

[0492] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the diffusion processing method described above.

[0493] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar terms and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0494] It should be understood that in this disclosure, "at least one item" refers to one or more items, and "more than one item" refers to two or more items. "And / or" is used to describe the relationship between related content, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related content are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0495] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0496] In the embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. The couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0497] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0498] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0499] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server 130, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0500] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0501] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A diffusion treatment method, characterized in that, The diffusion processing method is executed in multiple denoising steps. In each denoising step, a diffusion processing model is executed. This model is divided into a first number of model tasks, which are assigned to a first number of diffusion processing units for execution. When the diffusion processing unit executes the assigned model task in each denoising step, it needs to use the first key value generated during the execution of the diffusion processing model in the previous denoising step. The diffusion processing method includes: In the preheating step before the multiple denoising steps, the image to be diffused is divided into a second number of preheating blocks and assigned to a second number of preheating processing units for processing. Each of the preheating processing units sequentially executes the first number of model tasks in the preheating step to obtain the first subkey value corresponding to each of the model tasks. The first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can use the second number of first subkey values ​​generated by the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step, so that the first denoising step can execute the diffusion processing model. In the subsequent denoising steps following the first denoising step, the diffusion processing model is executed by each of the diffusion processing units using the first key value generated in the previous denoising step.

2. The diffusion treatment method according to claim 1, characterized in that, The preheating step is a plurality of preheating steps, wherein the first number of model tasks are assigned to the plurality of preheating steps, such that the number of model tasks assigned to each preheating step is a third number, and the third number in different preheating steps is not exactly the same. The sequential execution of the first number of model tasks in the preheating step includes: In each of the preheating steps, a third number of model tasks corresponding to the preheating step are executed, thereby completing the first number of model tasks in the plurality of preheating steps.

3. The diffusion treatment method according to claim 2, characterized in that, The first number of model tasks are ordered, and the first number of model tasks are distributed among the multiple warm-up steps in the following manner: Obtain the task size for each of the model tasks; By using multiple first candidate merging schemes, a portion of the sequentially adjacent model tasks are merged, such that the number of generated merged task blocks is equal to the number of preheating steps. Based on the task size, calculate the total task size of each merged task block in each of the first candidate merge schemes, and calculate the first variance of the total task size of each of the merged task blocks; The first candidate merging scheme with the smallest variance is determined as the target merging scheme, and based on the target merging scheme, a portion of the sequentially adjacent model tasks are merged into one warm-up step.

4. The diffusion treatment method according to claim 2, characterized in that, The number of preheating steps is determined in the following way: Obtain the model size of the diffusion processing model; Obtain the complexity of the diffusion process; The number of preheating steps is determined based on the model size and the diffusion processing complexity.

5. The diffusion treatment method according to claim 4, characterized in that, The process of obtaining the diffusion processing complexity includes: Obtain description information of the diffusion target; Based on the diffusion target description information, the number of image elements after diffusion and the relationship information between image elements after diffusion are obtained. The first sub-complexity is determined based on the number of elements in the diffused image; Based on the element relationship information of the diffused image, the second sub-complexity is determined; The diffusion processing complexity is determined based on the first sub-complexity and the second sub-complexity.

6. The diffusion treatment method according to claim 4, characterized in that, Determining the number of preheating steps based on the model size and the diffusion processing complexity includes: Based on the model size, determine the first score; The second score is determined based on the aforementioned diffusion processing complexity; The number of preheating steps is determined based on the first score and the second score.

7. The diffusion treatment method according to claim 1, characterized in that, The second number is determined in the following way: Get the size of the image to be diffused; Obtain the service type used for the diffusion process; Obtain the target diffusion processing efficiency; The second number is determined based on the size of the image to be diffused, the service type, and the target diffusion processing efficiency.

8. The diffusion treatment method according to claim 1, characterized in that, The first subkey value is represented as a first submatrix corresponding to the preheating block, and the first key value is represented as a first matrix corresponding to the image to be diffused; The process of synthesizing the first key value used in the first denoising step using the second number of first sub-key values ​​generated by the second number of preheating processing units in the preheating step includes: The first sub-matrices generated by the second number of preheating units, each corresponding to a preheating block, are integrated into the first matrix according to the order of the preheating blocks.

9. The diffusion treatment method according to claim 1, characterized in that, The preheating step consists of multiple preheating steps; The step of passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task includes: In the last preheating step of the plurality of preheating steps, the first subkey value corresponding to the model task is sequentially passed to the diffusion processing unit corresponding to the model task according to the order of the model tasks.

10. The diffusion treatment method according to claim 1, characterized in that, Each of the aforementioned model tasks begins execution in a different denoising sub-step of the first denoising step; The step of passing the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task includes: Get the first size of the first subkey value; Based on the first size, predict the first transmission time of the first subkey value; Obtain the start time of the denoising sub-step corresponding to the first subkey value; The transmission start time is determined based on the execution start time and the first transmission time; At the start time of the transmission, the first subkey value corresponding to the model task is passed to the diffusion processing unit corresponding to the model task.

11. The diffusion treatment method according to claim 1, characterized in that, Each of the aforementioned model tasks begins execution in a different denoising sub-step of the other denoising steps; The subsequent denoising steps after the first denoising step, through each of the diffusion processing units, utilize the first key value generated in the preceding denoising step to execute the diffusion processing model, including: In the other denoising steps after the first denoising step, for a single diffusion processing unit, the first current subkey value generated by the first model task corresponding to the single diffusion processing unit in the other denoising steps, and the first preceding subkey value generated by the preceding model task before the first model task in the other denoising steps are obtained. For the single diffusion processing unit, the first subsequent sub-key value corresponding to the subsequent model task after the first model task is extracted from the first key value generated in the previous denoising step of the other denoising steps. The first preceding subkey value, the first current subkey value, and the first following subkey value are combined into a first synthesized key value, and each of the diffusion processing units uses the first synthesized key value to execute each of the model tasks.

12. The diffusion treatment method according to claim 11, characterized in that, The step of performing each of the model tasks through each of the diffusion processing units using the first synthetic key value includes: The image to be diffused is divided into a fourth number of image blocks, the fourth number being equal to the number of denoising sub-steps in the denoising step, and the fourth number being greater than or equal to the first number. The first few denoising sub-steps in the fourth number of denoising sub-steps are respectively used as the first number of diffusion processing units to process the first image block to be diffused. The diffusion processing unit processes the second image block to the last image block to be diffused from the first number of denoising sub-steps after processing the first image block to be diffused in the fifth number of denoising sub-steps, respectively, using the first synthesis key value to process the second image block to the last image block to be diffused in the fourth number of denoising sub-steps, so as to execute each of the model tasks. The fifth number is the fourth number minus 1.

13. The diffusion treatment method according to claim 12, characterized in that, The process of processing the second image block to be diffused to the last image block of the fourth number using the first synthetic key value includes: Obtain description information of the diffusion target; Based on the diffusion target description information, a diffusion guidance vector is generated; Guided by the diffusion guiding vector, the second image block to be diffused and the last image block to be diffused of the fourth number are processed using the first synthetic key value.

14. A diffusion treatment apparatus, characterized in that, The diffusion processing device employs a method executed in multiple denoising steps. In each denoising step, a diffusion processing model is executed. This model is divided into a first number of model tasks, which are assigned to a first number of diffusion processing units for execution. In each denoising step, when a diffusion processing unit executes its assigned model task, it needs to use a first key value generated during the execution of the diffusion processing model in the previous denoising step. The diffusion processing device includes: The allocation unit is used to divide the image to be diffused into a second number of preheating blocks in the preheating step before the multiple denoising steps, and allocate them to a second number of preheating processing units for processing. Each of the preheating processing units sequentially executes the first number of model tasks in the preheating step to obtain a first subkey value corresponding to each of the model tasks. The transmission unit is used to transmit the first subkey value corresponding to the model task to the diffusion processing unit corresponding to the model task, so that the diffusion processing unit can use the second number of first subkey values ​​generated by the second number of preheating processing units in the preheating step to synthesize the first key value used in the first denoising step, so that the first denoising step can execute the diffusion processing model. An execution unit is used to execute the diffusion processing model in each of the diffusion processing units after the first denoising step, using the first key value generated in the previous denoising step.

15. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the diffusion processing method according to any one of claims 1 to 13.

16. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the diffusion processing method according to any one of claims 1 to 13.

17. A computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the diffusion processing method according to any one of claims 1 to 13.

Images