Diffusion processing method, related device and medium

Abstract

The invention provides a diffusion processing method, a related device and a medium, and the method comprises the steps: dividing a to-be-diffused image into a first number of image block groups, and distributing the image block groups to a first number of diffusion processing unit groups for processing; dividing each de-noising step into a third number of de-noising time slots, and executing a model task corresponding to the de-noising time slot on the image block in each de-noising time slot; when the model task is executed on the image blocks in the denoising time slot, obtaining a first sub-key value when the model task is executed on the same-order image blocks with the same serial number as the image blocks in other image block groups, and merging the first sub-key value with a second sub-key value when the model task is executed on the image blocks to obtain a first key value; the method is used for executing model tasks of image partitioning. According to the invention, the diffusion processing process can better adapt to underlying network hardware requirements, and the diffusion processing flexibility and accuracy are improved. The method and the device can be applied to scenes such as image generation and text graph generation.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and in particular to a diffusion processing method, related apparatus and medium. Background Technology

[0002] A diffusion model is a generative model that generates high-resolution images by simulating the diffusion process of matter in a medium. For example, in text-based image processing, a diffusion guide vector is generated based on the text description in the text-based image. The image to be diffused is input into the diffusion model, and under the guidance of the diffusion guide vector, progressive denoising is performed. That is, the diffusion process is divided into several diffusion steps, and each diffusion step removes a portion of the noise from the image to be diffused. In this way, after all the diffusion steps, the target image that matches the text description is obtained.

[0003] To improve the execution efficiency of diffusion models, existing technologies employ parallel strategies. However, once these parallel strategies are set, they cannot be adjusted, cannot adapt to the underlying network hardware, have poor flexibility, and only utilize the intermediate results generated by the current process, without utilizing the intermediate results generated by other processes. However, the overall execution of the diffusion model requires the use of intermediate results generated by other processes, and the inflexible overall communication mechanism leads to poor accuracy in diffusion processing. Summary of the Invention

[0004] This disclosure provides a diffusion processing method, related apparatus, and medium that can better adapt to the requirements of underlying network hardware and improve the flexibility and accuracy of diffusion processing.

[0005] According to one aspect of this disclosure, a diffusion processing method is provided, comprising:

[0006] The image to be diffused is divided into a first number of image block groups and distributed in parallel to a first number of diffusion processing unit groups for processing. Each of the diffusion processing unit groups includes a second number of diffusion processing units. The diffusion processing method includes multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into an ordered second number of model tasks, which are executed in an ordered manner by the second number of diffusion processing units.

[0007] Each denoising step is divided into a third number of denoising time slots, and the model task corresponding to the denoising time slot is executed on the image block in each denoising time slot, wherein the third number is the number of image blocks in the image block group, and the image blocks in the image block group are ordered;

[0008] When the model task is executed on the image block in the denoising time slot, the first sub-key value when executing the model task on the same-order image block with the same sequence number as the image block in other image block groups is obtained, and merged with the second sub-key value when executing the model task on the image block to obtain the first key value, which is used for the execution of the model task on the image block.

[0009] According to one aspect of this disclosure, a diffusion processing apparatus is provided, comprising:

[0010] The first slicing unit is used to divide the image to be diffused into a first number of image block groups and distribute them in parallel to the first number of diffusion processing unit groups for processing. Each of the diffusion processing unit groups includes a second number of diffusion processing units. The diffusion processing method includes multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into an ordered second number of model tasks, which are executed in an ordered manner by the second number of diffusion processing units.

[0011] A diffusion execution unit is configured to divide each denoising step into a third number of denoising time slots, and execute the model task corresponding to the denoising time slot on the image block in each denoising time slot, wherein the third number is the number of image blocks in the image block group, and the image blocks in the image block group are ordered;

[0012] The first acquisition unit is configured to, when performing the model task on the image block in the denoising time slot, acquire a first sub-key value when performing the model task on a sequential image block with the same index as the image block in other image block groups, and merge it with a second sub-key value when performing the model task on the image block to obtain a first key value, which is used for the execution of the model task on the image block.

[0013] Optionally, the diffusion execution unit is used for:

[0014] Based on the slot number of the denoising time slot, determine the sequence number of the image block to be started;

[0015] The first diffusion processing unit among the second number of diffusion processing units performs the first model task on the image block of the image block number;

[0016] In subsequent denoising time slots following the denoising time slot, other diffusion processing units divide the image into blocks according to the image block number and perform other model tasks.

[0017] Optionally, the diffusion execution unit is used for:

[0018] For each diffusion processing unit, while performing the corresponding model task on the image block acquired in the current denoising time slot, the image block after the diffusion processing unit performed the corresponding model task in the previous denoising time slot of the current denoising time slot is sent to the next diffusion processing unit in the corresponding diffusion processing unit group.

[0019] For each diffusion processing unit, while executing the corresponding model task on the image block obtained in the current denoising time slot, the image block that the diffusion processing unit needs to execute in the next denoising time slot is obtained from the previous diffusion processing unit in the corresponding diffusion processing unit group.

[0020] Optionally, in the first number of diffusion processing unit groups, the first number of diffusion processing units corresponding to the same model task constitute a parallel processing unit group. In a single denoising time slot, multiple diffusion processing units in a single parallel processing unit group execute the corresponding model task on the image blocks with the same index in each image block group. The diffusion execution unit is used for:

[0021] The first sub-block is determined based on the image blocks corresponding to the diffusion processing unit, and the first sub-block includes at least the second sub-key value corresponding to the diffusion processing unit;

[0022] The first sub-block of each of the diffusion processing units in a single parallel processing unit group is transformed to obtain the transformed sub-block.

[0023] Multi-head attention processing is performed on the transformed sub-blocks to obtain the first result sub-blocks, and the first result sub-blocks of each diffusion processing unit in a single parallel processing unit group are subjected to sub-block inverse transformation to obtain the multi-head attention result sub-blocks of each diffusion processing unit.

[0024] Based on the multi-head attention result sub-block, determine the image block of the next diffusion processing unit in the next denoising time slot in the corresponding diffusion processing unit group.

[0025] Optionally, the diffusion execution unit is used for:

[0026] The first sub-block corresponding to each of the diffusion processing units is divided along the first dimension into a first number of first blocks corresponding to a first number of diffusion processing units in the parallel processing unit group;

[0027] The first number of first blocks corresponding to a single diffusion processing unit are aggregated into the corresponding single diffusion processing unit, and the first number of first blocks corresponding to the single diffusion processing unit are spliced ​​along the second dimension to obtain the transformed sub-block.

[0028] Optionally, the diffusion execution unit is used for:

[0029] The query sub-block, the key block, and the value sub-block are respectively divided into a first number of query blocks, key blocks, and value blocks corresponding to a first number of diffusion processing units in the parallel processing unit group;

[0030] The first number of query blocks, key blocks, and value blocks corresponding to a single diffusion processing unit are aggregated into the single diffusion processing unit, and the first number of query blocks, the second number of key blocks, and the second number of value blocks are concatenated to obtain the transformed sub-block. The transformed sub-block includes at least a transformed query sub-block, a transformed key block, and a transformed value sub-block, wherein the transformed key block and the transformed value sub-block constitute the first key value.

[0031] Optionally, the diffusion execution unit is used for:

[0032] The transformed key sub-block and the transformed value sub-block are written into the first buffer of the diffusion processing unit;

[0033] In the first buffer, the transformed key block and the transformed value block are concatenated with the transformed key block and transformed value block obtained by the diffusion processing unit in multiple denoising time slots before the current denoising time slot to obtain the transformed key matrix and the transformed value matrix.

[0034] Optionally, the diffusion execution unit is used for:

[0035] Determine the buffer sub-region corresponding to the current denoising time slot;

[0036] If the transformed key block and transformed value block written in the previous denoising step exist in the cache sub-region corresponding to the current denoising time slot, the transformed key block and transformed value block in the cache sub-region corresponding to the current denoising time slot are updated using the transformed key block and transformed value block of the current denoising time slot.

[0037] Optionally, the diffusion execution unit is used for:

[0038] The transformed query sub-block, the transformed key matrix, and the transformed value matrix are respectively divided into transformed query blocks, transformed key blocks, and transformed value blocks corresponding to each attention head, and the transformed query blocks, the transformed key blocks, and the transformed value blocks are assigned to the corresponding attention heads;

[0039] In each of the attention heads, attention calculation is performed based on the corresponding transformed query block, transformed key block, and transformed value block.

[0040] Optionally, the diffusion execution unit is used for:

[0041] In each of the diffusion processing units, the first result sub-block is divided along the second dimension into a first number of result blocks corresponding to each diffusion processing unit in the parallel processing unit group;

[0042] The first number of results corresponding to a single diffusion processing unit are aggregated into the corresponding single diffusion processing unit, and the first number of results corresponding to the single diffusion processing unit are spliced ​​along the first dimension to obtain the multi-head attention result sub-block corresponding to the single diffusion processing unit.

[0043] Optionally, the diffusion execution unit is used for:

[0044] The second subkey value and query sub-block are determined based on the image block corresponding to the diffusion processing unit, and the second subkey value is written into the first buffer of the diffusion processing unit.

[0045] A single denoising time slot is divided into multiple ordered sub-denoising time slots. A first process is executed in each sub-denoising time slot. While executing the first process, the corresponding first sub-key value is obtained from other diffusion processing units in the parallel processing unit group and written into the first buffer. The first process includes: performing multi-head attention processing on the key sub-block and value sub-block corresponding to the sub-denoising time slot based on the query sub-block, to obtain the row normalization coefficient vector and attention intermediate result of each attention head corresponding to the sub-denoising time slot; wherein, the key sub-block and value sub-block corresponding to the first sub-denoising time slot are the second sub-key value, and the key sub-blocks and value sub-blocks corresponding to the other sub-denoising time slots are the first sub-key values ​​obtained in the previous sub-denoising time slot, or the first sub-key values ​​obtained in multiple denoising time slots before the current denoising time slot.

[0046] For each attention head, the row normalization coefficient vector and the intermediate attention result corresponding to each sub-denoising time slot of the attention head are accumulated respectively to obtain the corresponding target row normalization coefficient vector and the accumulated attention result. The accumulated attention result is then corrected based on the target row normalization coefficient vector to obtain the attention processing result corresponding to the attention head.

[0047] The attention processing results corresponding to each attention head are concatenated to obtain a multi-head attention result sub-block, and the image block of the next diffusion processing unit in the next denoising time slot is determined based on the multi-head attention result sub-block.

[0048] Optionally, the diffusion execution unit is used for:

[0049] The query sub-block, the key block and the value sub-block corresponding to the sub-denoising time slot are divided into query blocks, key blocks and value blocks corresponding to each attention head and assigned to the corresponding attention head;

[0050] For each attention head, a product matrix is ​​determined based on the corresponding query block and key block, and the product matrix is ​​indexed to obtain an indexed product matrix; the rows of the indexed product matrix are summed to obtain the row normalized coefficient vector corresponding to the attention head; the indexed product matrix and the corresponding value block are multiplied by matrix to obtain the attention intermediate result corresponding to the attention head.

[0051] Optionally, the diffusion processing apparatus further includes a number determination unit, which is used for:

[0052] The first number is determined based on the number of other diffusion processing units connected to a single diffusion processing unit via a first communication method;

[0053] The second number is determined based on the first number and the number of diffusion processing units;

[0054] A first number of diffusion processing units connected via a first communication method are respectively assigned to the first number of diffusion processing unit groups, and a second number of diffusion processing units in a single diffusion processing unit group are connected via a second communication method.

[0055] Optionally, the diffusion processing apparatus further includes a model task segmentation unit, which is used for:

[0056] The network layers of the diffusion processing model are divided into an input preprocessing layer, a noise prediction layer, and an output processing layer. The noise prediction layer includes a fourth number of cascaded diffusion converter modules.

[0057] Determine a first ratio between the fourth number and the second number, round the first ratio up to obtain a fifth number, and determine a first difference based on the product of the second number and the fifth number and the fourth number;

[0058] Obtain the total first parameter count of the input preprocessing layer and the total second parameter count of the output processing layer, and determine the second difference corresponding to the first model task and the third difference corresponding to the last model task based on the ratio of the total first parameter count to the total second parameter count and the first difference, wherein the sum of the second difference and the third difference is the first difference;

[0059] Based on the fifth number, the second difference, and the third difference, the number of the first layer and the number of the second layer are determined respectively;

[0060] The input preprocessing layer, the fourth number of diffusion converter modules, and the output processing layer are assigned to the second number of model tasks. The first model task consists of the input preprocessing layer and the first number of diffusion converter modules, the last model task consists of the second number of diffusion converter modules and the output processing layer, and each of the other model tasks consists of a fifth number of consecutive diffusion converter modules.

[0061] According to one aspect of this disclosure, an electronic device is provided, including a memory and a processor, the memory storing a computer program, the processor executing the computer program to implement the diffusion processing method as described above.

[0062] According to one aspect of this disclosure, a computer-readable storage medium is provided, the storage medium storing a computer program that, when executed by a processor, implements the diffusion processing method as described above.

[0063] According to one aspect of this disclosure, a computer program product is provided, the computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the diffusion processing method as described above.

[0064] In this embodiment, the image to be diffused is divided into a first number of image block groups, which are then distributed in parallel to a first number of diffusion processing unit groups for processing. Thus, different image block groups are processed in parallel, i.e., sequential parallelism. Furthermore, each diffusion processing unit group includes a second number of diffusion processing units. The diffusion processing model executed in each denoising step is divided into a second number of model tasks, which are executed sequentially by the second number of diffusion processing units, i.e., pipelined parallelism. This fully combines the advantages of sequential parallelism and pipelined parallelism, improving the degree of parallelism. This combination of sequential and pipelined parallelism better adapts to the requirements of the underlying network hardware. If the underlying network hardware has many processing devices but few cards per device, the second number can be increased while the first number can be decreased. Conversely, if the underlying network hardware has few processing devices but many cards per device, the first number can be increased while the second number can be decreased. This approach allows for flexible adaptation to various underlying network hardware requirements. This combination of sequential parallelism and pipelined parallelism, when performing model tasks on image blocks, requires not only the second sub-key value when performing the model task on the current image block in the current denoising time slot, but also the first sub-key value when other diffusion processing units perform model tasks on image blocks of equal status to the current image block (parallel image blocks with the same sequence number in their respective image block groups) in the current denoising time slot. Therefore, this embodiment obtains the first sub-key value of the parallel image blocks with the same sequence number as the current image block in other image block groups when performing model tasks, merges it with the second sub-key value when performing model tasks on the current image block, and obtains the first key value, which is used for the model task execution of the current image block. This improves the flexibility of diffusion processing while ensuring processing accuracy.

[0065] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0066] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0067] Figure 1 This is a system architecture diagram of the diffusion processing method applied according to embodiments of the present disclosure;

[0068] Figures 2A-2B This is an application scenario interface diagram of the embodiments of this disclosure applied in the context of text-based image processing.

[0069] Figure 3 This is a schematic diagram of the diffusion model based on the diffusion converter architecture;

[0070] Figure 4 This is a flowchart of a diffusion processing method according to an embodiment of the present disclosure;

[0071] Figure 5 This is a schematic diagram illustrating the segmentation of a diffusion processing model and its allocation to various diffusion processing units according to an embodiment of this disclosure;

[0072] Figure 6 This is a schematic diagram illustrating the sequential processing of each image block among the diffusion processing units in the corresponding diffusion processing unit group according to an embodiment of the present disclosure;

[0073] Figure 7 This is a flowchart illustrating the determination of a first number and a second number according to an embodiment of this disclosure;

[0074] Figure 8 This is a flowchart illustrating the segmentation of a diffusion processing model according to an embodiment of this disclosure;

[0075] Figure 9 This is a schematic diagram illustrating the segmentation of a diffusion processing model according to an embodiment of the present disclosure;

[0076] Figure 10 yes Figure 4 A flowchart of step 420;

[0077] Figure 11 yes Figure 4 A flowchart of step 420;

[0078] Figure 12 This is a schematic diagram of a parallel processing unit group in each diffusion processing unit group according to an embodiment of the present disclosure;

[0079] Figure 13 This is a flowchart of a model task according to an embodiment of the present disclosure;

[0080] Figure 14 yes Figure 13 A flowchart of step 1320;

[0081] Figure 15 This is a schematic diagram of a sub-block transformation process according to an embodiment of the present disclosure;

[0082] Figure 16 yes Figure 13 A flowchart of step 1320;

[0083] Figure 17This is a flowchart illustrating the construction of the transformed key matrix and transformed value matrix corresponding to all image blocks according to an embodiment of this disclosure;

[0084] Figure 18 yes Figure 17 A flowchart of step 1710;

[0085] Figure 19 yes Figure 13 A flowchart of step 1330;

[0086] Figure 20 yes Figure 13 A flowchart of step 1330;

[0087] Figure 21 This is a schematic diagram of the sub-block inverse transformation process according to an embodiment of the present disclosure;

[0088] Figure 22 This is a flowchart illustrating the execution of a model task through multiple sub-denoising time slots according to an embodiment of this disclosure;

[0089] Figure 23 This is a schematic diagram illustrating the execution of a model task through multiple sub-denoising time slots according to an embodiment of the present disclosure;

[0090] Figure 24 This is a flowchart of a first process according to an embodiment of the present disclosure;

[0091] Figure 25 This is a flowchart illustrating an application example of a diffusion processing method according to an embodiment of the present disclosure;

[0092] Figure 26 This is a block diagram of a diffusion processing apparatus according to an embodiment of the present disclosure;

[0093] Figure 27 It is performed according to the embodiments of this disclosure. Figure 4 The terminal structure diagram of the diffusion treatment method shown;

[0094] Figure 28 It is performed according to the embodiments of this disclosure. Figure 4 The server architecture diagram for the diffusion processing method is shown. Detailed Implementation

[0095] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0096] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0097] Diffusion models are generative models that generate high-resolution images by simulating the diffusion process of matter in a medium. Diffusion models are inspired by non-equilibrium thermodynamics. They define a Markov chain of diffusion steps to slowly add random noise to the data, then learn to reverse the diffusion process, constructing the desired data sample from the noise. Unlike VAEs or flow-based models, diffusion models learn with a fixed procedure, and the latent variables have high dimensionality (the same as the original data).

[0098] Forward diffusion: Noise is gradually added to the original image until it becomes completely noisy. The original image can be any image. Adding noise to the original image is done in noisy steps. In each noisy step, the image before and after adding noise is recorded for use in back diffusion.

[0099] Backdiffusion: Starting from a completely noisy state, noise is gradually removed to restore a clear image. This process is achieved through a series of denoising steps, which correspond to corresponding noise-adding steps. The noise-adding and denoising steps are collectively called diffusion steps. For example, in forward diffusion, the noise-adding steps are 1, 2, ..., T; in backdiffusion, the denoising steps are T, T-1, ..., 1. Here, denoising step T corresponds to noise-adding step T, denoising step T-1 corresponds to noise-adding step T-1, and so on. Because each noise-adding step in the forward diffusion process records the added noise and the image before noise addition, in backdiffusion, the noise predicted by the denoising step can be compared with the noise added in the corresponding noise-adding step to calculate the loss value. This allows the diffusion model to learn how to predict the corresponding noise based on a given label during backdiffusion, thus restoring the noisy image to the image corresponding to the given label.

[0100] Diffusion models are generative models that generate high-resolution images by simulating the diffusion process of matter in a medium. Diffusion models generate images by predicting noise at multiple time steps and subtracting the predicted noise from the initial image through a progressive denoising process. Because this process requires frequent and large-scale computations to predict noise corresponding to the image at each time step, the execution efficiency of diffusion processing is low.

[0101] To improve the execution efficiency of diffusion models, related technologies employ parallel strategies. However, once these parallel strategies are set, they cannot be adjusted, cannot adapt to the underlying network hardware, have poor flexibility, and only utilize the intermediate results generated by the current process, without utilizing the intermediate results generated by other processes. However, the overall execution of the diffusion model requires the use of intermediate results generated by other processes, and the inflexible overall communication mechanism leads to poor accuracy in diffusion processing.

[0102] System architecture and scenario description of the embodiments disclosed herein

[0103] Figure 1 This is a system architecture diagram of the diffusion processing method applied according to embodiments of the present disclosure. It includes: object terminal 110, Internet 120, gateway 130, and server 140.

[0104] The object terminal 110 is a device used by an object to issue an image generation request, instructing the diffusion processing model to generate a specific image and display the image finally generated by the diffusion processing. Furthermore, the actual diffusion processing of multiple diffusion steps in the diffusion model in this embodiment can also be performed by the object terminal 110. In this case, the object terminal 110 can execute the diffusion processing method of this embodiment independently. The object terminal 110 includes various forms such as desktop computers, laptops, PDAs (personal digital assistants), mobile phones, in-vehicle terminals, home theater terminals, and dedicated terminals. Additionally, it can be a single device or a collection of multiple devices. For example, multiple devices can be connected via a local area network, sharing a single display device to work collaboratively, collectively forming an object terminal. The object terminal 110 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0105] Gateway 130, also known as an internetwork connector or protocol converter, is a computer system or device that enables network interconnection at the transport layer and acts as a translator. It bridges the gap between two systems using different communication protocols, data formats, languages, or even completely different architectures. Gateway 130 also provides filtering and security functions. Messages sent from target terminal 110 to server 140 are forwarded to the corresponding server 140 via gateway 130. Messages sent from server 140 to target terminal 110 are also forwarded to the corresponding target terminal 110 via gateway 130.

[0106] Server 140 is a device that provides overall services to all networked object terminals 110. When the diffusion model is installed on server 140, server 140 is responsible for the actual diffusion processing of multiple diffusion steps in the diffusion model of this embodiment. Compared to object terminals 110, server 140 has higher requirements in terms of stability, security, and performance. Server 140 can be a single high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a single high-performance computer (e.g., a virtual machine), or a combination of portions of multiple high-performance computers (e.g., virtual machines). Server 140 can also communicate with the Internet 120 via wired or wireless means to exchange data.

[0107] The embodiments disclosed herein can be applied to various scenarios, such as the text-to-image scenario shown in Figure 2.

[0108] Due to work or other needs, an object may require the generation of corresponding images from text. In this case, the object enters the corresponding image description text in the interface of the text-to-image application, and the text-to-image application generates the corresponding image.

[0109] Reference Figure 2A The image description text entered by the object in the text generation application interface is "There are white clouds in the blue sky, a house on the ground, a tree on the side of the house, and people are enjoying the shade under the tree". After clicking OK, the text generation application receives the image description text and uses it as the input of the diffusion model to perform diffusion processing.

[0110] After obtaining the image description text, the diffusion model performs multi-step denoising. In each denoising step, it predicts the corresponding noise based on the image description text and progressively denoises the given initial image. After completing the denoising process at all time steps, it generates an image like... Figure 2B As shown, the target image corresponding to the image description text is displayed on the object's terminal interface.

[0111] It should be understood that the above description only illustrates some application scenarios of this disclosure. The business scenarios to which this disclosure can be applied may include, but are not limited to, the specific embodiments described above.

[0112] General Description of Embodiments in this Disclosure

[0113] According to one embodiment of this disclosure, a diffusion processing method is provided. Diffusion processing refers to the specific process in a diffusion model that comprehensively schedules various computing resources based on a target image description. This method can be executed by the target terminal 110, or by the server 140, or it can be partially executed by the target terminal 110 and partially by the server 140.

[0114] Before describing the diffusion processing method of the embodiments of this disclosure in detail, please refer to... Figure 3 First, we will explain the diffusion model based on the diffusion transformer (DiT) architecture.

[0115] Reference Figure 3 The diffusion model based on the diffusion transformer (DiT) architecture is a diffusion model with a diffusion transformer module at its core. It consists of a patchify module, an embedding module, multiple cascaded diffusion transformer blocks (DiT blocks), an adaptive normalization layer, and a linear transformation layer. Each diffusion transformer module has the same structure, as shown in the figure. Figure 3 Each diffusion transformer module includes a multi-head attention mechanism, a feedforward neural network, a context-conditioning input module, a multilayer perceptron, and multiple network layers for performing dimensionality scaling and offset operations. Residual connections are established at the inputs and outputs of the multi-head attention mechanism and the feedforward neural network. The context-conditioning module is connected to the multilayer perceptron, and the output of the multilayer perceptron is connected to each network layer performing the dimensionality scaling and offset operations. For multiple diffusion transformer modules cascaded sequentially, the output of each diffusion transformer module serves as the input to the next diffusion transformer module.

[0116] Next, based on the above diffusion model architecture, the execution process of the diffusion model will be explained:

[0117] In the first denoising step, the diffusion model generates a latent variable consisting entirely of random noise, which is a noisy image composed of random noise. This noisy image is used as the input image for the first denoising step.

[0118] In each denoising step, the diffusion model first performs a patching operation on the input image through a patching module, segmenting the input image into multiple fixed-size patch blocks. These patch blocks are then linearly mapped to obtain a matrix corresponding to the input image, which serves as the input matrix for the first diffusion transformer module. Simultaneously, the input label is obtained, which describes the final image to be generated. This label can be a text description of the image input via a terminal. Next, the embedding module converts the label and the step number of the denoising step into corresponding vector embeddings, which are then input into the multilayer perceptron of each diffusion transformer module. The multilayer perceptron calculates multiple dimensional scaling and offset values ​​based on the input vector embeddings. These dimensional scaling and offset values ​​are denoted as α1, β1, γ1, α2, β2, and γ2, respectively. These dimensional scaling and offset values ​​serve as parameters for the dimensional scaling and offset operations of each layer in the corresponding diffusion transformer module. Through these dimensional scaling and offset operations, the information from the input image description text is injected into the diffusion processing model, enabling the model to generate the target image corresponding to the image description text.

[0119] In each diffusion converter module, the input matrix is ​​first scaled and shifted based on the corresponding dimension scaling parameter α1 and offset parameter γ1. The scaled and shifted input matrix is ​​then fed into a multi-head attention mechanism for processing. The corresponding query matrix, key matrix, and value matrix are calculated based on the input matrix, and multi-head attention is performed on these matrices to obtain the result. This result is then scaled based on the corresponding dimension scaling parameter β1. Through residual connections, the scaled attention result is superimposed with the input matrix of the diffusion converter module to obtain a superimposed matrix. This superimposed matrix is ​​then normalized. Dimension scaling and shifting are performed on the normalized superimposed matrix based on the dimension scaling parameter α2 and offset parameter γ2. This is then input into a feedforward neural network for linear mapping to obtain a mapped matrix. Finally, the mapped matrix is ​​scaled based on the dimension scaling parameter β2, and through residual connections, the scaled mapped matrix and the superimposed matrix are superimposed to obtain the output matrix of the diffusion converter module.

[0120] The output matrix of the last diffusion converter module is decoded through an adaptive normalization layer and a linear transformation layer. The decoded result is then reshaped to obtain the noise predicted in the current denoising step and its corresponding covariance. Next, the noise predicted in the current denoising step is subtracted from the input image, and the subtracted noise is used as the input image for the next denoising step. This process continues until the last denoising step is completed, yielding the desired target image.

[0121] It should be noted that during the diffusion process, the step number of the denoising step is a value that decreases from the preset number of steps. For example, if the preset number of denoising steps is t, then the step number of the first denoising step is t. After the first denoising step is completed, the step number decreases to t-1, and so on. When the step number decreases to 0, it means that all denoising steps have been completed and the desired target image has been obtained.

[0122] Figure 4 A flowchart of a diffusion processing method according to an embodiment of the present disclosure is shown. Figure 4 As shown, the diffusion treatment method includes:

[0123] Step 410: Divide the image to be diffused into a first number of image block groups and distribute them in parallel to a first number of diffusion processing unit groups for processing. Each diffusion processing unit group includes a second number of diffusion processing units. The diffusion processing method includes multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into an ordered second number of model tasks, which are executed in an ordered manner by the second number of diffusion processing units.

[0124] Step 420: Divide each denoising step into a third number of denoising time slots, and perform the corresponding model task for each denoising time slot on the image block. Here, the third number is the number of image blocks in the image block group, and the image blocks in the image block group are ordered.

[0125] Step 430: When performing the model task on the image block in the denoising time slot, obtain the first sub-key value when performing the model task on the same-order image block with the same sequence number as the image block in other image block groups, and merge it with the second sub-key value when performing the model task on the image block to obtain the first key value, which is used for the execution of the model task on the image block.

[0126] In step 410, the image to be diffused is the initial image before the diffusion process. As described above, the training process of the diffusion model essentially involves learning how to predict the corresponding noise, then removing the predicted noise from the noisy image to generate the final image. Therefore, the image to be diffused can be a noisy image composed of random noise, such as an image composed of Gaussian noise.

[0127] The first number is the number of diffusion processing unit groups. Each diffusion processing unit group includes a second number of diffusion processing units. Each diffusion processing unit can be a computing unit used to deploy a diffusion processing model. For example, a diffusion processing unit can be a graphics processing unit (GPU). The second number of diffusion processing units in the same diffusion processing unit group are ordered, and multiple diffusion processing units in the same diffusion processing unit group can be deployed in different processing devices. The processing device can be a server node composed of multiple graphics processing units.

[0128] Image block groups are groups of image blocks, obtained by segmenting the image to be diffused. For example, the image to be diffused can be divided into 8 image blocks, and then each group of 4 image blocks can be grouped together, resulting in 2 image block groups. It can be understood that the image blocks in each image block group constitute a part of the image to be diffused, and each image block group is non-overlapping. The image blocks in a first number of image block groups are merged to form a complete image to be diffused. Specifically, the image to be diffused can be segmented based on a first number of sub-blocks, and then each sub-block can be segmented into a third number of image blocks, thus obtaining a first number of image block groups. During subsequent model task execution, the processing units in each diffusion processing unit group are used to execute the corresponding model task on the image blocks in the corresponding image block group.

[0129] Reference Figure 3During the execution of the model task, image blocks are first input into the patching module to be divided into multiple patch blocks. Each patch block is then mapped to a corresponding vector and concatenated to form the image matrix corresponding to the image block. The size of this image matrix determines the computational cost required to execute the model task on the image block, i.e., it determines the time required for the diffusion processing unit to execute the model task on the image block. If the time required for each diffusion processing unit in the diffusion processing unit group to execute the model task on each image block is inconsistent, a diffusion processing unit may not be able to obtain the next image block from the previous diffusion processing unit in a timely manner after completing the corresponding model task on one image block, resulting in wasted computational resources. The size of the image matrix depends on the number of patch blocks obtained from the image block segmentation. Since each patch block is the same size, the size of the image matrix depends on the size of the image block. Based on this, in this disclosure, the image to be diffused can be divided into a first number of sub-blocks, and then each sub-block can be divided into a third number of image blocks as an image block group. In this way, the size of each image block is consistent, and the size of the corresponding image matrix is ​​also consistent. The time required for the diffusion processing unit to perform the corresponding model task on each image block can be kept consistent. Thus, in each denoising time slot, each diffusion processing unit can complete the corresponding model task on the image block relatively synchronously, and obtain a new image block from the previous diffusion processing unit in the diffusion processing unit group, and perform the corresponding model task on the new image block in the next denoising time slot.

[0130] Reference Figure 5 The initial number is 2. The diffusion processing unit is divided into two groups: Pg_1 and Pg_2. The image to be diffused is divided into 8 image blocks, P_0 to P_7. P_0 to P_3 form the image block group corresponding to Pg_1, and P_4 to P_7 form the image block group corresponding to Pg_2. During the diffusion process, the processing units in Pg_1 execute the corresponding model tasks on P_0 to P_3, and the processing units in Pg_2 execute the corresponding model tasks on P_0 to P_3.

[0131] The diffusion processing model, as mentioned above, is a neural network model that progressively denoises a noisy image by predicting the corresponding noise at each denoising step and subtracting the predicted noise from the noisy image, thereby generating the corresponding image. For example, the diffusion processing model disclosed herein can be a model based on the diffusion converter architecture mentioned above. The model task refers to a part of the processing task performed by the diffusion processing model. The diffusion processing model can include multiple neural network layers, such as cascaded diffusion converter modules. The diffusion processing model predicts the corresponding noise by processing the input of the diffusion processing model sequentially through these neural network layers. These neural network layers constitute multiple processing modules, such as diffusion converter modules. These modules are executed serially, and the execution process of adjacent modules is relatively independent; only the output of the previous module needs to be passed to the next module as its input. For example, for multiple cascaded diffusion converter modules, the i-th diffusion converter module only needs to pass its output to the (i+1)-th diffusion converter module as its input. Based on this, the different neural network layers of the diffusion processing model can be segmented, thus dividing the diffusion processing model into a second number of model tasks. Each model task corresponds to a portion of the neural network layers in the diffusion processing model. It can be understood that the multiple neural network layers corresponding to one model task are consecutive. After segmenting the diffusion processing model into a second number of model tasks, these second number of model tasks are assigned to a second number of diffusion processing units within each diffusion processing unit group. The model tasks assigned to the second number of processing units in each diffusion processing unit group are different, and the model tasks corresponding to the second number of processing units in a single diffusion processing unit group constitute a complete diffusion processing model. Specifically, based on the second number, the neural network layers of the diffusion processing model can be evenly divided into a second number of parts and deployed to the second number of diffusion processing units in a single diffusion processing unit group, thereby assigning the second number of model tasks to the second number of diffusion processing units. For example, refer to... Figure 5 The second number is 4, the first number is 2, and there are a total of 8 processing units, namely D_0 to D_7. Among them, D_0, D_2, D_4, and D_6 belong to the diffusion processing unit group Pg_1, and D_1, D_3, D_5, and D_7 belong to the diffusion processing unit group Pg_2. The diffusion processing model is divided into 4 model tasks, namely M_0 to M_3. M_0 ​​is deployed to D_0 and D_1 at the same time, M_1 is deployed to D_2 and D_3 at the same time, and so on. Thus, the diffusion processing model is divided into 4 model tasks and deployed to the 4 diffusion processing units in each diffusion processing unit group.

[0132] Understandably, since the second number of model tasks correspond to different consecutive neural network layers in the diffusion processing model, and these neural network layers are executed sequentially, the second number of model tasks are ordered. Correspondingly, within each diffusion processing unit group, the diffusion processing units corresponding to different model tasks are also ordered. For example, in... Figure 5 In the diffusion processing model, M_0 to M_3 execute corresponding model tasks on the input sequentially. That is, the input to the diffusion processing model first passes through M_0 and executes the corresponding model task, then passes through M_1, M_2, and M_3 in sequence and executes the corresponding model tasks. Accordingly, the diffusion processing units D_0 and D_1, which are deployed with M_0, are the first diffusion processing units in diffusion processing unit groups Pg_1 and Pg_2, respectively, and so on, forming an ordered second number of diffusion processing units. After the image is divided into blocks and input into the corresponding diffusion processing unit groups, the second number of model tasks can be executed on the image blocks in an orderly manner through the ordered second number of diffusion processing units, thereby predicting the corresponding noise.

[0133] In step 420, the denoising step refers to a complete time step in which the diffusion processing model predicts the corresponding noise based on the input. It can be understood that during training, the diffusion processing model needs to perform forward diffusion and backward diffusion, i.e., adding and denoising. However, in practical applications, the diffusion processing model only needs to perform the backward diffusion denoising process. Therefore, each time step in the diffusion processing model's diffusion processing is a denoising step.

[0134] The third number is the number of image blocks in a single image block group, and this third number can be preset. In one embodiment, when the third number is less than the second number, that is, when the number of image blocks in a single image block group is less than the number of diffusion processing units in a single diffusion processing unit group, at least one diffusion processing unit in the diffusion processing unit group will lack an image block that can perform the model task. For example, if the second number is 4 and the third number is 3, that is, the diffusion processing unit group includes 4 diffusion processing units, while the corresponding image block group includes 3 image blocks, then at the same time, the corresponding model task can be performed on the 3 image blocks by the 3 diffusion processing units respectively, and at least one diffusion processing unit is idle, resulting in wasted computing resources and reduced efficiency of diffusion processing. Based on this, to ensure that in each denoising time slot, the second number of processing units in each diffusion processing unit group can simultaneously execute the corresponding model tasks on different image blocks in the corresponding image block group, the third number is not less than the second number. In this way, in the same denoising time slot, the second number of diffusion processing units in the diffusion processing unit group can simultaneously execute the corresponding model tasks on different image blocks, thereby avoiding the waste of computing resources caused by a certain diffusion processing unit being idle.

[0135] Specifically, the third number of image blocks in the image block group can be assigned a corresponding sequence number, and the third number of image blocks can be sequentially input into the first diffusion processing unit in the corresponding diffusion processing unit group according to the assigned sequence number. After the first diffusion processing unit finishes executing the model task corresponding to the previous image block, the next image block is input into the first diffusion processing unit in the corresponding diffusion processing unit group, thereby dividing the denoising step into the third number of denoising time slots.

[0136] For each image block, after the corresponding model task is executed in the first diffusion processing unit, it will be sequentially input into other diffusion processing units in the diffusion processing unit group to execute the corresponding model task. For example, since the third number of image blocks in the image block group are sequentially input into the first diffusion processing unit in the corresponding diffusion processing unit group and execute the corresponding model task, different image blocks pass through the other diffusion processing units in the diffusion processing unit group sequentially. Therefore, in each denoising time slot, each diffusion processing unit in the diffusion processing unit group will execute the corresponding model task for different image blocks.

[0137] The details of step 420 will be explained later and will not be repeated here.

[0138] In step 430, the second sub-key value is the key block and value sub-block determined by the diffusion processing unit based on the corresponding image block in the current denoising time slot; the first sub-key value refers to the key block and value sub-block determined based on the same-order image blocks with the same sequence number as the image block corresponding to the diffusion processing unit in other image block groups, where other image block groups refer to other image block groups that do not correspond to the diffusion processing unit group to which the diffusion processing unit belongs. It can be understood that the first sub-key value and the second sub-key value are relative concepts. The key block and value sub-block obtained based on an image block serve as both the second sub-key value of the corresponding diffusion processing unit and the first sub-key value of the diffusion processing unit in other diffusion processing unit groups.

[0139] It is understandable that the image blocks within each image block group are ordered, and different image block groups are independent of each other. Therefore, image blocks with the same sequence number may exist in different image block groups. For example, in... Figure 5In this model, P_0 to P_3 form the image block group corresponding to diffusion processing unit group Pg_1, and are sequentially assigned numbers 1 to 4. P_4 to P_7 form the image block group corresponding to diffusion processing unit group Pg_2, and are sequentially assigned numbers 1 to 4. Therefore, P_0 and P_4 correspond to the same number, and similarly, P_1 and P_5 also correspond to the same number, and so on. In step 420, the image blocks in each image block group are sequentially input into the first diffusion processing unit in the corresponding diffusion processing unit group to perform the first model task. Based on this, in a single denoising time slot, the image blocks obtained by the first diffusion processing unit of each diffusion processing unit group correspond to the same number. Furthermore, referring to the relevant description in step 410, since the second number of diffusion processing units in each diffusion processing unit group are ordered, after each image block passes through the first diffusion processing unit in the corresponding diffusion processing unit group and executes the corresponding model task, it will be sequentially transmitted to the diffusion processing unit corresponding to each model task in multiple subsequent denoising time slots and execute the corresponding model task in the diffusion processing unit. In a single denoising time slot, image blocks with the same sequence number in each image block group will be executed by the diffusion processing unit with the same model task in the corresponding diffusion processing unit group, and the corresponding model task will be executed by them respectively.

[0140] Reference Figure 3 The diffusion processing model includes multiple diffusion converter modules, each with a multi-head attention mechanism. This means that multi-head attention processing is required for each model task, and this process involves attention calculation based on the complete key and value matrices of the image to be diffused. Since the image to be diffused is divided into multiple image block groups, and each image block group is only input to its corresponding diffusion processing unit group to execute its respective model task, each diffusion processing unit in the diffusion processing unit group can only determine the corresponding second sub-key value based on the image blocks within its corresponding image block group. For image blocks in other image block groups, these blocks are only input to the diffusion processing units in the diffusion processing unit group corresponding to that image block group to determine the corresponding key and value sub-blocks. For example, referring to… Figure 6The diffusion processing units (D_0, D_2, D_4, D_6) in diffusion processing unit group Pg_1 can only obtain P_0 to P_3 and determine the corresponding first subkey values. P_4 to P_7 are not input to the diffusion processing units in diffusion processing unit group Pg_1. Therefore, the diffusion processing units in diffusion processing unit group Pg_1 cannot directly calculate the key sub-blocks and value sub-blocks corresponding to P_4 to P_7 locally. As can be seen from the above description, in each image block group, image blocks with the same index are subjected to the same model task in the processing units of the corresponding diffusion processing unit group in the same denoising time slot. Based on this, in each denoising time slot, the first sub-key value corresponding to the same-order image block with the same index as the image block currently being processed by the diffusion processing unit in other image block groups is obtained and saved to the buffer of the diffusion processing unit. Thus, since each image block corresponding to each index will pass through different diffusion processing units in different denoising time slots, the buffer of the diffusion processing unit stores the key sub-blocks and value sub-blocks corresponding to each image block of the image to be diffused. It can be understood that among multiple diffusion processing unit groups, each diffusion processing unit only needs to communicate with diffusion processing units in other diffusion processing unit groups that correspond to the same model task, and does not need to communicate with diffusion processing units in other diffusion processing unit groups that correspond to different model tasks. For example, refer to... Figure 6 The diffusion processing unit D_0 is equipped with a model task M_0. In the diffusion processing unit group Pg_2, the diffusion processing unit D_1 is also equipped with a model task M_0. D_0 and D_1 are connected by communication. D_0 obtains the first sub-key value calculated by the diffusion processing unit D_1 in each denoising time slot, and thus obtains the first sub-key values ​​corresponding to P_4 to P_7, thereby obtaining the key sub-blocks and value sub-blocks corresponding to all image blocks.

[0141] Specifically, refer to Figure 6There are two groups of image blocks corresponding to the diffusion processing unit groups. Each group of image blocks includes four image blocks. The image block groups corresponding to the first diffusion processing unit group include P_0 to P_3, with corresponding image block numbers 1 to 4. The image block groups corresponding to the second diffusion processing unit group include P_4 to P_7, with corresponding numbers 1 to 4. For the first diffusion processing unit D_0 in the first diffusion processing unit group, in the first denoising time slot of the denoising step, diffusion processing unit D_0 calculates the corresponding second sub-key value based on image block P_0, and simultaneously obtains the second sub-key value from the diffusion processing unit. D_1 obtains the image block corresponding to index 1 in another image block group, i.e., image block P_4, and calculates the corresponding first sub-key value; in the second denoising time slot, the diffusion processing unit D_0 calculates the corresponding second sub-key value based on image block P_1, and obtains the first sub-key value corresponding to the image block corresponding to index 2 in another image block group, i.e., image block P_5; and so on. After each denoising time slot in the denoising step, the diffusion processing unit D_0 can obtain the key matrix and value matrix corresponding to the image blocks P_0 to P_7, thus obtaining the key matrix and value matrix corresponding to the entire image to be diffused.

[0142] Based on this, this embodiment divides the image to be diffused into image block groups corresponding to a first number of diffusion processing unit groups, and simultaneously divides the diffusion processing model into a second number of model tasks and assigns them to the second number of diffusion processing units in a single diffusion processing unit group. Then, in each denoising time slot, the diffusion processing unit in each diffusion processing unit group executes the corresponding model task on the image block in the corresponding image block group, thereby enabling parallel execution of model tasks on image blocks in different image block groups by different diffusion processing unit groups, forming sequential parallelism. Simultaneously, each diffusion processing unit in a single diffusion processing unit group can execute corresponding model tasks on different image blocks in the same denoising time slot, and a single image block will also be sequentially executed on by the second number of diffusion processing units in different denoising time slots, thus forming pipelined parallelism. Therefore, the first number of diffusion processing unit groups and the second number of diffusion processing units in a single diffusion processing unit group can be adjusted more flexibly according to the hardware conditions of the diffusion processing unit, to better adapt to the hardware of the diffusion processing unit.

[0143] It should be noted that, during the diffusion process, the input to each denoising step is actually the output of the previous denoising step, and the difference between the input and output of the previous denoising step is the predicted noise of that step. For example, if the input of the diffusion model in the i-th denoising step is z... i And the noise predicted by the diffusion processing model in the i-th denoising step is σ. iTherefore, the input for the (i+1)th denoising step is z. i -σ i Meanwhile, since the diffusion processing model generates images through multi-step denoising, the noise predicted in a single denoising step is relatively small. Therefore, the inputs of the diffusion processing model in the i-th and (i+1)-th denoising steps are actually quite similar. Correspondingly, the key and value matrices generated in the same diffusion converter module in the i-th and (i+1)-th denoising steps are also quite similar. Therefore, during multi-head attention processing, a portion of the key and value sub-blocks obtained in the previous denoising step can be reused, ensuring that the output of the diffusion processing model does not change significantly. Based on this, the diffusion processing in each diffusion processing unit group can construct a complete key and value matrix corresponding to the image to be diffused based on the first or second sub-key values ​​of a portion of the image blocks stored in the cache corresponding to the first or second sub-key values ​​in the previous denoising step, and the first and second sub-key values ​​of a portion of the image blocks obtained in each denoising time slot already executed in the current denoising step. This matrix is ​​then used for multi-head attention processing to execute the corresponding model task.

[0144] Furthermore, although the key and value matrices of each image block are relatively similar in adjacent denoising steps, to further improve the accuracy of the diffusion processing model's output, the model should utilize the key and value matrices of each image block in the current denoising step more extensively. Therefore, multi-head attention processing should be performed only after obtaining the first sub-key value corresponding to the same-order image block. This necessitates efficient communication between diffusion processing units corresponding to the same model task within different diffusion processing unit groups. This allows a diffusion processing unit to quickly obtain the corresponding first sub-key value from other diffusion processing units deployed with the same model task, enabling it to rapidly perform multi-head attention processing based on the obtained first sub-key value without waiting for it to obtain the first sub-key value from other diffusion processing units. Based on this, multiple diffusion processing units corresponding to the same model task in different diffusion processing unit groups can be deployed on the same processing device, allowing each diffusion processing unit to efficiently obtain the corresponding first sub-key value from other diffusion processing units with the same model task.

[0145] The details of step 430 will be explained later and will not be repeated here.

[0146] In the embodiments disclosed in steps 410 to 430, the image to be diffused is divided into a first number of image block groups and assigned to a first number of diffusion processing unit groups. Multiple image block groups can be processed in parallel in different diffusion processing unit groups. Simultaneously, the diffusion processing model is divided into a second number of model tasks and assigned to a second number of diffusion processing units within a diffusion processing unit group. Each second number of diffusion processing units in a single diffusion processing unit group deploys a complete diffusion processing model. After the image blocks in the image block groups are sequentially input into their corresponding diffusion processing unit groups, each image block is sequentially executed with its corresponding model task by the second number of diffusion processing units, thus forming pipelined parallelism within the diffusion processing unit groups. This combined sequential parallelism and pipelined parallelism processing method can be flexibly adjusted according to the configuration of the underlying network hardware. When there are many diffusion processing units that can be deployed on a single processing device for efficient communication, the first number can be increased and the second number decreased. Conversely, when there are few diffusion processing units that can be deployed on a single processing device for efficient communication, the first number can be decreased and the second number increased, thereby better adapting to the underlying network hardware. Meanwhile, by deploying diffusion processing units corresponding to the same model task in each diffusion processing unit group onto a single processing device, in each denoising time slot, the diffusion processing unit can efficiently obtain the first sub-key value corresponding to the same image block in other image block groups from other diffusion processing units that have the same model task deployed, and perform multi-head attention processing, thereby completing the corresponding model task and improving the accuracy of the output of the diffusion processing model.

[0147] The above provides an overall description of steps 410 to 430. The following provides a detailed description of the details of each step 410 to 430.

[0148] Detailed description of step 410

[0149] In step 410, the image to be diffused is divided into a first number of image block groups and distributed in parallel to a first number of diffusion processing unit groups for processing. Each diffusion processing unit group includes a second number of diffusion processing units. The diffusion processing method includes multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into an ordered second number of model tasks, which are executed in an ordered manner by the second number of diffusion processing units.

[0150] In one embodiment, a diffusion processing unit can connect to other processing units via a first communication method or a second communication method, where the first and second communication methods are implemented based on two different communication protocols supported by the diffusion processing unit. It is understood that these communication methods rely on certain communication hardware implementations, such as communication interfaces, switches, etc. Specifically, the bandwidth of the first communication method is greater than that of the second communication method; implementing the first communication method requires relatively high-performance hardware, which makes its implementation cost relatively high. Alternatively, the first communication method may have limitations in the number of connections and scalability, resulting in a single diffusion processing unit being able to connect to only a limited number of other diffusion processing units via the first communication method. Based on this, in one embodiment, referring to... Figure 7 The diffusion treatment method also includes:

[0151] Step 710: Determine the first number based on the number of other diffusion processing units connected to a single diffusion processing unit via the first communication method;

[0152] Step 720: Determine the second number based on the first number and the number of diffusion processing units;

[0153] Step 730: The first number of diffusion processing units connected by the first communication method are respectively assigned to the first number of diffusion processing unit groups, and the second number of diffusion processing units in a single diffusion processing unit group are connected by the second communication method.

[0154] In step 710, referring to the description in step 430, in each denoising time slot, the diffusion processing unit needs to obtain the first sub-key value corresponding to the same image block from other diffusion processing unit groups that have deployed the same model task. After deploying multiple diffusion processing units corresponding to the same model task in each diffusion processing unit group on the same processing device, a communication connection needs to be established between multiple diffusion processing units on a single processing device through a first communication method, so that multiple diffusion processing units corresponding to the same model task can communicate at high speed. Therefore, the first number can be determined based on the number of other diffusion processing units connected to a single diffusion processing unit through the first communication method. Specifically, the first number can be obtained by adding 1 to the number of other processing units connected to a single diffusion processing unit through the first communication method.

[0155] In one embodiment, the number of other processing units connected to a single diffusion processing unit via a first communication method can be determined based on the connection limit of the first communication method itself. For example, the diffusion processing unit is a graphics processing unit (GPU), and the first communication method can be a communication method in which graphics processing units are directly connected to other diffusion processing units through the NVlink interface of the diffusion processing unit itself. A single graphics processing unit can be connected to N-1 other graphics processing units through the NVlink interface. That is, the communication method of direct connection through the NVlink interface allows a maximum of N graphics processors to form a mesh topology and interconnect with each other. In this case, N is determined as the first number.

[0156] In another embodiment, the number of other processing units connected to a single diffusion processing unit via the first communication method can be determined based on the amount of hardware resources required to implement the first communication method. It is understood that, as described above, the first communication method itself relies on high-performance communication hardware. Due to limitations in hardware resources, a maximum of several groups of diffusion processing units interconnected via the first communication method can be deployed. In this case, all diffusion processing units need to be divided into several groups, and multiple diffusion processing units within the same group are interconnected via the first communication method. This allows the determination of the number of other diffusion processing units connected to a single diffusion processing unit via the first communication method, and thus, the determination of the first number. For example, the diffusion processing unit is a graphics processing unit (GPU). The first communication method can be NVlink communication implemented through a dedicated switch. The number of switches used for NVlink communication is N, and the number of diffusion processing units is M. In this case, the diffusion processing units are divided into N groups, and the number of diffusion processing units in each group is M / N. The M / N diffusion processing units in the same group use a dedicated switch for NVlink communication. The number of other diffusion processing units connected to a single diffusion processing unit through the first communication method is M / N-1, and the first number is M / N.

[0157] The above illustrates an embodiment where the number of other diffusion processing units connected to a single diffusion processing unit via the first communication method is determined based on limitations of hardware resources or the connection limit of the first communication method, and thus a first number is determined. In practical applications, the first number can also be determined by combining both hardware resources and the connection limit of the first communication method.

[0158] In step 720, it is understood that since the number of diffusion processing units is fixed, and the first number is the number of diffusion processing unit groups, after determining the first number, all diffusion processing units are evenly distributed into the first number of diffusion processing unit groups, thus determining the number of diffusion processing units in each diffusion processing unit group, i.e., the second number. For example, if the number of diffusion processing units is 8 and the first number is 2, then the corresponding second number is 8 / 2 = 4.

[0159] In step 730, the second communication method is a communication method with lower bandwidth, lower implementation cost, and fewer limitations in the number of connections and scalability compared to the first communication method. For example, the first communication method can be an NVlink-based communication method, and the second communication method can be a PCIe-based communication method or an Ethernet-based communication method.

[0160] After determining the first number, the first number of diffusion processing units interconnected through the first communication method are respectively assigned to different diffusion processing unit groups. Each diffusion processing unit group then obtains a second number of diffusion processing units. Each diffusion processing unit in a single diffusion processing unit group can be connected to a diffusion processing unit in other diffusion processing unit groups through the first communication method. Then, the same model task is deployed to the first number of diffusion processing units interconnected through the first communication method. In this way, when the model task is executed subsequently, the first sub-key value generated by the same image block in other image block groups when executing the model task can be quickly obtained from other diffusion processing units that have the same model task deployed through the first communication method, so as to obtain the key matrix and value matrix corresponding to all image blocks required to execute the corresponding model task.

[0161] Referring to the relevant description in step 410, the execution process of each model task is actually the calculation process of the corresponding multiple neural network modules. The calculation process of two adjacent diffusion converter modules is relatively independent. That is, the process of each diffusion processing unit in a single diffusion processing unit group executing the corresponding model task is relatively independent. In the process of each diffusion processing unit performing the model task on the image block of the current denoising time slot, it does not need to obtain data from other diffusion processing units in the same diffusion processing unit group. For the second number of diffusion processing units in the same diffusion processing unit group, each diffusion processing unit only needs to obtain the image block to be executed in the next denoising time slot from the previous diffusion processing unit in the diffusion processing unit group in each denoising time slot. The process of obtaining the image block to be executed in the next denoising time slot from the previous diffusion processing unit can be executed in parallel with the process of the diffusion processing unit executing the corresponding model task on the image block of the current denoising time slot. That is to say, within a single diffusion processing unit group, the data transmission between the diffusion processing units and the calculation process of the diffusion processing units are executed in parallel. Data transmission does not cause additional time consumption. This makes the communication between diffusion processing units in a single diffusion processing unit group have low requirements for communication rate. Therefore, for the second number of diffusion processing units in the same diffusion processing unit group, they can be connected through the second communication method.

[0162] Furthermore, for the ordered second number of diffusion processing units in a single diffusion processing unit group, it is only necessary to obtain the corresponding image blocks from the previous diffusion processing unit and send the image blocks after performing the corresponding model task to the next diffusion processing unit. Therefore, each diffusion processing unit in a single diffusion processing unit group only needs to communicate with the previous and next diffusion processing units through the second communication method, without needing to communicate with other diffusion processing units in the group. It should be noted that the last diffusion processing unit in the group needs to connect to the first diffusion processing unit to send the result of the previous denoising step to the first diffusion processing unit so that the first diffusion processing unit can begin executing the next denoising step. For example, in... Figure 6 In the above, for diffusion processing unit group Pg_1, the diffusion processing unit group includes diffusion processing units D_0, D_2, D_4 and D_6. For diffusion processing unit D_0, it only needs to be connected to diffusion processing units D_2 (the diffusion processing unit after D_0) and D_6 (equivalent to the diffusion processing unit before D_0) through the second communication method, and does not need to be connected to diffusion processing unit D_4.

[0163] In the embodiments disclosed in steps 710 to 730, a first number and a second number are determined based on the number of other diffusion processing units connected to a single diffusion processing unit via a first communication method. The first number of diffusion processing units connected via the first communication method are then allocated to a first number of diffusion processing unit groups, and a second number of diffusion processing units within a single diffusion processing unit group are connected using a second communication method. Thus, the same model task can be subsequently deployed to the first number of diffusion processing units connected via the first communication method, while different model tasks can be deployed to the second number of diffusion processing units within a single diffusion processing unit group. This allows for adjustment of the first and second numbers of diffusion processing units based on the underlying network hardware, while ensuring the accuracy of each diffusion processing unit in executing model tasks, thereby improving the flexibility of diffusion processing.

[0164] After determining the first number and the second number, the multiple diffusion processing units are divided into a first number of diffusion processing unit groups, and the communication method between the diffusion processing units is determined. Then, based on the first number, the image to be diffused is divided into a first number of image block groups and assigned to the first number of diffusion processing unit groups. Specifically, referring to the relevant description of step 410, the image to be diffused can be evenly divided into a first number of image sub-blocks, and then each image sub-block can be evenly divided into a third number of image blocks as an image block group, thereby obtaining the first number of image block groups and assigning them to the first number of diffusion processing unit groups.

[0165] In one embodiment, in order to enable each diffusion processing unit to complete the model task for an image block relatively synchronously within a single denoising time slot, the diffusion processing model should be relatively evenly divided into a second number of model tasks and then distributed to a second number of diffusion processing units, while referring to... Figure 3 Since a diffusion converter module includes multiple neural network layers, including residual connections and cross-layer connections that inject dimension scaling and offset parameters from a multilayer perceptron into the corresponding operational layers, distributing the multiple neural network layers of a single diffusion converter module into different diffusion processing units would increase the number of parameters that need to be passed between the diffusion processing units. Therefore, in one embodiment, referring to... Figure 8 The process of dividing the diffusion processing model into a second number of model tasks includes:

[0166] Step 810: Divide the network layer of the diffusion processing model into an input preprocessing layer, a noise prediction layer and an output processing layer, wherein the noise prediction layer includes a fourth number of cascaded diffusion converter modules.

[0167] Step 820: Determine the first ratio between the fourth number and the second number, round the first ratio up to obtain the fifth number, and determine the first difference based on the product of the second number and the fifth number and the fourth number;

[0168] Step 830: Obtain the total first parameter of the input preprocessing layer and the total second parameter of the output processing layer, and determine the second difference corresponding to the first model task and the third difference corresponding to the last model task based on the ratio of the total first parameter and the total second parameter and the first difference, wherein the sum of the second difference and the third difference is the first difference;

[0169] Step 840: Based on the fifth number, the second difference, and the third difference, determine the number of the first layer and the number of the second layer, respectively;

[0170] Step 850: Assign the input preprocessing layer, the fourth number of diffusion converter modules, and the output processing layer to the second number of model tasks. The first model task consists of the input preprocessing layer and the first number of diffusion converter modules, the last model task consists of the second number of diffusion converter modules and the output processing layer, and each of the other model tasks consists of a fifth number of consecutive diffusion converter modules.

[0171] In step 810, the input preprocessing layer refers to a module consisting of neural network layers that preprocess the image blocks after they have been input into the diffusion processing model and before noise prediction is performed by the noise prediction layer. For example, refer to... Figure 9 The input preprocessing layer may include a patching module for converting image blocks into corresponding input matrices, and an embedding layer for converting the step numbers and labels of the denoising steps into corresponding vector embeddings. The noise prediction layer is a neural network layer for predicting noise corresponding to the image blocks. The output processing layer refers to a neural network layer that further processes the output of the noise prediction layer to generate the noise for the current denoising step. For example, referring to... Figure 9 The output processing layer may include an adaptive normalization layer and a linear transformation layer. The noise prediction layer refers to multiple network layers in the diffusion processing model used to calculate the noise of image blocks. The noise prediction layer includes a fourth cascaded diffusion converter module, for example, 30 cascaded diffusion converter modules.

[0172] In step 820, the first ratio reflects the number of diffusion converter modules per model task when the fourth number of diffusion converter modules are evenly distributed among the second number of model tasks. This ratio may not be an integer, and when splitting the diffusion processing model, the input preprocessing layer and the output processing layer also need to be assigned to separate model tasks. Based on this, the first ratio is rounded up to obtain the fifth number, which is the maximum number of diffusion converter modules in a single model task when the fourth number of diffusion converter modules are relatively evenly distributed among the second number of model tasks. The product of the fifth number and the second number represents the theoretical number of diffusion converter modules required for the entire diffusion processing model when each model task is assigned the fifth number of diffusion converter modules; the first difference represents the difference between the theoretical and actual number of diffusion converter modules.

[0173] In step 830, the input preprocessing layer preprocesses the input image blocks and then inputs the preprocessed image blocks into the noise prediction layer; the output processing layer processes the output of the noise prediction layer, and the second number of model tasks are also ordered. Based on this, the input preprocessing layer and the output processing layer need to be allocated to the first and last model tasks, respectively. That is, the first model task includes the input preprocessing layer in addition to the diffusion converter module; the last model task includes the output processing layer in addition to the diffusion converter module. At this time, when the actual number of diffusion converter modules in the diffusion processing model is insufficient to allocate a fifth number of diffusion converter modules to each model task, fewer than a fifth number of diffusion converter modules can be allocated to the first and last model tasks. The input preprocessing layer and the output processing layer are used to supplement the first and last model tasks, ensuring that the computational load of the second number of model tasks remains relatively even, while the network layers in a single diffusion converter module are not split into different diffusion processing units. Based on this, the total number of first parameters of the input preprocessing layer and the total number of second parameters of the output processing layer are determined. These total parameters represent the computational load of the input preprocessing layer and the output processing layer, respectively. Based on the total number of first parameters and the total number of second parameters, the number of diffusion converter modules required by the input preprocessing layer and the output processing layer can be determined, thereby determining the difference between the number of diffusion converter modules allocated to the first model task and the last model task and the fifth number.

[0174] For example, if the first difference is 5, the total number of the first parameters of the input preprocessing layer is 60, and the total number of the second parameters of the output processing layer is 40, then the input preprocessing layer should be equivalent to 5*60 / (60+40) = 3 diffusion converter modules, and the output processing layer should be equivalent to 5*40 / (60+40) = 2 diffusion converter modules. Thus, the first model task can be allocated 3 fewer diffusion converter modules, and the last model task can be allocated 2 fewer diffusion converter modules. That is, the second difference corresponding to the first model task is 3, and the third difference corresponding to the last model task is 2. It can be understood that when the second difference and the third difference are not integers, they can be rounded to the nearest integer.

[0175] In step 840, the second difference and the third difference reflect the differences between the number of diffusion converter modules allocated to the first model task and the fifth number, respectively. Based on the difference between the fifth number and the second difference, the number of diffusion converter modules allocated to the first model task, i.e., the number of the first layer, can be determined. Similarly, the number of the second layer can be determined.

[0176] In step 850, after determining the number of the first layer and the number of the second layer, the first number of diffusion converter modules and the input preprocessing layer in the noise prediction layer are assigned as the first model task, the last number of diffusion converter modules in the second layer and the output processing layer in the noise prediction layer are assigned as the last model task, and the remaining diffusion converter modules are assigned as a model task for every fifth adjacent diffusion converter module. This results in a relatively even number of second-number model tasks, and each diffusion converter module is assigned as a whole to a model task and deployed to the corresponding diffusion processing unit.

[0177] Reference Figure 9 The noise prediction layer includes diffusion converter modules 1 to 30, a total of 30 diffusion converters. The diffusion processing model needs to be divided into 4 model tasks. At this time, the first ratio is 7.5, which, after rounding up, results in a fifth number of 8. The corresponding first difference is 2. At this time, the input preprocessing layer and the output processing layer can be equivalent to one diffusion converter module, that is, the second difference and the third difference are 1. The corresponding number of the first layer and the second layer is 7. Based on this, the first model task M_0 includes the input preprocessing layer and diffusion converter modules 1 to 7; the second model task M_1 includes diffusion converter modules 8 to 15; the third model task M_2 includes diffusion converter modules 16 to 23; and the last model task M_3 includes diffusion converter modules 24 to 30, as well as the output processing layer.

[0178] In the embodiments disclosed in steps 810 to 850, the network layer of the diffusion processing model is divided into an input preprocessing layer, a noise prediction layer, and an output processing layer. Based on the number of diffusion converter modules in the noise prediction layer and the second number, the maximum number of diffusion converter modules allocated to a single model task when the fourth number of diffusion converter modules are relatively evenly distributed to the second number of model tasks is determined, and the first difference is determined accordingly. Based on the total number of parameters of the input preprocessing layer and the output processing layer, the number of diffusion converter modules required by the input preprocessing layer and the output processing layer is determined, so as to determine the number of diffusion converter modules required by the first model task and the last model task. The remaining diffusion converter modules are then evenly distributed to the other model tasks except for the first model task and the last model task. In this way, while ensuring that a single diffusion converter module can be completely allocated to a single diffusion processing unit, the diffusion processing model can be relatively evenly distributed to the second number of model tasks.

[0179] Detailed description of step 420

[0180] In step 420, each denoising step is divided into a third number of denoising time slots, and the corresponding model task for each denoising time slot is performed on the image block. Here, the third number is the number of image blocks in the image block group, and the image blocks in the image block group are ordered.

[0181] The implementation method of dividing the denoising step into a third number of denoising time slots has been described in detail above and will not be repeated here. In one embodiment, referring to... Figure 10 In step 420, the model task corresponding to the denoising time slot is performed on the image block in each denoising time slot, including:

[0182] Step 1010: Determine the image block sequence number to be started based on the time slot number of the denoising time slot;

[0183] Step 1020: The first diffusion processing unit in the second number of diffusion processing units divides the image into blocks according to the image block number and executes the first model task.

[0184] Step 1030: In the subsequent denoising time slot after the denoising time slot, other diffusion processing units divide the image into blocks according to the image block sequence number and perform other model tasks.

[0185] In step 1010, the time slot number is used to indicate which denoising time slot in a single denoising step the current denoising time slot is; the image block number is the sequence number corresponding to each image block in a single image block group, and the image block number is used to determine the order in which each image block in a single image block group is input into the diffusion processing model; the image block number to be started refers to the image block number corresponding to the image block that needs to be input into the corresponding diffusion processing unit group in the denoising time slot corresponding to the time slot number. It can be understood that in each denoising time slot of a single denoising step, one image block from the image block group needs to be input into the first diffusion processing unit in the corresponding diffusion processing unit group. Based on this, the image block number corresponding to the image block that needs to be input in the current denoising time slot can be determined according to the time slot number.

[0186] Understandably, the diffusion process involves multiple denoising steps, and each denoising step includes a third number of denoising time slots. Since each denoising step uses the result of the previous denoising step as input to the image to be diffused, for the i-th denoising step, the image block input to the diffusion processing unit group in each denoising time slot is actually the result of the corresponding denoising time slot in the (i-1)-th denoising step inputting the denoised image block into the diffusion processing unit group. Therefore, after executing the third number of denoising time slots in a single denoising step, the time slot number needs to be initialized to ensure that each image block is input to the corresponding diffusion processing unit group in the same order across different denoising steps.

[0187] For example, a loop from 0 to the third number can be set in the Central Processing Unit (CPU) to schedule the image blocks in the image block group. Specifically, in the first denoising time slot of the first denoising step, the time slot number is set to 0; then, after the first diffusion processing unit in the diffusion processing unit group finishes executing the corresponding model task for the image block input in the current denoising time slot, the time slot number is incremented by 1 to enter the next denoising time slot. Based on the updated time slot number, a new image block sequence number is determined and the corresponding image block is scheduled to the first diffusion processing unit in the diffusion processing unit group to execute the corresponding model task, until the time slot number is incremented to the third number, the time slot number is initialized to 0 and the next denoising step begins, until the last denoising step.

[0188] In step 1020, after determining the image block number corresponding to the current denoising time slot, the image blocks that need to be input into the diffusion processing unit group in the current denoising time slot are determined. Each image block, after being input into the diffusion processing unit group, needs to pass through the second number of diffusion processing units in the diffusion processing unit group in an orderly manner to execute the second number of model tasks of the diffusion processing model on the image block. Based on this, after determining the image block input in the current time slot, the first diffusion processing unit in the second number of diffusion processing units, i.e., the first diffusion processing unit in the diffusion processing unit group corresponding to the image block, executes the first model task on the image block corresponding to the image block number. For example, referring to... Figure 6 The third number is 4. In the denoising step with step number t, in the first denoising time slot (i.e., denoising time slot 0), the image block P_0 with image block number 0 is input to the first diffusion processing unit D_0 to perform the first model task on P_0 through the diffusion processing unit D_0; in the second denoising time slot (i.e., denoising time slot 1), the image block P_1 with image block number 1 is input to the diffusion processing unit D_0 to perform the first model task, and so on. When the image block P_3 is input to the diffusion processing unit D_0 and the corresponding model task is completed in the fourth denoising time slot (i.e., denoising time slot 3), the time slot number is incremented by 1, and the time slot number reaches the third number, i.e., 4. At this time, the time slot number is initialized to 0, and the next denoising step, i.e., the denoising step corresponding to step number t-1, begins.

[0189] In step 1030, subsequent denoising time slots are multiple denoising time slots after the first diffusion processing unit performs the first model task on the image block. For example, if the image block is input to the first diffusion processing unit and performs the corresponding model task in the second denoising time slot, then the third denoising time slot and the multiple denoising time slots thereafter are all subsequent denoising time slots.

[0190] Understandably, for each image block input to the diffusion processing unit group, after the first model task is executed on the image block by the first diffusion processing unit, it still needs to have its various model tasks executed sequentially by other diffusion processing units in the diffusion processing unit group to complete one denoising step and predict the corresponding noise. Based on this, in subsequent denoising time slots, image blocks are sequentially transmitted to the diffusion processing units corresponding to each model task. Specifically, the second number of diffusion processing units in a single diffusion processing unit group are ordered. After each diffusion processing unit completes the corresponding model task for an image block input in the current denoising time slot, it sends the image block to the next diffusion processing unit in the same diffusion processing unit group, so that the next diffusion processing unit can execute the next model task for the image block in the next denoising time slot. At the same time, the diffusion processing unit receives the next image block from the previous diffusion processing unit in the diffusion processing unit group to execute the corresponding model task for the next image block in the next denoising time slot.

[0191] For example, refer to Figure 6 In the first denoising time slot (denoising time slot 0), image block P_0 with image block number 0 is input to diffusion processing unit D_0 and the first model task is executed. Then, in the second denoising time slot (denoising time slot 1), image block P_0 is input to diffusion processing unit D_2 in diffusion processing unit group Pg_1, corresponding to the second model task, so that the second model task is executed on image block P_0 through diffusion processing unit D_2. This process continues until image block P_0 has completed a second number of model tasks. Similarly, for image block P_1, in the second denoising time slot, it is input to diffusion processing unit D_0 to execute the first model task on image block P_1. Then, in the third denoising time slot, the second model task is executed on image block P_1 through diffusion processing unit D_2, and so on.

[0192] Based on this, an image block will be executed with corresponding model tasks by each diffusion processing unit in an orderly manner in each denoising time slot. A single diffusion processing unit can also execute corresponding model tasks on different image blocks in each denoising time slot, thus forming a pipelined parallel processing.

[0193] In steps 1010 to 1030, the corresponding image block is determined by identifying the image block number to be started based on the corresponding time slot number in each denoising time slot. The image block is then input to the first diffusion processing unit in the diffusion processing unit group to execute the first model task. After the first model task is completed, the image block is sequentially passed to the other diffusion processing units in the diffusion processing unit group. Thus, by inputting each image block in the image block group into the diffusion processing unit group in its respective denoising time slot, multiple image blocks are subjected to different model tasks in different denoising time slots. In a single denoising time slot, each diffusion processing unit in the diffusion processing unit group executes the corresponding model task for the image blocks input in different denoising time slots. This allows multiple diffusion processing units in the diffusion processing unit group to process different image blocks in parallel, improving the utilization rate of the diffusion processing units and thus increasing the efficiency of the diffusion processing.

[0194] In one embodiment, refer to Figure 11 In step 420, the model task corresponding to the denoising time slot is performed on the image block in each denoising time slot, which also includes:

[0195] Step 1110: For each diffusion processing unit, while executing the corresponding model task on the image block obtained in the current denoising time slot, the image block after the diffusion processing unit executed the corresponding model task in the previous denoising time slot is sent to the next diffusion processing unit in the corresponding diffusion processing unit group.

[0196] Step 1120: For each diffusion processing unit, while executing the corresponding model task on the image block obtained in the current denoising time slot, obtain the image block that the diffusion processing unit needs to execute in the next denoising time slot from the previous diffusion processing unit in the corresponding diffusion processing unit group.

[0197] In step 1110, it is understood that each image block needs to pass through the second number of diffusion processing units in a single diffusion processing unit group in an orderly manner in order to predict the corresponding noise. Based on this, after each diffusion processing unit has completed the corresponding model task for an image block, it needs to send the image block after the model task has been completed to the next diffusion processing unit in the diffusion processing unit group.

[0198] It is understandable that after the diffusion processing unit finishes the corresponding model task for an image block, it will start the next denoising time slot and perform the corresponding model task for the new image block. Then, the image block after the model task is completed is the image block corresponding to the diffusion processing unit in the previous denoising time slot.

[0199] For example, refer to Figure 6After performing the corresponding model task on P_0 in denoising time slot 0, diffusion processing unit D_0 will enter denoising time slot 1 and perform the corresponding model task on P_1. At this time, P_0 is the image block after D_0 performed the corresponding model task in the previous denoising time slot. In denoising time slot 1, while performing the corresponding model task on P_1, diffusion processing unit D_0 sends P_0 to the next diffusion processing unit in the diffusion processing unit group, namely diffusion processing unit D_2.

[0200] In step 1120, each diffusion processing unit obtains the image block after the previous diffusion processing unit has completed its corresponding model task in the previous denoising time slot from the previous diffusion processing unit in the diffusion processing unit group. Since each diffusion processing unit can only start executing its corresponding model task after obtaining the corresponding image block from the previous diffusion processing unit in the diffusion processing unit group, the cycle of the diffusion processing unit in a single diffusion processing unit group executing the corresponding model task for an image block is not completely overlapping. This allows the diffusion processing unit to obtain the image block after the previous diffusion processing unit has completed its corresponding model task from the previous diffusion processing unit in the diffusion processing unit group while executing the corresponding model task for the current image block, that is, to obtain the image block that the diffusion processing unit needs to execute in the next denoising time slot.

[0201] For example, refer to Figure 6 In the first denoising step, after completing the corresponding model task for image block P_0, diffusion processing unit D_0 immediately starts the next denoising time slot (i.e., denoising time slot 1) and executes the corresponding model task for image block P_1. Diffusion processing unit D_2, however, needs to obtain image block P_0 from diffusion processing unit D_0 before executing the corresponding model task for P_0. This results in a time difference between the start times of D_0 and D_2 executing the model task for the corresponding image blocks in denoising time slot 1. Since the size of each image block is equal, the time difference between each diffusion processing unit executing the corresponding model task for one image block is... The time difference is also equal. At this time, there is a time difference between the end time of D_2 executing the corresponding model task for P_0 and the end time of D_0 executing the corresponding model task for image block P_1. This time difference is caused by the delay of the diffusion processing unit D_0 transmitting the image block to the diffusion processing unit D_2. The time required for the diffusion processing unit D_0 to transmit the image block of the same size to the diffusion processing unit D_2 is equal. Based on this, the time difference is sufficient for the diffusion processing unit D_2 to obtain the image block P_1 that needs to be executed in the next denoising time slot from D_0 before the diffusion processing unit D_2 finishes executing the corresponding model task for image block P_0.

[0202] Based on this, the time it takes for each diffusion processing unit to obtain the image block to be processed in the next denoising time slot from the previous diffusion processing unit in its corresponding diffusion processing unit group can be overlapped with the time it takes for the diffusion processing unit to execute the corresponding model task on the image block of the current denoising time slot. This allows for parallel processing of the image block in the current denoising time slot and parallel acquisition of the image block to be processed in the next denoising time slot. Consequently, data transmission between diffusion processing units within the same diffusion processing unit group does not incur additional time consumption. This reduces the communication rate requirements between diffusion processing units within a single diffusion processing unit group, thus improving compatibility with network hardware with lower bandwidth and enabling the diffusion processing process to adapt more flexibly to different underlying network hardware.

[0203] In the embodiments disclosed in steps 1110 to 1120, each diffusion processing unit, while executing the model task corresponding to the current denoising time slot, obtains the image block to be executed in the next denoising time slot from the previous diffusion processing unit in the diffusion processing unit group, and sends the image block executed in the previous denoising time slot to the next diffusion processing unit in the corresponding diffusion processing unit group. This makes the process of transmitting data between diffusion processing units in the same diffusion processing unit group overlap with the process of executing the model task. As a result, the data transmission between diffusion processing units in the same diffusion processing unit group will not cause additional latency. The communication between diffusion processing units in a single diffusion processing unit group has low requirements for communication rate, which can better accommodate network hardware with low bandwidth and make the diffusion processing process more flexibly adaptable to different underlying network hardware.

[0204] Detailed description of step 430

[0205] (I) A specific implementation of step 430

[0206] In step 430, when performing a model task on an image block in the denoising time slot, the first sub-key value is obtained when performing a model task on a sequential image block with the same index as the image block in other image block groups, and merged with the second sub-key value when performing a model task on the image block to obtain a first key value, which is used for the execution of the model task on the image block.

[0207] In one embodiment, within the first number of diffusion processing unit groups, the first number of diffusion processing units corresponding to the same model task constitute a parallel processing unit group, for example, referring to... Figure 12There are two diffusion processing unit groups, Pg_1 and Pg_2. Pg_1 includes D_0, D_2, D_4, and D_6, while Pg_2 includes D_1, D_3, D_5, and D_7. D_0 and D_1 correspond to the same model task, D_2 and D_3 correspond to the same model task, and so on. D_0 and D_1 form a parallel processing unit group Sg_1, D_2 and D_3 form a parallel processing unit group Sg_2, and so on.

[0208] It is understandable that, referring to the descriptions of steps 710 to 730 above, multiple diffusion processing units corresponding to the same model task communicate with each other through a first communication method, thereby efficiently transmitting data. Referring to the relevant descriptions of steps 810 to 830, in a single denoising time slot, multiple diffusion processing units of the same parallel processing unit group execute the corresponding model task on image blocks with the same index in each image block group. For example, in the i-th denoising time slot, multiple diffusion processing units corresponding to the first model task will execute the first model task on the image block with image block index i in each image block group. At this time, referring to... Figure 13 The model tasks include:

[0209] Step 1310: Determine a first sub-block based on the image blocks corresponding to the diffusion processing unit. The first sub-block includes at least the second sub-key value corresponding to the diffusion processing unit.

[0210] Step 1320: Perform sub-block transformation on the first sub-block of each diffusion processing unit in a single parallel processing unit group to obtain the transformed sub-block;

[0211] Step 1330: Perform multi-head attention processing based on the transformed sub-blocks to obtain the first result sub-blocks, and perform sub-block inverse transformation on the first result sub-blocks of each diffusion processing unit in a single parallel processing unit group to obtain the multi-head attention result sub-blocks of each diffusion processing unit.

[0212] Step 1340: Based on the multi-head attention result sub-block, determine the image block of the next diffusion processing unit in the next denoising time slot in the corresponding diffusion processing unit group.

[0213] It should be noted that after the image is divided into blocks and input into the first diffusion processing unit, refer to Figure 3The input image blocks will first be converted into corresponding matrices through the patchify module to execute subsequent model tasks. Before all model tasks are executed, the image blocks exist in the form of matrices. It should be understood that in this disclosure, the image blocks mentioned in the process of executing model tasks can refer to a matrix corresponding to the image blocks obtained in the process of executing model tasks on the image blocks.

[0214] In step 1310, the first sub-block refers to the matrix sub-block obtained based on the image matrix corresponding to the image block, which is used for subsequent multi-head attention processing. Referring to the relevant description in step 410, the model task of each diffusion processing unit is actually to sequentially process the image matrix obtained based on the image block through the diffusion converter module deployed in the diffusion processing unit. After the image matrix is ​​input into the diffusion processing unit, referring to... Figure 3 The diffusion converter module deployed in the diffusion processing unit first performs dimensionality scaling and offset processing on the image matrix, and then inputs the dimensionality-scaled and offset image matrix into the multi-head attention mechanism. In the multi-head attention mechanism, based on the input image matrix, query sub-blocks, key sub-blocks, and value sub-blocks corresponding to the image blocks are generated. These query sub-blocks, key sub-blocks, and value sub-blocks constitute the first sub-block corresponding to the image block, while the key sub-blocks and value sub-blocks constitute the second sub-key value corresponding to the image block, i.e., the second sub-key value corresponding to the diffusion processing unit. Specifically, the image matrix input to the multi-head attention mechanism and the query weight matrix, key weight matrix, and value weight matrix corresponding to the multi-head attention mechanism can be multiplied by matrix operations to obtain the corresponding query sub-blocks, key blocks, and value blocks.

[0215] In step 1320, as described in step 430, when performing multi-head attention processing, it is necessary to perform multi-head attention calculation based on the key matrix and value matrix corresponding to the entire image to be diffused. Since the diffusion processing unit can only process the image blocks in the image block group corresponding to its own diffusion processing unit group, and the first number of diffusion processing units in a parallel processing unit group belong to the first number of diffusion processing unit groups respectively, that is, the first number of diffusion processing units in a parallel processing unit group will perform the same model task on the image blocks in the first number of image block groups respectively. Each diffusion processing unit in the parallel processing unit group will determine the corresponding first sub-block based on the corresponding image block.

[0216] Understandably, the first sub-block is obtained based on the image matrix corresponding to the image blocks. It is a matrix sub-block used for multi-head attention computation. As mentioned above, the image matrix is ​​obtained by dividing the image block into multiple patch blocks, mapping each patch block to a corresponding vector, and then concatenating the vectors corresponding to each patch block. Each patch block can be regarded as a sequence element of the image block. In the process of multi-head attention computation, when mapping the image matrix to the corresponding first sub-block, the image matrix is ​​mapped to different hidden layers. Based on this, the first sub-block includes at least two dimensions: hidden layer dimension and sequence dimension. The sequence dimension corresponds to each element in the image matrix, and the hidden layer dimension corresponds to each hidden layer. For example, for the query sub-block in the first sub-block, each matrix row of the query sub-block corresponds to an element in the image matrix (i.e., a patch block), and each matrix column corresponds to a hidden layer. That is, the matrix row dimension of the query sub-block is the sequence dimension, and the matrix column dimension of the query matrix is ​​the hidden layer dimension.

[0217] Sub-block transformation is the process of converting the first sub-blocks corresponding to all hidden layers of a single image block in each diffusion processing unit within a parallel processing unit group into transformed sub-blocks corresponding to a portion of the hidden layers of multiple sequential image blocks. Specifically, the data corresponding to the same portion of hidden layers in the first sub-blocks of each image block can be aggregated into a single diffusion processing unit and concatenated into transformed sub-blocks. Thus, the transformed sub-blocks include the data corresponding to the same portion of hidden layers of a first number of image blocks with the same image block number in each image block group.

[0218] Further details of step 1320 will be explained later and will not be repeated here.

[0219] In step 1330, after obtaining the transformed sub-blocks, the first sub-key values ​​corresponding to all sequential image blocks with the same block number as the image block processed by the diffusion processing unit in the current denoising time slot are obtained. Since the diffusion processing unit performs corresponding model tasks on image blocks with different block numbers for each denoising time slot... Thus, in each denoising time slot, the first sub-key value corresponding to the image block with different image block number in each image block group can be obtained to form the transformed sub-block. By writing the key sub-block and value sub-block in the transformed sub-block obtained in each denoising time slot into the cache of the diffusion processing unit, the cache of the diffusion processing unit can store the key sub-block and value sub-block corresponding to a part of the hidden layer of all image blocks. As can be seen from the description of step 430 above, when performing multi-head attention processing, reusing a part of the key matrix and value matrix obtained in the previous denoising step will not have a significant impact on the calculation result. Thus, the diffusion processing unit can construct the key matrix and value matrix corresponding to all image blocks based on the key sub-blocks and value sub-blocks obtained in each denoising time slot already executed in the current denoising step, as well as the key sub-blocks and value sub-blocks obtained in multiple denoising time slots of the previous denoising step, to perform multi-head attention processing. For example, a single parallel processing unit group includes two diffusion processing units. The first diffusion processing unit is used to perform the corresponding model task on P_0 to P_3 (image block numbers are 0 to 3 in sequence), and the second diffusion processing unit is used to perform the corresponding model task on P_4 to P_7 (image block numbers are 0 to 3 in sequence). A denoising step includes four denoising time slots. In the first denoising time slot, after performing sub-block transformation, the diffusion processing unit obtains the key sub-blocks and value sub-blocks corresponding to P_0 and P_4 in the current denoising step. At the same time, the buffer of the diffusion processing unit stores the key sub-blocks and value sub-blocks corresponding to P_1 to P_3 and P_5 to P_7 in the previous denoising step. Based on the key sub-blocks and value sub-blocks corresponding to P_0 and P_4 in the current denoising step, as well as the key sub-blocks and value sub-blocks corresponding to P_1 to P_3 and P_5 to P_7 in the previous denoising step, the key matrix and value matrix corresponding to the entire image to be diffused can be constructed and multi-head attention processing can be performed.

[0220] Because the query sub-block also needs to be transformed during sub-block transformation to allocate the attention calculation tasks corresponding to different hidden layers to different diffusion processing units, the transformed sub-block corresponds to a portion of the hidden layers of multiple image blocks with the same image block number. After multi-head attention processing, the first result sub-block also corresponds to a portion of the hidden layers of multiple image blocks with the same image block number, rather than all the hidden layers of a single image block corresponding to the diffusion processing unit in the current denoising slot. Therefore, it is necessary to perform inverse sub-block transformation on the first result sub-block within the first number of diffusion processing units in the same parallel processing unit group. This transforms the first result sub-block, which corresponds to a portion of the hidden layers of multiple image blocks with the same image block number, into a multi-head attention result sub-block corresponding to all the hidden layers of a single image block corresponding to the diffusion processing unit in the current denoising slot. It is understandable that the shape of the resulting multi-head attention result sub-block is the same as the shape of the image matrix corresponding to the image block input to the multi-head attention mechanism.

[0221] In step 1340, refer to Figure 3 The diffusion converter module also includes multiple residual connections, a feedforward neural network, and processing layers for dimensionality scaling and offset operations. After performing multi-head attention processing to obtain multi-head attention result sub-blocks, the multi-head attention result sub-blocks are dimensionally scaled, and then superimposed with the input image matrix through residual connections. The superimposed result is then subjected to dimensionality scaling and offset processing, and nonlinear mapping is performed through the feedforward neural network. The result of the nonlinear mapping is then dimensionally scaled again to obtain the output matrix of the diffusion converter module. This output matrix is ​​the result of performing the model task corresponding to the diffusion processing unit on the image block, that is, the image block that the next diffusion processing unit in the diffusion processing unit group needs to process in the next denoising time slot.

[0222] In the embodiments disclosed in steps 1310 to 1340, before performing multi-head attention processing, the first sub-block of each diffusion processing unit is transformed between the first number of diffusion processing units in the same parallel processing unit group, so that the diffusion processing units in each diffusion processing unit group can obtain the data corresponding to the same image blocks of other diffusion processing unit groups, thereby obtaining the transformed sub-blocks corresponding to a portion of the hidden layers of the first number of same image blocks. By performing sub-block transformation in each denoising time slot and writing the first key value in the transformed sub-block into the buffer of the diffusion processing unit, the diffusion processing units in each diffusion processing unit group can gradually obtain the data corresponding to a portion of the hidden layers of all image blocks, thereby accurately executing the corresponding model tasks in parallel.

[0223] It should be noted that the above description assumes that a model task for a diffusion processing unit includes only one diffusion converter module. In practical applications, the diffusion processing model includes multiple cascaded diffusion converter modules. After dividing the diffusion processing model into a second number of model tasks and assigning them to various diffusion processing units, the model task corresponding to a diffusion processing unit can include multiple cascaded diffusion converter modules, where the output matrix of each diffusion converter module serves as the input matrix of the next diffusion converter module. Each diffusion converter module includes a multi-head attention mechanism. Therefore, in each multi-head attention mechanism, a corresponding first sub-block needs to be generated based on the input matrix of the diffusion converter module, and steps 1310 to 1340 above are executed. This continues until the output matrix of the last diffusion converter module in the model task is obtained. This output matrix is ​​then transmitted to the next diffusion processing unit in the diffusion processing unit group to execute the subsequent model task. This output matrix serves as the image block corresponding to the next diffusion processing unit in the next denoising time slot. Similarly, in the following detailed description of steps 1310 to 1340, it is also described with the case that each model task includes a diffusion converter module. In actual application, the model task corresponding to a diffusion processing unit may include multiple diffusion converter modules that are cascaded in sequence, and each diffusion converter module needs to perform the corresponding operation separately.

[0224] In one embodiment, refer to Figure 14 Step 1320 may include:

[0225] Step 1410: Divide the first sub-block corresponding to each diffusion processing unit into a first number of first blocks along the first dimension, corresponding to the first number of diffusion processing units in the parallel processing unit group.

[0226] Step 1420: Aggregate the first number of first blocks corresponding to a single diffusion processing unit to the corresponding single diffusion processing unit, and splice the first number of first blocks corresponding to a single diffusion processing unit along the second dimension to obtain the transformed sub-block.

[0227] In step 1410, the first dimension is the hidden layer dimension in step 1320. It can be understood that, in this embodiment, the multi-head attention processing tasks corresponding to different hidden layers of the image blocks can be assigned to a first number of diffusion processing units in the parallel processing unit group for parallel execution. Based on this, by dividing the first sub-block along the first dimension, the first sub-block is divided into a first number of first blocks corresponding to the first number of diffusion processing units in the parallel processing unit group, and the first blocks are sent to the corresponding diffusion processing units. Thus, the diffusion processing units can receive the first blocks corresponding to each image block in the same multiple hidden layers. Specifically, the hidden layers that the first number of diffusion processing units in the parallel processing unit group need to calculate can be predetermined, and then each first sub-block is divided based on the hidden layers that each diffusion processing unit needs to calculate, thereby dividing each first sub-block into a first number of blocks corresponding to the first number of diffusion processing units.

[0228] In step 1420, the second dimension is the sequence dimension mentioned in step 1320. It can be understood that the parallel processing unit group includes a first number of diffusion processing units, each of which divides its corresponding first sub-block into a first number of first blocks corresponding to the first number of diffusion processing units. Specifically, since the first number of diffusion processing units in a parallel processing unit group are interconnected via a first communication method, after each diffusion process divides its corresponding first sub-block, it can send each first block to its corresponding diffusion processing unit via the first communication method, thereby aggregating the first number of first blocks corresponding to a single diffusion processing unit into the corresponding diffusion process.

[0229] Since each first sub-block corresponds to a different image block, and each first sub-block is divided along the first dimension into a first number of first blocks corresponding to a first number of diffusion processing units in the parallel processing unit group, and the first number of first blocks corresponding to a single diffusion processing unit correspond to the same hidden layer of different image blocks, based on this, each first block should be concatenated along the sequence dimension, i.e., the second dimension, to obtain a transformed sub-block corresponding to a portion of the hidden layers of multiple image blocks.

[0230] Specifically, refer to Figure 15The parallel processing unit group includes four diffusion processing units, D_0 to D_3. In the current denoising slot, D_0 to D_3 are used to perform model tasks on image blocks P_0 to P_3, respectively. After inputting the multi-head attention mechanism, the first sub-block corresponding to each image block is obtained. Then, each processing unit divides the first sub-block into four blocks along the hidden layer dimension. For example, in diffusion processing unit D_0, the first sub-block corresponding to image block P_0 is divided into four first blocks corresponding to D_0 to D_3, namely P0_1, P0_2, P0_3, and P0_4. Similarly, in D_1 to D_3, image blocks P_1 to P_3 are divided into their corresponding four first blocks. At this point, the diffusion processing unit D_0 corresponds to the four first blocks P0_1, P1_1, P2_1, and P3_1. Aggregating these four first blocks into D_0 and concatenating them along the sequence dimension yields the transformed sub-blocks corresponding to D_0. D_0 then obtains the key and value sub-blocks corresponding to P_1 to P_3. Similarly, D_1 to D_3 also obtain their corresponding transformed sub-blocks, thus obtaining the data corresponding to the image blocks in other image block groups for subsequent multi-head attention processing.

[0231] In steps 1410 to 1420, each first sub-block is divided into a first number of first blocks corresponding to a first number of diffusion processing units along the hidden layer dimension. Then, all first blocks corresponding to the same diffusion processing unit are aggregated into the corresponding diffusion processing unit and spliced ​​along the second dimension. In this way, the first sub-blocks corresponding to all hidden layers of a single image block in each diffusion processing unit are transformed into transformed sub-blocks corresponding to a portion of the hidden layers of all image blocks with the same sequence number. Thus, the attention calculation tasks corresponding to different hidden layers are assigned to each diffusion processing unit in the parallel processing unit group. In this way, multi-head attention processing is performed in parallel by each diffusion processing unit in the parallel processing unit group, that is, the same model task is executed in parallel.

[0232] In one embodiment, refer to Figure 16 Step 1320 may include:

[0233] Step 1610: Divide the query sub-block, key sub-block, and value sub-block into a first number of query blocks, key blocks, and value blocks corresponding to a first number of diffusion processing units in the parallel processing unit group, respectively.

[0234] Step 1620: Aggregate the first number of query blocks, key blocks, and value blocks corresponding to a single diffusion processing unit into a single diffusion processing unit, and concatenate the first number of query blocks, key blocks, and value blocks respectively to obtain transformed sub-blocks. The transformed sub-blocks include at least transformed query sub-blocks, transformed key blocks, and transformed value sub-blocks, wherein the transformed key blocks and transformed value sub-blocks constitute the first key-value pair.

[0235] In step 1610, referring to the description in step 1310, the first sub-block includes a query sub-block, a key sub-block, and a value sub-block obtained based on the image matrix corresponding to the image block. When performing sub-block transformation, the query sub-block, the key sub-block, and the value sub-block need to be transformed separately. Specifically, the query sub-block, the key sub-block, and the value sub-block are all divided along the hidden layer dimension, and are respectively divided into a first number of query blocks, key blocks, and value blocks corresponding to a first number of diffusion processing units in the parallel processing unit group. Among them, the query blocks, key blocks, and value blocks corresponding to the same diffusion processing unit constitute the first block corresponding to that diffusion processing unit.

[0236] In step 1620, after dividing the query sub-blocks, key sub-blocks, and value sub-blocks corresponding to each diffusion processing unit along the first dimension, the first number of query sub-blocks, the first number of key sub-blocks, and the first number of value sub-blocks corresponding to a single diffusion processing unit are aggregated into the corresponding diffusion processing unit. Then, the query sub-blocks, key sub-blocks, and value sub-blocks are concatenated along the sequence dimension to obtain the transformed query sub-blocks, transformed key sub-blocks, and transformed value sub-blocks. Thus, the transformed sub-blocks corresponding to the diffusion processing unit are obtained. The transformed key sub-blocks and transformed value sub-blocks are the first key-value pairs corresponding to the diffusion processing unit in the current denoising time slot.

[0237] In the embodiments disclosed in steps 1610 to 1620, the query sub-block, key sub-block, and value sub-block in the first sub-block are respectively divided into query blocks, key blocks, and value blocks corresponding to a first number of diffusion processing units. Then, the first number of query blocks, key blocks, and value blocks corresponding to a single diffusion processing unit are aggregated into the corresponding diffusion processing unit and concatenated along the second dimension. In this way, each diffusion processing unit obtains the transformed blocks required to calculate the multi-head attention of multiple image blocks in a part of the hidden layer, so that multi-head attention processing can be performed in parallel by the first number of diffusion processing units in the parallel processing unit group, thereby improving the efficiency of multi-head attention processing and thus improving the efficiency of diffusion processing.

[0238] In one embodiment, refer to Figure 17 After performing sub-block transformation to obtain the transformed sub-blocks, the diffusion processing method also includes:

[0239] Step 1710: Write the transformed key blocks and transformed value blocks into the first buffer of the diffusion processing unit;

[0240] Step 1720: In the first buffer, the transformed key blocks and transformed value blocks are concatenated with the transformed key blocks and transformed value blocks obtained by the diffusion processing unit in multiple denoising time slots before the current denoising time slot, respectively, to obtain the transformed key matrix and transformed value matrix.

[0241] In step 1710, during multi-head attention processing, each attention head needs to calculate the key blocks and value blocks of the corresponding hidden layers for all image blocks. However, the diffusion processing unit can only obtain the key blocks and value blocks of multiple image blocks corresponding to one image block number in a single denoising time slot. Image blocks corresponding to different image block numbers are input to the diffusion processing unit in different denoising time slots to perform corresponding model tasks. Since multi-head attention calculation requires the use of key blocks and value blocks of all image blocks in their corresponding hidden layers, in each denoising time slot, a sub-block transformation is performed on the first sub-block. After obtaining the transformed sub-block, the transformed key blocks and transformed value blocks are written into the first buffer of the diffusion processing unit. Thus, during multi-head attention processing in each denoising time slot, the key blocks and value blocks of the image blocks corresponding to other image block numbers obtained from previous denoising time slots can be read from the first buffer, thereby constructing a key matrix and value matrix corresponding to the entire image to be diffused for multi-head attention processing.

[0242] In step 1720, after writing the transformed key blocks and transformed value blocks obtained in the current denoising time slot into the first buffer, they are concatenated along the sequence dimension with the transformed key blocks and transformed value blocks obtained by the diffusion processing unit in the previous multiple denoising time slots, which correspond to the sequence numbers of each image block respectively. In this way, the first buffer stores the key matrix and value matrix corresponding to a part of the hidden layer of all image blocks, that is, the transformed key matrix and transformed value matrix.

[0243] In the embodiments disclosed in steps 1710 to 1720, in each denoising time slot, the transformed key blocks and transformed value blocks obtained by the diffusion processing unit are written into the first buffer of the diffusion processing unit. Thus, the first buffer of the diffusion processing unit will cache the transformed key blocks and transformed value blocks obtained in multiple denoising time slots, which correspond to the sequence numbers of each image block respectively. Then, in the first buffer, the transformed key blocks and transformed value blocks obtained in the current denoising time slot are concatenated with the transformed key blocks and transformed value blocks obtained in the previous multiple denoising time slots. In this way, the transformed key matrix and transformed value matrix corresponding to a part of the hidden layer of all image blocks are obtained. Subsequently, multi-head attention processing can be performed based on the transformed key matrix and transformed value matrix to predict the corresponding noise.

[0244] In one embodiment, since the image blocks corresponding to different image block numbers are input to the diffusion processing unit in different denoising time slots to execute the corresponding model tasks in each denoising step, the diffusion processing unit needs to write the corresponding transformed key blocks and transformed value blocks into the first buffer in each denoising time slot to obtain the key blocks and value blocks corresponding to all image blocks. However, since the denoising time slots corresponding to the same time slot number in each denoising step execute the corresponding model tasks for the image blocks corresponding to the same image block number, the first buffer can be divided into buffer sub-regions corresponding to each denoising time slot, and the transformed key blocks and transformed value blocks obtained in each denoising time slot can be written into the buffer sub-region corresponding to the denoising time slot. (Refer to...) Figure 18 Step 1710 includes:

[0245] Step 1810: Determine the cache sub-region corresponding to the current denoising time slot;

[0246] Step 1820: If the key block and value block written in the previous denoising step exist in the cache sub-area corresponding to the current denoising time slot, update the transformed key block and transformed value block in the cache sub-area corresponding to the current denoising time slot using the transformed key block and transformed value block of the current denoising time slot.

[0247] In step 1810, it can be understood that the first buffer includes a buffer sub-area corresponding to each denoising time slot. The current denoising time slot can be determined as which denoising time slot in the denoising step it is based on the time slot number, and the corresponding buffer sub-area can be found.

[0248] In step 1820, it is understood that since the acquired transformed key blocks and transformed value blocks are written to the corresponding buffer sub-area of ​​the denoising time slot in each denoising step, the first buffer of the diffusion processing unit can store the key blocks and value blocks corresponding to a portion of the hidden layers of all image blocks for multi-head attention processing.

[0249] In different denoising steps, the denoising time slots corresponding to the same time slot number all perform the corresponding model task on the image block corresponding to the same image block sequence number. That is, the transformed key blocks and transformed value blocks stored in the buffer sub-area of ​​the current denoising time slot correspond to the same image block as the transformed key blocks and transformed value blocks obtained by performing sub-block transformation in the current denoising time slot. For example, in the j-th denoising time slot of the i-th denoising step, after sub-block transformation, the transformed key blocks and transformed value blocks corresponding to multiple image blocks with image block sequence number j in the i-th denoising step are obtained. The buffer sub-area corresponding to the j-th denoising time slot stores the transformed key blocks and transformed value blocks corresponding to multiple image blocks with image block sequence number j in the (i-1)-th denoising step. At this point, by updating the corresponding cache sub-regions based on the transformed key blocks and transformed value blocks obtained from the sub-block transformation in the current denoising time slot, more key blocks and value blocks obtained in the current denoising step can be used in subsequent multi-head attention processing, reducing the use of outdated key blocks and value blocks obtained in previous denoising steps, thereby further improving the accuracy of multi-head attention processing.

[0250] In the embodiments disclosed in steps 1810 to 1820, the first buffer is divided into buffer sub-regions corresponding to each denoising time slot. Each buffer sub-region is used to store the transformed key blocks and transformed value blocks obtained in the corresponding denoising time slot. In this way, each buffer sub-region corresponds to multiple sequential image blocks corresponding to the same image block number. In each denoising time slot, the buffer sub-region corresponding to the denoising time slot is updated using the transformed key blocks and transformed value blocks obtained in the current denoising time slot. This ensures that the first buffer contains the key blocks and value blocks corresponding to all image blocks, and the transformed key blocks and transformed value blocks corresponding to each image block number are updated step by step. This allows more transformed key blocks and transformed value blocks obtained in the current denoising step to be used during multi-head attention processing, further improving the accuracy of multi-head attention processing.

[0251] After performing the sub-block transformation, each diffusion processing unit obtains the transformed query sub-block corresponding to a portion of the hidden layers of the image block that the diffusion processing unit needs to process in the current denoising slot, as well as the transformed key matrix and transformed value matrix corresponding to a portion of the hidden layers of all image blocks, which are required for multi-head attention processing. Thus, multi-head attention computation can begin. In one embodiment, refer to... Figure 19 The process of multi-head attention processing includes:

[0252] Step 1910: Divide the transformed query sub-block, transformed key matrix, and transformed value matrix into transformed query blocks, transformed key blocks, and transformed value blocks corresponding to each attention head, respectively, and assign the transformed query blocks, transformed key blocks, and transformed value blocks to the corresponding attention heads.

[0253] Step 1920: In each attention head, attention is calculated based on the corresponding transformed query block, transformed key block, and transformed value block.

[0254] In step 1910, in the multi-head attention mechanism, attention results for a portion of the hidden layers are calculated using multiple attention heads. After obtaining the transformed query sub-block, transformed key matrix, and transformed value matrix, these blocks are segmented along the first dimension, resulting in transformed query blocks, transformed key blocks, and transformed value blocks corresponding to each attention head. Specifically, the segmentation step size can be determined based on the hidden layer size corresponding to a single attention head to segment the transformed query sub-block, transformed key matrix, and transformed value matrix. For example, if the hidden layer size corresponding to a single attention head is 32 hidden layers, then every 32 hidden layers in the transformed query sub-block, transformed key matrix, and transformed value matrix are segmented into a corresponding transformed query block, transformed key block, or transformed value block. It should be noted that the segmentation rules are consistent when segmenting the transformed query block, transformed key block, and transformed value block. For example, when segmenting the transformed query sub-blocks, each 32 hidden layers are treated as one transformed query block corresponding to an attention head. Similarly, for the transformed key matrix and transformed value matrix, each 32 hidden layers are treated as one transformed key block and transformed value block corresponding to an attention head. Then, each transformed query block, transformed key block, and transformed value block is assigned to its corresponding attention head. It can be understood that the transformed query block, transformed key block, and transformed value block corresponding to a single attention head correspond to the same number of hidden layers.

[0255] In step 1920, after each attention head obtains the corresponding transformed query block, transformed key block, and transformed value block, attention calculation is performed. Specifically, the attention calculation process can be referred to the following formula:

[0256]

[0257] Where Q represents the transformed query block; K T This represents the key block after transpose; V represents the value block after transformation; softmax() represents the exponential normalization function; d kO represents the size of the hidden layer dimension, i.e., the size of the hidden layer of a single attention head; O represents the attention calculation result of a single attention head.

[0258] After obtaining the attention calculation results of each attention head, the attention calculation results of each attention head are concatenated along the hidden layer dimension to obtain the corresponding first result sub-block.

[0259] In the embodiments disclosed in steps 1910 to 1920, the transformed query sub-block, the transformed key matrix, and the transformed value matrix are divided into parts corresponding to each attention head, and each attention head is assigned to perform the corresponding attention calculation task, thereby obtaining the corresponding multi-head attention processing result.

[0260] In one embodiment, referring to the relevant description of step 1330 above, the first result sub-block obtained by each diffusion processing unit corresponds to a portion of the hidden layer of multiple image blocks with the same image block number. This requires sub-block transformation of the first result sub-blocks of each diffusion processing unit in a single parallel processing unit group, referring to... Figure 20 In one embodiment, in step 1330, a sub-block inverse transformation is performed on the first result sub-block of each diffusion processing unit in a single parallel processing unit group to obtain the multi-head attention result sub-block of each diffusion processing unit, including:

[0261] Step 2010: In each diffusion processing unit, the first result sub-block is divided along the second dimension into a first number of result blocks corresponding to each diffusion processing unit in the parallel processing unit group;

[0262] Step 2020: Aggregate the first number of result blocks corresponding to a single diffusion processing unit into the corresponding single diffusion processing unit, and splice the first number of result blocks corresponding to a single diffusion processing unit along the first dimension to obtain the multi-head attention result sub-block corresponding to a single diffusion processing unit.

[0263] In step 2010, since the query blocks corresponding to each image block are concatenated into the transformed query block along the second dimension during the sub-block transformation, and after multi-head attention processing is performed based on the transformed query block, the parts corresponding to each image block in the first result sub-block are also arranged along the second dimension. Based on this, the first result sub-block can be divided into result blocks corresponding to each image block along the second dimension. Each image block corresponds to a diffusion processing unit in the parallel processing unit group. Thus, the diffusion processing unit corresponding to each result block can be determined. For example, the first number is 4. Each image block corresponds to a query sub-block with 32 matrix rows and 32 matrix columns. During the sub-block transformation, each query sub-block is divided into 4 query blocks of shape 32*8 and sent to 4 diffusion processing units in the parallel processing unit group. A single diffusion processing unit can obtain 4 query blocks of shape 32*8. After concatenation along the second dimension, a transformed query block of shape 128*8 is obtained. In this transformed query block, every 32 matrix rows correspond to an image block. After multi-head attention processing, the first result sub-block is also a 128*8 matrix, where every 32 matrix rows are the result block corresponding to an image block.

[0264] In step 2020, each result block corresponding to a single diffusion processing unit is the attention calculation result of the image block corresponding to that diffusion processing unit in the current denoising time slot at each different hidden layer. Based on this, by aggregating the first number of result blocks corresponding to a single diffusion processing unit to the corresponding diffusion processing unit, and then concatenating them along the hidden layer dimension, the multi-head attention calculation result corresponding to all hidden layers of the image block is obtained, i.e., the multi-head attention result sub-block.

[0265] Specifically, refer to Figure 21The parallel processing unit group comprises four diffusion processing units, D_0 to D_3. During the current denoising time slot, D_0 to D_3 are used to execute corresponding model tasks on P_0 to P_3, respectively. After multi-head attention processing, diffusion processing unit D_0 obtains a corresponding first result sub-block, which includes four parts: Out0_1, Out1_1, Out2_1, and Out3_1. These are the attention results of P_0 to P_3 calculated in diffusion processing unit D_0 for a portion of the hidden layers. The first result sub-blocks obtained by D_1 to D_3 each contain result blocks corresponding to P_0 in different hidden layers, namely Out0_2, Out0_3, and Out0_4. At this point, Out0_1, Out0_2, Out0_3, and Out0_4 are aggregated into D_0 and concatenated along the hidden layer dimension, thus obtaining the attention results of P_0 for all hidden layers within diffusion processing unit D_0. Similarly, by aggregating Out1_1, Out1_2, Out1_3, and Out1_4 into D_1 and concatenating them along the hidden layer dimension, D_1 yields the attention results of P_1 across all hidden layers.

[0266] In the embodiments disclosed in steps 2010 to 2020, after obtaining the first result sub-block, the first result sub-block is divided into result blocks corresponding to each image block along the sequence dimension. Then, the result blocks corresponding to the image blocks of other diffusion processing units in the parallel processing unit group are sent to the corresponding diffusion processing units. At the same time, the result blocks corresponding to different hidden layers of the corresponding image blocks are obtained from the other diffusion processing units in the parallel processing unit group. Thus, the first result sub-blocks obtained by multi-head attention processing, which correspond to a portion of the hidden layers of multiple image blocks, are transformed into multi-head attention result sub-blocks corresponding to all the hidden layers of the image blocks processed by the diffusion processing units. Therefore, by performing corresponding processing on the multi-head attention result sub-blocks, the model task that needs to be performed on the input image blocks is completed.

[0267] (II) Another specific implementation of step 430

[0268] The above text illustrates one implementation of step 430. The following text illustrates another implementation of step 430, in which the first sub-key value is obtained when performing a model task on a sequential image block with the same index as the image block in other image block groups, and is merged with the second sub-key value when performing a model task on the image block to obtain a first key value, and the model task is performed based on the first key value.

[0269] In one embodiment, refer to Figure 22 The model tasks include:

[0270] Step 2210: Determine the corresponding second subkey value and query sub-block based on the image block corresponding to the diffusion processing unit, and write the second subkey value into the first buffer of the diffusion processing unit;

[0271] Step 2220: Divide a single denoising time slot into multiple ordered sub-denoising time slots, execute the first process in each sub-denoising time slot, and simultaneously obtain the corresponding first sub-key value from other diffusion processing units in the parallel processing unit group and write it into the first buffer. The first process includes: performing multi-head attention processing on the key sub-block and value sub-block corresponding to the query sub-block and the sub-denoising time slot to obtain the row normalized coefficient vector and attention intermediate result of each attention head in the sub-denoising time slot; wherein, the key sub-block and value sub-block corresponding to the first sub-denoising time slot are the second sub-key value, and the key sub-block and value sub-block corresponding to each other sub-denoising time slot are the first sub-key value obtained in the previous sub-denoising time slot, or the first sub-key value obtained in multiple denoising time slots before the current denoising time slot;

[0272] Step 2230: For each attention head, the row normalization coefficient vector and the intermediate attention result corresponding to each sub-denoising time slot of the attention head are accumulated to obtain the corresponding accumulated attention result. The accumulated attention result is then corrected based on the corresponding target row normalization coefficient vector to obtain the attention processing result corresponding to the attention head.

[0273] Step 2240: The attention processing results corresponding to each attention head are spliced ​​together to obtain a multi-head attention result sub-block, and the image block of the next diffusion processing unit in the next denoising time slot is determined based on the multi-head attention result sub-block.

[0274] In step 2210, when the diffusion processing unit performs the model task on the corresponding image block, after the image matrix corresponding to the image block is input into the multi-head attention mechanism, the image matrix is ​​multiplied with the query weight matrix, key weight matrix and value weight matrix of the multi-head attention mechanism to obtain the corresponding query sub-block, key block and value sub-block. The key block and value sub-block constitute the second sub-key value corresponding to the image block.

[0275] In step 2220, a sub-denoising slot refers to the processing cycle corresponding to the attention processing performed on the query sub-block and the key sub-blocks and value sub-blocks corresponding to a single image block. It can be understood that when executing the model task, multi-head attention computation is required based on the query sub-block, the key matrix composed of the key sub-blocks corresponding to all image blocks, and the value matrix composed of the value sub-blocks corresponding to all image blocks. This process can be decomposed into multiple sub-processes that perform attention computation based on the query sub-block and the key and value sub-blocks corresponding to a single image block, with each sub-process corresponding to a sub-denoising slot. The first process is the attention computation performed within a single sub-denoising slot.

[0276] Specifically, a single diffusion processing unit only performs the corresponding model task on image blocks within the image block group corresponding to its own diffusion processing unit group, and locally calculates the key blocks and value blocks corresponding to these image blocks. It cannot locally calculate the key blocks and value blocks corresponding to image blocks in other image block groups. However, the first few diffusion processing units in the parallel processing unit group belong to different diffusion processing unit groups and are used to process image blocks in different image block groups. Therefore, during the execution of the model task, the diffusion processing unit needs to obtain the key blocks and value blocks corresponding to image blocks in other image block groups from other diffusion processing units in the same parallel processing unit group to execute the model task. If the diffusion processing unit performs attention processing only after obtaining the corresponding first sub-key value from other diffusion processing units in the parallel processing unit group, the efficiency of the diffusion processing unit in executing the model task will be low.

[0277] Based on this, in this embodiment, a denoising time slot is divided into multiple sub-denoising time slots. In each sub-denoising time slot, the diffusion processing unit performs a first process based on the query sub-block, the key sub-block, and the value sub-block corresponding to the sub-denoising time slot. Simultaneously, it obtains the corresponding first sub-key value from other diffusion processing units in the parallel processing unit group and writes the obtained first sub-key value into a first buffer. Since the steps of performing the first process and obtaining the first sub-key value corresponding to the same-order image blocks from other diffusion processing units are performed in parallel, this ensures the efficiency of the diffusion processing unit in performing the model task while obtaining the first sub-key value required for performing the model task from other diffusion processing units in the parallel processing unit group. The obtained first sub-key value is then written into the first buffer. After completing one denoising step, the first buffer of the diffusion processing unit will store the key blocks and value sub-blocks corresponding to all image blocks.

[0278] Specifically, refer to Figure 23The diffusion processing unit D_0, along with multiple diffusion processing units such as D_1, D_2, and D_3, forms a parallel processing unit group. In the current denoising time slot, D_0 obtains Q_0, K_0, and V_0 for multi-head attention processing based on the corresponding image blocks; D_1 obtains Q_1, K_1, and V_1 for multi-head attention processing based on the corresponding image blocks, and so on. In the first sub-denoising time slot (i.e., sub-denoising time slot 0), the diffusion processing unit D_0 first performs the first process (i.e., attention calculation) based on Q_0, K_0, and V_0, and simultaneously obtains K_1 and V_1 from the diffusion processing unit D_1. In the second sub-denoising time slot (i.e., sub-denoising time slot 1), D_0 performs the first process based on K_1, V_1, and Q_0 obtained in the first sub-denoising time slot, and simultaneously obtains K_2 and V_2 from the diffusion processing unit D_2, and so on, until all key blocks and value blocks obtained by the diffusion processing units in the parallel processing unit group in the current denoising time slot are obtained and the corresponding first process is executed. Afterwards, attention is calculated using the key blocks and value blocks of the image blocks corresponding to other image block numbers, obtained from multiple denoising time slots before the current denoising time slot and stored in the first buffer, thus completing the corresponding model task based on the key blocks and value blocks of all image blocks.

[0279] Furthermore, referring to the description of step 430 above, using more image blocks in the key and value sub-blocks corresponding to the current denoising step during the execution of the model task can further improve the accuracy of the noise predicted in the current denoising step. In this embodiment, after obtaining the corresponding first sub-key value in each denoising sub-slot, the obtained first sub-key value is used as the key and value sub-blocks corresponding to the next sub-denoising sub-slot. Thus, in the next sub-denoising sub-slot, the first process can be performed based on the first sub-key value obtained in the previous sub-denoising sub-slot, thereby using more image blocks in the key and value blocks corresponding to the current denoising step during the execution of the model task, further improving the accuracy when executing the model task.

[0280] Specifically, in one embodiment, reference is made to Figure 24 In step 2220, the first process includes:

[0281] Step 2410: Divide the query sub-block, the key sub-block and the value sub-block corresponding to the sub-denoising time slot into query blocks, key blocks and value blocks corresponding to each attention head and assign them to the corresponding attention heads;

[0282] Step 2420: For each attention head, determine the product matrix based on the corresponding query block and key block, and perform indexing on the product matrix to obtain the indexed product matrix; accumulate each row of the indexed product matrix to obtain the row normalized coefficient vector corresponding to the attention head; perform matrix multiplication on the indexed product matrix and the corresponding value block to obtain the attention intermediate result corresponding to the attention head.

[0283] The details of step 2410 can be found in the description of step 1810 above, and will not be repeated here.

[0284] In step 2420, after each attention head obtains its corresponding query block, key block, and value block, it first multiplies the query block and the transposed key block to obtain the corresponding product matrix. Then, the product matrix is ​​exponentialized, meaning each element of the product matrix is ​​used as the exponent of the natural base e, resulting in an exponentialized product matrix. When performing attention calculations, the exponentialized product matrix of the query sub-block and the key matrix needs to be normalized. However, since only the product matrix between the query sub-block and the key sub-block is calculated in each sub-denoising time slot, and not the product matrix between the query sub-block and the key matrices corresponding to all image blocks, the product matrix obtained in a single sub-denoising time slot is incomplete compared to the product matrix between the query sub-block and the key matrices corresponding to all image blocks. Therefore, directly performing exponential normalization based on the product matrix obtained in the current denoising time slot will result in inaccurate normalization. The normalization process involves exponentializing the elements of each row in the sub-index matrix and then dividing them by the normalization coefficient corresponding to that row (i.e., the sum of all elements in that row). Based on the properties of matrix operations: (a*A)*B=a*(A*B), where A and B are two matrices to be multiplied and a is a constant, we can first perform matrix multiplication between the exponentialized product matrix obtained from the current sub-denoising slot and the corresponding value sub-block to obtain the attention intermediate result. Then, based on the product of the query sub-block and the complete key matrix, we determine the normalization coefficient corresponding to each row and perform normalization on the attention intermediate result.

[0285] During attention calculation, the key matrix needs to be transposed before matrix multiplication with the query sub-block. After transposing the key matrix, the sequence dimension becomes the column dimension of the matrix, and the key sub-blocks corresponding to each image block become concatenated along the column dimensions of the matrix. This is based on the following properties of matrix operations:

[0286] A*B=A*(B1|B2)=(A*B1|A*B2)

[0287] Here, A and B represent two matrices being multiplied. (B1|B2) indicates that matrix B is divided into two sub-blocks, B1 and B2, along the column dimension. (A*B1|A*B2) indicates that matrix A is multiplied by sub-blocks B1 and B2 respectively, and the products are concatenated into a single matrix. It can be seen that the matrix obtained by exponentializing the product of the query sub-block and the complete key matrix is ​​actually obtained by exponentializing the product of the query sub-block and each key sub-block separately, and then concatenating the products along the column dimension. Correspondingly, the sum of the elements in each row of the matrix obtained after exponentializing the product of the query sub-block and the complete key matrix can be obtained by performing matrix multiplication on the query sub-block and each key sub-block respectively, obtaining multiple product matrices, exponentializing them, and then summing the elements in the same rows of the multiple exponentialized product matrices. Based on this, in each sub-denoising time slot, the rows of the exponentialized product matrix obtained in the current sub-denoising time slot can be summed to obtain the row normalized coefficient vector corresponding to each row. Then, the row normalized coefficient vectors obtained in all sub-denoising time slots of a single denoising time slot are accumulated to obtain the accurate normalized coefficients corresponding to each row. After that, the intermediate attention results are normalized to obtain the accurate attention calculation results.

[0288] In addition, in one embodiment, the elements in the product matrix can be scaled before exponentialization is performed to reduce the excessive relative differences between the elements in the product matrix caused by exponentialization. The scaling factor in the scaling process can be the square root of the hidden layer size.

[0289] In the embodiments disclosed in steps 2410 to 2420, attention is directly calculated based on the key blocks and value blocks corresponding to the query block and a single image block in each sub-denoising time slot. During the attention calculation process, the normalization process is not performed on the exponentialized product matrix; instead, the exponentialized product matrix and the value block are directly multiplied to obtain the corresponding intermediate attention result. Simultaneously, the rows of the exponentialized product matrix are summed, and the summation result is stored as a row normalization coefficient vector. This is used to calculate the accurate normalization coefficients corresponding to each matrix row when the query block is multiplied by the complete key matrix, based on the row normalization coefficient vectors obtained from all sub-denoising time slots. Then, the intermediate attention result sub-block is normalized. This avoids the inaccuracy in the normalization process caused by only calculating the product matrix of the query block and a single key block in each sub-denoising time slot. Therefore, while ensuring the accuracy of the attention calculation, the attention processing process is divided into multiple sub-processes. This allows for direct attention calculations based on the key and value sub-blocks of a single image block. Simultaneously, the first sub-key value corresponding to other image blocks in the current denoising step can be obtained from other diffusion processing units in the parallel processing unit group. While ensuring the efficiency of the diffusion processing unit in executing the model task, more key and value sub-blocks corresponding to the current denoising step of the image block can be used to execute the model task, thereby improving the accuracy when executing the model task.

[0290] In step 2230, referring to the relevant description of step 2420 above, by accumulating the row normalization coefficient vectors corresponding to each sub-denoising time slot of the attention head, the summation result of each matrix row corresponding to the matrix multiplication of the query sub-block with the complete key matrix can be obtained, that is, the accurate normalization coefficients corresponding to each matrix row, thereby obtaining the target normalization coefficient vector.

[0291] Referring to the description in step 2420, the exponentialized product matrices corresponding to each sub-denoising slot are concatenated along the column dimension to form the matrix obtained by multiplying the query sub-block with the complete key matrix (the matrix obtained by concatenating the key sub-blocks of all image blocks along the sequence dimension) and then exponentializing it. This matrix is ​​the attention score matrix, where each exponentialized product matrix is ​​equivalent to a matrix block obtained by partitioning the attention score matrix along the column dimension. Similarly, the value sub-blocks corresponding to each image block are concatenated along the row dimension to form the value matrix corresponding to all image blocks. Each value sub-block is a matrix block obtained by partitioning the value matrix along the row dimension. The result of multi-head attention processing is obtained by normalizing the attention score matrix and then multiplying it with the value matrix, based on the properties of matrix operations.

[0292]

[0293] Where A*B represents multiplying matrix A and matrix B; (A1|A2) represents dividing matrix A into two matrix blocks A1 and A2 along the column dimension; This indicates that matrix B is divided into two matrix blocks, B1 and B2, along the row dimension. It can be seen that the intermediate attention results of the attention head in each sub-denoising slot can be accumulated to obtain the result of multiplying the attention score matrix and the value matrix, and then normalized to obtain the multi-head attention result sub-block. Therefore, by accumulating the intermediate attention results of the attention head in each sub-denoising slot, the accumulated attention result is obtained.

[0294] Specifically, this involves dividing each element of the matrix row in the accumulated attention result by the normalization coefficient of the corresponding matrix row in the target normalization coefficient vector. For example, the normalization coefficient corresponding to the first matrix row in the target normalization coefficient vector is e. 12 Then divide each element of the first row of the accumulated attention result by e. 12 This corrects the cumulative attention result to the result under the normalized state, thus obtaining the multi-head attention processing result.

[0295] Understandably, storing the normalized coefficient vectors obtained by the attention head in each sub-denoising time slot separately, and then accumulating them in the last sub-denoising time slot, would consume a significant amount of memory space in the diffusion processing unit. Therefore, in one embodiment, after obtaining the row normalized coefficient vector corresponding to the attention head in each sub-denoising time slot, it can be accumulated into the row normalized coefficient vector corresponding to the attention head in the previous sub-denoising time slot. This way, for each attention head, only the memory space of one row normalized coefficient vector is needed throughout the entire denoising time slot. Similarly, the intermediate attention results of a single attention head in each sub-denoising time slot can also be directly accumulated into the intermediate attention results of the previous attention head in the previous sub-denoising time slot.

[0296] In step 2240, after the first process corresponding to the last sub-denoising slot is executed and the attention processing results corresponding to each attention head are obtained, the attention processing results are concatenated along the hidden layer dimension to obtain the corresponding multi-head attention result sub-block. Subsequently, the process of determining the image block segmentation of the next diffusion processing unit in the next denoising slot based on the multi-head attention result sub-block can be referred to the relevant description in step 1340, and will not be repeated here.

[0297] In the embodiments disclosed in steps 2210 to 2240, the denoising time slot is divided into multiple denoising sub-time slots, and in each denoising sub-time slot, a first process is performed based on the query sub-block and the key sub-block and value sub-block corresponding to the denoising sub-time slot. The corresponding first sub-key value is obtained from other diffusion processing units in the parallel processing unit group, so that the attention calculation process and the process of obtaining the first sub-key value are executed in parallel. While ensuring that the efficiency of attention calculation is not reduced, the first sub-key value corresponding to each image block is obtained in a timely manner so that it can be used in the process of executing the model task.

[0298] It should be noted that, referring to the relevant descriptions of steps 1310 to 1340, this embodiment describes a model task of a diffusion processing unit that includes only one diffusion converter module. In practical applications, the model task corresponding to a diffusion processing unit may include multiple diffusion converter modules cascaded in sequence. The output matrix of each diffusion converter module serves as the input matrix of the next diffusion converter module. Each diffusion converter module needs to use the output matrix of the previous diffusion converter module as the image matrix corresponding to the image block to perform the operations of steps 2210 to 2240 and steps 2410 to 2420 until the output matrix of the last diffusion converter module in the model task is obtained. The output matrix of the last diffusion converter module then serves as the image block corresponding to the next diffusion processing unit in the next denoising time slot in the diffusion processing unit group.

[0299] Description of detailed embodiments in this disclosure

[0300] The following is for reference. Figure 25 This document describes in detail an exemplary application process according to an embodiment of the present disclosure. The process includes, but is not limited to, steps 2501 to 2528, in which the server undertakes all tasks of the diffusion processing method. In actual use, the server and the terminal may each undertake a portion of the tasks of the diffusion processing method.

[0301] 2501. Determine the first number based on the number of other diffusion processing units connected to a single diffusion processing unit via a first communication method;

[0302] 2502. Determine the second number based on the first number and the number of diffusion processing units;

[0303] 2503. The first number of diffusion processing units connected by the first communication method are respectively assigned to the first number of diffusion processing unit groups, and the second number of diffusion processing units in a single diffusion processing unit group are connected by the second communication method;

[0304] 2504. The network layers of the diffusion processing model are divided into an input preprocessing layer, a noise prediction layer, and an output processing layer. The noise prediction layer includes a fourth number of cascaded diffusion converter modules.

[0305] 2505. Determine the first ratio between the fourth number and the second number, round the first ratio up to obtain the fifth number, and determine the first difference based on the product of the second number and the fifth number and the fourth number;

[0306] 2506. Obtain the total first parameter of the input preprocessing layer and the total second parameter of the output processing layer, and based on the ratio of the total first parameter and the total second parameter and the first difference, determine the second difference corresponding to the first model task and the third difference corresponding to the last model task, wherein the sum of the second difference and the third difference is the first difference;

[0307] 2507. Based on the fifth number, the second difference, and the third difference, determine the number of the first layer and the number of the second layer, respectively;

[0308] 2508. Assign the input preprocessing layer, the fourth number of diffusion converter modules, and the output processing layer to the second number of model tasks;

[0309] 2509. Divide the image to be diffused into a first number of image block groups and distribute them in parallel to the first number of diffusion processing unit groups for processing;

[0310] 2510. Divide each denoising step into a third number of denoising time slots, and determine the image block sequence number to be started based on the time slot number of the denoising time slot;

[0311] 2511. The first diffusion processing unit in the second number of diffusion processing units performs the first model task on the image block of the image block number;

[0312] 2512. In subsequent denoising time slots following the denoising time slot, other diffusion processing units divide the image into blocks according to the image block sequence number and perform other model tasks.

[0313] 2513. For each diffusion processing unit, while executing the corresponding model task on the image block acquired in the current denoising time slot, the image block after the diffusion processing unit executed the corresponding model task in the previous denoising time slot of the current denoising time slot is sent to the next diffusion processing unit in the corresponding diffusion processing unit group.

[0314] 2514. For each diffusion processing unit, while executing the corresponding model task on the image block obtained in the current denoising time slot, the image block that the diffusion processing unit needs to execute in the next denoising time slot is obtained from the previous diffusion processing unit in the corresponding diffusion processing unit group;

[0315] 2515. Determine the first sub-block based on the image blocks corresponding to the diffusion processing unit;

[0316] 2516. Divide the first sub-block corresponding to each diffusion processing unit into a first number of first blocks corresponding to a first number of diffusion processing units in the parallel processing unit group along the first dimension;

[0317] 2517. Aggregate the first number of first blocks corresponding to a single diffusion processing unit to the corresponding single diffusion processing unit, and splice the first number of first blocks corresponding to a single diffusion processing unit along the second dimension to obtain the transformed sub-block, and determine the buffer sub-region corresponding to the current denoising time slot;

[0318] 2518. If the transformed key block and transformed value block written in the previous denoising step exist in the cache sub-area corresponding to the current denoising time slot, update the transformed key block and transformed value block in the cache sub-area corresponding to the current denoising time slot using the transformed key block and transformed value block of the current denoising time slot.

[0319] 2519. In the first buffer, the transformed key blocks and transformed value blocks are concatenated with the transformed key blocks and transformed value blocks obtained by the diffusion processing unit in multiple denoising time slots before the current denoising time slot, respectively, to obtain the transformed key matrix and transformed value matrix;

[0320] 2520. Divide the transformed query sub-block, transformed key matrix, and transformed value matrix into transformed query blocks, transformed key blocks, and transformed value blocks corresponding to each attention head, and assign the transformed query blocks, transformed key blocks, and transformed value blocks to the corresponding attention heads;

[0321] 2521. In each attention head, attention is calculated based on the corresponding transformed query block, transformed key block, and transformed value block to obtain the first result sub-block;

[0322] 2522. In each diffusion processing unit, the first result sub-block is divided along the second dimension into a first number of result blocks corresponding to each diffusion processing unit in the parallel processing unit group;

[0323] 2523. Aggregate the first number of results corresponding to a single diffusion processing unit into the corresponding single diffusion processing unit, and concatenate the first number of results corresponding to a single diffusion processing unit along the first dimension to obtain the multi-head attention result sub-block corresponding to a single diffusion processing unit;

[0324] 2524. Based on the multi-head attention result sub-block, determine the image block in the next denoising time slot for the next diffusion processing unit in the corresponding diffusion processing unit group;

[0325] 2525. Determine the corresponding second subkey value and query sub-block based on the image block corresponding to the diffusion processing unit, and write the second subkey value into the first buffer of the diffusion processing unit;

[0326] 2526. Divide a single denoising time slot into multiple ordered sub-denoising time slots, execute the first process in each sub-denoising time slot, and while executing the first process, obtain the corresponding first sub-key value from other diffusion processing units in the parallel processing unit group and write it into the first buffer. The first process includes: performing multi-head attention processing based on the query sub-block, the key sub-block and value sub-block corresponding to the sub-denoising time slot, and obtaining the row normalized coefficient vector and attention intermediate result of each attention head in the sub-denoising time slot.

[0327] 2527. For each attention head, the row normalized coefficient vector and the intermediate attention result corresponding to each sub-denoising time slot of the attention head are accumulated respectively to obtain the corresponding target row normalized coefficient vector and the accumulated attention result. The accumulated attention result is corrected based on the target row normalized coefficient vector to obtain the attention processing result corresponding to the attention head.

[0328] 2528. Concatenate the attention processing results corresponding to each attention head to obtain a multi-head attention result sub-block, and determine the image block of the next diffusion processing unit in the next denoising time slot based on the multi-head attention result sub-block.

[0329] It is understandable that in practical applications, some steps in the above example are used selectively. For example, steps 2515 to 2524 and steps 2525 to 2528 are two implementations of obtaining the first subkey value to complete the model task during the execution of the model task. In practical applications, one of the two is chosen.

[0330] It is understood that although the steps in the above flowcharts are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated in this embodiment, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the above flowcharts may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps.

[0331] It should be noted that in various specific embodiments of this application, when processing is required based on data related to the characteristics of the target object, such as target object attribute information or a set of attribute information, the permission or consent of the target object will be obtained first. Furthermore, the collection, use, and processing of this data will comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require obtaining target object attribute information, separate permission or consent from the target object will be obtained through pop-ups or redirection to a confirmation page. Only after obtaining the target object's separate permission or consent will the necessary target object-related data for the normal operation of the embodiments of this application be obtained.

[0332] Description of apparatus and devices according to embodiments of this disclosure

[0333] Reference Figure 26 , Figure 26 This is a schematic diagram of the structure of the diffusion processing apparatus 2600 provided in an embodiment of the present disclosure. The diffusion processing apparatus 2600 includes:

[0334] The first slicing unit 2610 is used to divide the image to be diffused into a first number of image block groups and distribute them in parallel to the first number of diffusion processing unit groups for processing. Each diffusion processing unit group includes a second number of diffusion processing units. The diffusion processing method includes multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into an ordered second number of model tasks, which are executed in an ordered manner by the second number of diffusion processing units.

[0335] The diffusion execution unit 2620 is used to divide each denoising step into a third number of denoising time slots, and to perform the corresponding model task for each denoising time slot on the image block in each denoising time slot, wherein the third number is the number of image blocks in the image block group, and the image blocks in the image block group are ordered.

[0336] The first acquisition unit 2630 is used to acquire the first sub-key value when performing a model task on an image block with the same sequence number as the image block in other image block groups during the denoising time slot, and merge it with the second sub-key value when performing a model task on the image block to obtain the first key value, which is used for the execution of the model task on the image block.

[0337] Optionally, the diffusion execution unit 2620 is used for:

[0338] Based on the slot number of the denoising time slot, determine the sequence number of the image block to be started;

[0339] The first diffusion processing unit in the second number of diffusion processing units divides the image into blocks according to the image block number and executes the first model task.

[0340] In subsequent denoising time slots following the denoising time slot, other diffusion processing units divide the image into blocks according to the image block sequence number and perform other model tasks.

[0341] Optionally, the diffusion execution unit 2620 is used for:

[0342] For each diffusion processing unit, while executing the corresponding model task on the image block acquired in the current denoising time slot, the image block after the diffusion processing unit executed the corresponding model task in the previous denoising time slot is sent to the next diffusion processing unit in the corresponding diffusion processing unit group.

[0343] For each diffusion processing unit, while executing the corresponding model task on the image block obtained in the current denoising time slot, the image block that the diffusion processing unit needs to execute in the next denoising time slot is obtained from the previous diffusion processing unit in the corresponding diffusion processing unit group.

[0344] Optionally, in the first number of diffusion processing unit groups, the first number of diffusion processing units corresponding to the same model task constitute a parallel processing unit group. In a single denoising time slot, multiple diffusion processing units in a single parallel processing unit group execute the corresponding model task on the image blocks with the same index in each image block group. The diffusion execution unit 2620 is used for:

[0345] The first sub-block is determined based on the image blocks corresponding to the diffusion processing unit, and the first sub-block includes at least the second sub-key value corresponding to the diffusion processing unit;

[0346] The first sub-block of each diffusion processing unit in a single parallel processing unit group is transformed to obtain the transformed sub-block.

[0347] Multi-head attention processing is performed on the transformed sub-blocks to obtain the first result sub-block. Then, the first result sub-blocks of each diffusion processing unit in a single parallel processing unit group are subjected to sub-block inverse transformation to obtain the multi-head attention result sub-blocks of each diffusion processing unit.

[0348] Based on the multi-head attention result sub-block, determine the image block of the next diffusion processing unit in the next denoising time slot in the corresponding diffusion processing unit group.

[0349] Optionally, the diffusion execution unit 2620 is used for:

[0350] The first sub-block corresponding to each diffusion processing unit is divided along the first dimension into a first number of first blocks corresponding to a first number of diffusion processing units in the parallel processing unit group.

[0351] The first number of first blocks corresponding to a single diffusion processing unit are aggregated into the corresponding single diffusion processing unit, and the first number of first blocks corresponding to a single diffusion processing unit are spliced ​​along the second dimension to obtain the transformed sub-block.

[0352] Optionally, the diffusion execution unit 2620 is used for:

[0353] The query sub-block, key sub-block, and value sub-block are respectively divided into a first number of query blocks, key blocks, and value blocks corresponding to a first number of diffusion processing units in the parallel processing unit group;

[0354] The first number of query blocks, key blocks, and value blocks corresponding to a single diffusion processing unit are aggregated into a single diffusion processing unit, and the first number of query blocks, the second number of key blocks, and the second number of value blocks are concatenated to obtain transformed sub-blocks. The transformed sub-blocks include at least transformed query sub-blocks, transformed key blocks, and transformed value sub-blocks, wherein the transformed key blocks and transformed value sub-blocks constitute the first key-value pair.

[0355] Optionally, the diffusion execution unit 2620 is used for:

[0356] The transformed key blocks and transformed value blocks are written into the first buffer of the diffusion processing unit;

[0357] In the first buffer, the transformed key blocks and transformed value blocks are concatenated with the transformed key blocks and transformed value blocks obtained by the diffusion processing unit in multiple denoising time slots before the current denoising time slot, respectively, to obtain the transformed key matrix and transformed value matrix.

[0358] Optionally, the diffusion execution unit 2620 is used for:

[0359] Determine the buffer sub-region corresponding to the current denoising time slot;

[0360] If the transformed key block and transformed value block written in the previous denoising step exist in the cache sub-area corresponding to the current denoising time slot, the transformed key block and transformed value block in the cache sub-area corresponding to the current denoising time slot are updated using the transformed key block and transformed value block of the current denoising time slot.

[0361] Optionally, the diffusion execution unit 2620 is used for:

[0362] The transformed query sub-block, transformed key matrix, and transformed value matrix are divided into transformed query blocks, transformed key blocks, and transformed value blocks corresponding to each attention head, respectively, and the transformed query blocks, transformed key blocks, and transformed value blocks are assigned to the corresponding attention heads.

[0363] In each attention head, attention is calculated based on the corresponding transformed query block, transformed key block, and transformed value block.

[0364] Optionally, the diffusion execution unit 2620 is used for:

[0365] In each diffusion processing unit, the first result sub-block is divided along the second dimension into a first number of result blocks corresponding to each diffusion processing unit in the parallel processing unit group;

[0366] The first number of results corresponding to a single diffusion processing unit are aggregated into the corresponding single diffusion processing unit, and the first number of results corresponding to a single diffusion processing unit are spliced ​​along the first dimension to obtain the multi-head attention result sub-block corresponding to a single diffusion processing unit.

[0367] Optionally, the diffusion execution unit 2620 is used for:

[0368] The second subkey value and query sub-block are determined based on the image block corresponding to the diffusion processing unit, and the second subkey value is written into the first buffer of the diffusion processing unit.

[0369] A single denoising time slot is divided into multiple ordered sub-denoising time slots. A first process is executed in each sub-denoising time slot. While executing the first process, the corresponding first sub-key value is obtained from other diffusion processing units in the parallel processing unit group and written into the first buffer. The first process includes: performing multi-head attention processing on the key sub-block and value sub-block corresponding to the sub-denoising time slot based on the query sub-block, to obtain the row normalized coefficient vector and attention intermediate result of each attention head in the sub-denoising time slot; wherein, the key sub-block and value sub-block corresponding to the first sub-denoising time slot are the second sub-key value, and the key sub-block and value sub-block corresponding to each other sub-denoising time slot are the first sub-key value obtained in the previous sub-denoising time slot, or the first sub-key value obtained in multiple denoising time slots before the current denoising time slot;

[0370] For each attention head, the row normalized coefficient vector and the intermediate attention result corresponding to each sub-denoising time slot of the attention head are accumulated to obtain the corresponding target row normalized coefficient vector and the accumulated attention result. The accumulated attention result is then corrected based on the target row normalized coefficient vector to obtain the attention processing result corresponding to the attention head.

[0371] The attention processing results corresponding to each attention head are concatenated to obtain a multi-head attention result sub-block, and the image block of the next diffusion processing unit in the next denoising time slot is determined based on the multi-head attention result sub-block.

[0372] Optionally, the diffusion execution unit 2620 is used for:

[0373] The query sub-blocks and the key blocks and value sub-blocks corresponding to the sub-denoising time slots are divided into query blocks, key blocks and value blocks corresponding to each attention head and assigned to the corresponding attention heads;

[0374] For each attention head, the product matrix is ​​determined based on the corresponding query block and key block, and the product matrix is ​​indexed to obtain the indexed product matrix. The rows of the indexed product matrix are accumulated to obtain the row normalized coefficient vector corresponding to the attention head. The indexed product matrix and the corresponding value block are multiplied by matrix to obtain the attention intermediate result corresponding to the attention head.

[0375] Optionally, the diffusion processing apparatus 2600 further includes a number determination unit (not shown), the number determination unit being used for:

[0376] The first number is determined based on the number of other diffusion processing units connected to a single diffusion processing unit via a first communication method;

[0377] The second number is determined based on the first number and the number of diffusion processing units;

[0378] The first number of diffusion processing units connected via a first communication method are respectively assigned to the first number of diffusion processing unit groups, and the second number of diffusion processing units in a single diffusion processing unit group are connected via a second communication method.

[0379] Optionally, the diffusion processing apparatus 2600 further includes a model task segmentation unit (not shown), which is used for:

[0380] The network layers of the diffusion processing model are divided into an input preprocessing layer, a noise prediction layer, and an output processing layer. The noise prediction layer includes a fourth cascaded diffusion converter module.

[0381] Determine the first ratio between the fourth number and the second number, round the first ratio up to obtain the fifth number, and determine the first difference based on the product of the second number and the fifth number and the fourth number;

[0382] Obtain the total first parameter count of the input preprocessing layer and the total second parameter count of the output processing layer. Based on the ratio of the total first parameter count and the total second parameter count and the first difference, determine the second difference corresponding to the first model task and the third difference corresponding to the last model task. The sum of the second difference and the third difference is the first difference.

[0383] Based on the fifth number, the second difference, and the third difference, determine the number of the first layer and the number of the second layer, respectively.

[0384] The input preprocessing layer, the fourth number of diffusion converter modules, and the output processing layer are assigned to the second number of model tasks. The first model task consists of the input preprocessing layer and the first number of diffusion converter modules, the last model task consists of the second number of diffusion converter modules and the output processing layer, and each of the other model tasks consists of a fifth number of consecutive diffusion converter modules.

[0385] Reference Figure 27 , Figure 27 The structural block diagram of the target terminal 110 for implementing the diffusion processing method of this embodiment includes: a radio frequency (RF) circuit 2710, a memory 2715, an input unit 2730, a display unit 2740, a sensor 2750, an audio circuit 2760, a wireless fidelity (WiFi) module 2770, a processor 2780, and a power supply 2790, etc. Those skilled in the art will understand that... Figure 27 The illustrated terminal 110 structure does not constitute a limitation on a mobile phone or computer, and may include more or fewer components than illustrated, or combine certain components, or have different component arrangements.

[0386] The RF circuit 2710 can be used to receive and transmit signals during information transmission or calls. In particular, it receives downlink information from the base station and processes it with the processor 2780; in addition, it transmits uplink data to the base station.

[0387] The memory 2715 can be used to store software programs and modules, and the processor 2780 executes various functional applications and data processing of the content terminal by running the software programs and modules stored in the memory 2715.

[0388] The input unit 2730 can be used to receive input numeric or character information, and to generate key signal inputs related to the settings and function control of the content terminal. Specifically, the input unit 2730 may include a touch panel 2731 and other input devices 2732.

[0389] Display unit 2740 can be used to display input or provided information, as well as various menus of the content terminal. Display unit 2740 may include display panel 2741.

[0390] Audio circuitry 2760, speaker 2761, and microphone 2762 provide an audio interface.

[0391] In this embodiment, the processor 2780 included in the object terminal 110 can execute the diffusion processing method of the previous embodiment.

[0392] The target terminal 110 in this disclosure includes, but is not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, vehicle terminals, and aircraft. This invention can be applied to various scenarios, including but not limited to textural graphs and molecular graph modeling.

[0393] Figure 28 This is a partial structural block diagram of a server 140 implementing the diffusion processing method of this disclosure embodiment. The server 140 can vary significantly due to different configurations or performance characteristics, and may include one or more central processing units (CPUs) 2822 (e.g., one or more processors) and memory 2832, and one or more storage media 2830 (e.g., one or more mass storage devices) for storing application programs 2842 or data 2844. The memory 2832 and storage media 2830 may be temporary or persistent storage. The program stored in the storage media 2830 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the server. Furthermore, the CPU 2822 may be configured to communicate with the storage media 2830 and execute the series of instruction operations in the storage media 2830 on the server.

[0394] Server 140 may also include one or more power supplies 2833, one or more wired or wireless network interfaces 2850, one or more input / output interfaces 2858, and / or one or more operating systems 2841, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0395] The central processing unit 2822 in server 140 can be used to execute the diffusion processing method of the present disclosure embodiments.

[0396] This disclosure also provides a computer-readable storage medium for storing program code for executing the diffusion processing methods of the foregoing embodiments.

[0397] This disclosure also provides a computer program product comprising a computer program. A processor of a computer device reads and executes the computer program, causing the computer device to perform the diffusion processing method described above.

[0398] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar terms and are not necessarily used to describe a particular order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0399] It should be understood that in this disclosure, "at least one item" refers to one or more items, and "more than one item" refers to two or more items. "And / or" is used to describe the relationship between related content, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related content are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0400] It should be understood that in the description of the embodiments disclosed herein, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0401] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0402] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0403] In addition, in the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0404] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0405] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0406] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A diffusion treatment method, characterized in that, include: The image to be diffused is divided into a first number of image block groups and distributed in parallel to a first number of diffusion processing unit groups for processing. Each of the diffusion processing unit groups includes a second number of diffusion processing units. The diffusion processing method includes multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into an ordered second number of model tasks, which are executed in an ordered manner by the second number of diffusion processing units. Each denoising step is divided into a third number of denoising time slots, and the model task corresponding to the denoising time slot is executed on the image block in each denoising time slot, wherein the third number is the number of image blocks in the image block group, and the image blocks in the image block group are ordered; When the model task is executed on the image block in the denoising time slot, the first sub-key value when executing the model task on the same-order image block with the same sequence number as the image block in other image block groups is obtained, and merged with the second sub-key value when executing the model task on the image block to obtain the first key value, which is used for the execution of the model task on the image block.

2. The diffusion treatment method according to claim 1, characterized in that, include: The step of performing the model task corresponding to the denoising time slot for each denoising time slot on the image block includes: Based on the slot number of the denoising time slot, determine the sequence number of the image block to be started; The first diffusion processing unit among the second number of diffusion processing units performs the first model task on the image block of the image block number; In subsequent denoising time slots following the denoising time slot, other diffusion processing units divide the image into blocks according to the image block number and perform other model tasks.

3. The diffusion treatment method according to claim 2, characterized in that, The step of performing the model task corresponding to the denoising time slot for each denoising time slot of the image block further includes: For each diffusion processing unit, while performing the corresponding model task on the image block acquired in the current denoising time slot, the image block after the diffusion processing unit performed the corresponding model task in the previous denoising time slot of the current denoising time slot is sent to the next diffusion processing unit in the corresponding diffusion processing unit group. For each diffusion processing unit, while executing the corresponding model task on the image block obtained in the current denoising time slot, the image block that the diffusion processing unit needs to execute in the next denoising time slot is obtained from the previous diffusion processing unit in the corresponding diffusion processing unit group.

4. The diffusion treatment method according to claim 1, characterized in that, In the first number of diffusion processing unit groups, the first number of diffusion processing units corresponding to the same model task constitute a parallel processing unit group. In a single denoising time slot, multiple diffusion processing units in a single parallel processing unit group respectively execute the corresponding model task on the image blocks with the same index in each image block group. The model task includes: The first sub-block is determined based on the image blocks corresponding to the diffusion processing unit, and the first sub-block includes at least the second sub-key value corresponding to the diffusion processing unit; The first sub-block of each of the diffusion processing units in a single parallel processing unit group is transformed to obtain the transformed sub-block. Multi-head attention processing is performed on the transformed sub-blocks to obtain the first result sub-blocks, and the first result sub-blocks of each diffusion processing unit in a single parallel processing unit group are subjected to sub-block inverse transformation to obtain the multi-head attention result sub-blocks of each diffusion processing unit. Based on the multi-head attention result sub-block, determine the image block of the next diffusion processing unit in the next denoising time slot in the corresponding diffusion processing unit group.

5. The diffusion treatment method according to claim 4, characterized in that, The first sub-block includes a first dimension and a second dimension. The sub-block transformation of each of the diffusion processing units in a single parallel processing unit group to obtain the transformed sub-block includes: The first sub-block corresponding to each of the diffusion processing units is divided along the first dimension into a first number of first blocks corresponding to a first number of diffusion processing units in the parallel processing unit group; The first number of first blocks corresponding to a single diffusion processing unit are aggregated into the corresponding single diffusion processing unit, and the first number of first blocks corresponding to the single diffusion processing unit are spliced ​​along the second dimension to obtain the transformed sub-block.

6. The diffusion treatment method according to claim 5, characterized in that, The step of performing a sub-block inverse transformation on the first result sub-block of each of the diffusion processing units in a single parallel processing unit group to obtain the multi-head attention result sub-block of each diffusion processing unit includes: In each of the diffusion processing units, the first result sub-block is divided along the second dimension into a first number of result blocks corresponding to each diffusion processing unit in the parallel processing unit group; The first number of results corresponding to a single diffusion processing unit are aggregated into the corresponding single diffusion processing unit, and the first number of results corresponding to the single diffusion processing unit are spliced ​​along the first dimension to obtain the multi-head attention result sub-block corresponding to the single diffusion processing unit.

7. The diffusion treatment method according to claim 4, characterized in that, The first sub-block includes a query sub-block, a key sub-block, and a value sub-block; the second sub-key value includes the key sub-block and the value sub-block; the sub-block transformation of the sub-blocks of each of the diffusion processing units in a single parallel processing unit group to obtain the transformed sub-block includes: The query sub-block, the key block, and the value sub-block are respectively divided into a first number of query blocks, key blocks, and value blocks corresponding to a first number of diffusion processing units in the parallel processing unit group; The first number of query blocks, key blocks, and value blocks corresponding to a single diffusion processing unit are aggregated into the single diffusion processing unit, and the first number of query blocks, the second number of key blocks, and the second number of value blocks are concatenated to obtain the transformed sub-block. The transformed sub-block includes at least a transformed query sub-block, a transformed key block, and a transformed value sub-block, wherein the transformed key block and the transformed value sub-block constitute the first key value.

8. The diffusion treatment method according to claim 7, characterized in that, After performing sub-block transformation on the sub-blocks of each of the diffusion processing units in the single parallel processing unit group to obtain transformed sub-blocks, the diffusion processing method further includes: The transformed key sub-block and the transformed value sub-block are written into the first buffer of the diffusion processing unit; In the first buffer, the transformed key block and the transformed value block are concatenated with the transformed key block and transformed value block obtained by the diffusion processing unit in multiple denoising time slots before the current denoising time slot to obtain the transformed key matrix and the transformed value matrix.

9. The diffusion treatment method according to claim 8, characterized in that, The first buffer includes a buffer sub-region corresponding to each denoising slot in a single diffusion step. The step of writing the transformed key sub-block and the transformed value sub-block into the first buffer of the diffusion processing unit includes: Determine the buffer sub-region corresponding to the current denoising time slot; If the transformed key block and transformed value block written in the previous denoising step exist in the cache sub-region corresponding to the current denoising time slot, the transformed key block and transformed value block in the cache sub-region corresponding to the current denoising time slot are updated using the transformed key block and transformed value block of the current denoising time slot.

10. The diffusion treatment method according to claim 8, characterized in that, The model task includes multiple attention heads, and the multi-head attention processing based on the transformed sub-blocks includes: The transformed query sub-block, the transformed key matrix, and the transformed value matrix are respectively divided into transformed query blocks, transformed key blocks, and transformed value blocks corresponding to each attention head, and the transformed query blocks, the transformed key blocks, and the transformed value blocks are assigned to the corresponding attention heads; In each of the attention heads, attention calculation is performed based on the corresponding transformed query block, transformed key block, and transformed value block.

11. The diffusion treatment method according to claim 2, characterized in that, In the first number of diffusion processing unit groups, the first number of diffusion processing units corresponding to the same model task constitute a parallel processing unit group. In a single denoising time slot, multiple diffusion processing units in the same parallel processing unit group execute the corresponding model task on the image blocks with the same index in each image block group. The model task includes: The second subkey value and query sub-block are determined based on the image block corresponding to the diffusion processing unit, and the second subkey value is written into the first buffer of the diffusion processing unit. A single denoising time slot is divided into multiple ordered sub-denoising time slots. A first process is executed in each sub-denoising time slot. While executing the first process, the corresponding first sub-key value is obtained from other diffusion processing units in the parallel processing unit group and written into the first buffer. The first process includes: performing multi-head attention processing on the key sub-block and value sub-block corresponding to the sub-denoising time slot based on the query sub-block, to obtain the row normalization coefficient vector and attention intermediate result of each attention head corresponding to the sub-denoising time slot; wherein, the key sub-block and value sub-block corresponding to the first sub-denoising time slot are the second sub-key value, and the key sub-blocks and value sub-blocks corresponding to the other sub-denoising time slots are the first sub-key values ​​obtained in the previous sub-denoising time slot, or the first sub-key values ​​obtained in multiple denoising time slots before the current denoising time slot. For each attention head, the row normalization coefficient vector and the intermediate attention result corresponding to each sub-denoising time slot of the attention head are accumulated respectively to obtain the corresponding target row normalization coefficient vector and the accumulated attention result. The accumulated attention result is then corrected based on the target row normalization coefficient vector to obtain the attention processing result corresponding to the attention head. The attention processing results corresponding to each attention head are concatenated to obtain a multi-head attention result sub-block, and the image block of the next diffusion processing unit in the next denoising time slot is determined based on the multi-head attention result sub-block.

12. The diffusion treatment method according to claim 11, characterized in that, Based on the query sub-block, the key block and value sub-block corresponding to the sub-denoising time slot undergo multi-head attention processing to obtain the row normalized coefficient vector and attention intermediate results corresponding to the sub-denoising time slot, including: The query sub-block, the key block and the value sub-block corresponding to the sub-denoising time slot are divided into query blocks, key blocks and value blocks corresponding to each attention head and assigned to the corresponding attention head; For each attention head, a product matrix is ​​determined based on the corresponding query block and key block, and the product matrix is ​​indexed to obtain an indexed product matrix; the rows of the indexed product matrix are summed to obtain the row normalized coefficient vector corresponding to the attention head; the indexed product matrix and the corresponding value block are multiplied by matrix to obtain the attention intermediate result corresponding to the attention head.

13. The diffusion treatment method according to claim 1, characterized in that, The diffusion processing unit is connected to other diffusion processing units via a first communication method or a second communication method, wherein the bandwidth of the first communication method is greater than the bandwidth of the second communication method, and the diffusion processing method further includes: The first number is determined based on the number of other diffusion processing units connected to a single diffusion processing unit via a first communication method; The second number is determined based on the first number and the number of diffusion processing units; A first number of diffusion processing units connected via a first communication method are respectively assigned to the first number of diffusion processing unit groups, and a second number of diffusion processing units in a single diffusion processing unit group are connected via a second communication method.

14. The diffusion treatment method according to claim 1, characterized in that, The diffusion treatment method further includes: The network layers of the diffusion processing model are divided into an input preprocessing layer, a noise prediction layer, and an output processing layer. The noise prediction layer includes a fourth number of cascaded diffusion converter modules. Determine a first ratio between the fourth number and the second number, round the first ratio up to obtain a fifth number, and determine a first difference based on the product of the second number and the fifth number and the fourth number; Obtain the total first parameter count of the input preprocessing layer and the total second parameter count of the output processing layer, and determine the second difference corresponding to the first model task and the third difference corresponding to the last model task based on the ratio of the total first parameter count to the total second parameter count and the first difference, wherein the sum of the second difference and the third difference is the first difference; Based on the fifth number, the second difference, and the third difference, the number of the first layer and the number of the second layer are determined respectively; The input preprocessing layer, the fourth number of diffusion converter modules, and the output processing layer are assigned to the second number of model tasks. The first model task consists of the input preprocessing layer and the first number of diffusion converter modules, the last model task consists of the second number of diffusion converter modules and the output processing layer, and each of the other model tasks consists of a fifth number of consecutive diffusion converter modules.

15. A diffusion treatment apparatus, characterized in that, The device includes: The first slicing unit is used to divide the image to be diffused into a first number of image block groups and distribute them in parallel to the first number of diffusion processing unit groups for processing. Each of the diffusion processing unit groups includes a second number of diffusion processing units. The diffusion processing method includes multiple denoising steps. In each denoising step, a diffusion processing model is executed. The diffusion processing model is divided into an ordered second number of model tasks, which are executed in an ordered manner by the second number of diffusion processing units. A diffusion execution unit is configured to divide each denoising step into a third number of denoising time slots, and execute the model task corresponding to the denoising time slot on the image block in each denoising time slot, wherein the third number is the number of image blocks in the image block group, and the image blocks in the image block group are ordered; The first acquisition unit is configured to, when performing the model task on the image block in the denoising time slot, acquire a first sub-key value when performing the model task on a sequential image block with the same index as the image block in other image block groups, and merge it with a second sub-key value when performing the model task on the image block to obtain a first key value, which is used for the execution of the model task on the image block.

16. An electronic device comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, it implements the diffusion processing method according to any one of claims 1 to 14.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the diffusion processing method according to any one of claims 1 to 14.

18. A computer program product comprising a computer program that is read and executed by a processor of a computer device, causing the computer device to perform the diffusion processing method according to any one of claims 1 to 14.

Images