Thread group-based operator fusion method and device, equipment, medium and product

Abstract

Embodiments of the present disclosure provide a thread group based operator fusion method and device, equipment, medium and product, and relate to the technical field of parallel computing, deep learning inference and training optimization. The method comprises: in response to the range of the output data block of the first thread group of the first operator containing the range of the input data block of the second thread group of the second operator, fusing the calculation logic of the first thread group and the calculation logic of the second thread group to generate first fused calculation logic; and executing the first fused calculation logic by the first fused thread group, wherein the first local memory allocated for the first fused thread group stores the intermediate result of the first fused calculation logic.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the fields of parallel computing, deep learning inference and training optimization, and in particular to operator fusion methods and apparatus based on thread groups, electronic devices, readable storage media and computer program products. Background Technology

[0002] In parallel computing fields such as graphics rendering, scientific computing, and the rapidly developing deep learning inference and training, computational tasks can be organized as data flow graphs consisting of a large number of computational task units (hereinafter referred to as "operators"). To reduce the global access overhead and task scheduling latency caused by data transfer between operators, and to improve hardware utilization, fusion techniques that combine multiple operators that meet certain conditions into fusion operators have become an optimization method in the field of parallel computing.

[0003] In some operator fusion schemes that focus on operator-level fusion and thread-level fusion, when faced with a computing model that uses thread groups as the execution unit and data blocks as the read and write carrier, it is difficult to effectively identify and utilize the data production and consumption relationships between thread groups. This results in limited fusion opportunities, high global access overhead, and restrictions on further improvement of hardware computing performance. Summary of the Invention

[0004] Embodiments of this disclosure provide a thread group-based operator fusion method and apparatus, electronic device, readable storage medium, and computer program product that can at least partially solve the above-described problems or other problems in the art.

[0005] According to a first aspect of this disclosure, an operator fusion method based on thread groups is provided, the method comprising: fusion of computational logic of the first thread group and computational logic of the second thread group in response to the range of the output data block of the first thread group containing the range of the input data block of the second thread group containing the second operator, generating first fused computational logic; and execution of the first fused computational logic by the first fused thread group, wherein a first local memory allocated to the first fused thread group stores intermediate results of the first fused computational logic.

[0006] In some embodiments of this disclosure, the execution of the first fusion computing logic by the first fusion thread group includes: executing the computing logic corresponding to the first thread group in the first fusion computing logic to generate a first intermediate result; storing the first intermediate result in a first local memory allocated to the first fusion thread group; and reading the first intermediate result from the first local memory and executing the computing logic corresponding to the second thread group in the first fusion computing logic based on the first intermediate result.

[0007] In some embodiments of this disclosure, a first fusion thread group is executed by a stream processor, and a first local memory is allocated to the first fusion thread group by the stream processor.

[0008] In some embodiments of this disclosure, the method further includes: determining the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group, including: determining the inclusion relationship based on the configuration information of the first thread group and the second thread group; or, determining the inclusion relationship based on the running record information of the first thread group and the second thread group.

[0009] In some embodiments of this disclosure, determining the inclusion relationship based on the configuration information of the first thread group and the second thread group includes: determining the first coordinate range of the output data block of the first thread group in the output tensor of the first operator; determining the second coordinate range of the input data block of the second thread group in the output tensor; and in response to the first coordinate range including the second coordinate range, merging the computational logic of the first thread group and the computational logic of the second thread group to generate the first fused computational logic.

[0010] In some embodiments of this disclosure, determining the first coordinate range of the output data block of the first thread group in the output tensor of the first operator includes: determining the first coordinate range according to the logical identifier of the first thread group and the output data layout of the first operator, wherein the output data layout of the first operator includes at least one of the shape of the output tensor, the dimensional order of the output tensor, and the memory arrangement format of the output tensor.

[0011] In some embodiments of this disclosure, determining the second coordinate range of the input data block of the second thread group in the output tensor includes: determining the third coordinate range of the input data block of the second thread group in the input tensor of the second operator based on the logical identifier of the second thread group and the input data layout of the second operator; and mapping the third coordinate range to the output tensor based on the data flow relationship between the first operator and the second operator to determine the second coordinate range, wherein the input data layout of the second operator includes at least one of the following: the shape of the input tensor, the dimensional order of the input tensor, and the memory arrangement format of the input tensor.

[0012] In some embodiments of this disclosure, determining the inclusion relationship based on the runtime record information of the first thread group and the second thread group includes: obtaining a first tag and a second tag from historical execution data, wherein the first tag at least indicates the logical identifier of the first thread group and the range of the output data block of the first thread group, and the second tag at least indicates the logical identifier of the second thread group and the range of the input data block of the second thread group; and determining the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group based on the first tag and the second tag.

[0013] In some embodiments of this disclosure, determining the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group based on the first label and the second label includes: determining a first coordinate range of the output data block of the first thread group in the output tensor of the first operator based on the first label; determining a third coordinate range of the input data block of the second thread group in the input tensor of the second operator based on the second label; mapping the third coordinate range to the output tensor based on the data flow relationship between the first operator and the second operator to determine a second coordinate range of the input data block of the second thread group in the output tensor; and determining that the range of the output data block of the first thread group includes the range of the input data block of the second thread group based on the first coordinate range including the second coordinate range.

[0014] In some embodiments of this disclosure, the method further includes: estimating the storage capacity required to fuse the first fusion computing logic with the computing logic of the third thread group in response to the range of the output data block of the third thread group of the third operator including the range of the input data block of the first thread group; and fusion of the first fusion computing logic with the computing logic of the third thread group in response to the storage capacity being equal to or less than the capacity of the first local memory, so as to update the first fusion computing logic.

[0015] In some embodiments of this disclosure, the method further includes: in response to the range of the output data block of the Mth thread group including the range of the input data block of the Nth thread group, estimating the storage capacity required to merge the current first fusion computing logic with the computing logic of the Mth thread group, wherein the current first fusion computing logic includes the computing logic of the Nth thread group, and the input of the Nth thread group is the input of the current first fusion computing logic; and in response to the storage capacity being equal to or less than the capacity of the first local memory, merging the computing logic of the Mth thread group with the current first fusion computing logic to update the first fusion computing logic, wherein M and N are different and are both positive integers greater than 2.

[0016] In some embodiments of this disclosure, the method further includes: executing updated first fusion computing logic by a first fusion thread group, including: executing computing logic corresponding to a third thread group in the updated first fusion computing logic to generate a second intermediate result; and storing the second intermediate result in a first local memory, wherein, based on the second intermediate result read from the first local memory, the computing logic corresponding to the first thread group in the first fusion computing logic is executed.

[0017] In some embodiments of this disclosure, the method further includes: estimating the storage capacity required to fuse the first fusion computing logic with the computing logic of the third thread group in response to the range of the output data block of the third thread group including the range of the input data block of the first thread group; and storing the result of executing the computing logic of the first fusion computing logic corresponding to the second thread group in a global memory different from the first local memory in response to the storage capacity being greater than the capacity of the first local memory.

[0018] In some embodiments of this disclosure, the method further includes: executing a first fusion thread group by a stream processor, and allocating a first local memory for the first fusion thread group by the stream processor, wherein the global memory is accessed by multiple stream processors.

[0019] In some embodiments of this disclosure, the method further includes: estimating the storage capacity required to fuse the first fusion computing logic with the computing logic of the fourth thread group in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator; and fusion of the first fusion computing logic with the computing logic of the fourth thread group in response to the storage capacity being equal to or less than the capacity of the first local memory, so as to update the first fusion computing logic.

[0020] In some embodiments of this disclosure, the method further includes: executing updated first fusion computing logic by a first fusion thread group, including: storing a third intermediate result generated by executing the computing logic corresponding to the second thread group in the first fusion computing logic to a first local memory; and reading the third intermediate result from the first local memory and executing the computing logic corresponding to the fourth thread group in the first fusion computing logic based on the third intermediate result.

[0021] In some embodiments of this disclosure, the method further includes: estimating the storage capacity required to fuse the current first fusion computing logic with the computing logic of the Q thread group in response to the range of the output data block of the P thread group including the range of the input data block of the Q thread group, wherein the current first fusion computing logic includes the computing logic of the P thread group, and the output of the P thread group is the output of the current first fusion computing logic; and fusion of the computing logic of the Q thread group with the current first fusion computing logic in response to the storage capacity being equal to or less than the capacity of the first local memory, to update the first fusion computing logic, wherein P and Q are different and are both positive integers greater than 2.

[0022] In some embodiments of this disclosure, the method further includes: in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator, estimating the storage capacity required to fuse the first fusion computing logic with the computing logic of the fourth thread group; and in response to the storage capacity being greater than the capacity of the first local memory, storing the result generated by executing the computing logic of the first fusion computing logic corresponding to the second thread group to a global memory different from the first local memory.

[0023] In some embodiments of this disclosure, the method further includes: storing a first intermediate result in a first storage space of a first local memory; and after the first intermediate result is read, storing a third intermediate result in the first storage space to overwrite the first intermediate result.

[0024] In some embodiments of this disclosure, the method further includes: determining the size of the input data block of the first thread group; determining the data reuse rate based on the ratio between the size of the input data block of the first thread group and the size of the input data block of the second thread group; and, in response to the data reuse rate being greater than or equal to a predetermined threshold, merging the computational logic of the first thread group and the computational logic of the second thread group to generate a first fused computational logic.

[0025] In some embodiments of this disclosure, the method further includes: in response to the range of the output data block of the fifth thread group of the first operator including the range of the input data block of the sixth thread group of the fifth operator, fusing the computational logic of the fifth thread group and the computational logic of the sixth thread group to generate a second fused computational logic; and executing the second fused computational logic by the second fused thread group, including: executing the computational logic corresponding to the fifth thread group in the second fused computational logic to generate a fourth intermediate result; storing the fourth intermediate result in a second local memory allocated to the second fused thread group; and reading the fourth intermediate result from the second local memory and executing the computational logic corresponding to the sixth thread group in the second fused computational logic based on the fourth intermediate result.

[0026] In some embodiments of this disclosure, the method further includes: executing a first fusion thread group and a second fusion thread group by the same stream processor, and allocating a first local memory and a second local memory to the first fusion thread group and the second fusion thread group by the same stream processor, respectively.

[0027] According to a second aspect of this disclosure, an operator fusion apparatus based on thread groups is provided. The apparatus includes: a fusion unit configured to, in response to a first operator, fuse the computational logic of the first thread group with the computational logic of the second thread group to generate first fused computational logic, wherein the range of the output data block of the first thread group includes the range of the input data block of the second thread group of the second operator; and a computation unit configured to execute the first fused computational logic by the first fused thread group, wherein a first local memory allocated to the first fused thread group stores intermediate results of the first fused computational logic.

[0028] In some embodiments of this disclosure, the fusion unit is further configured to: execute the computation logic corresponding to the first thread group in the first fusion computation logic to generate a first intermediate result; store the first intermediate result in a first local memory allocated to the first fusion thread group; and read the first intermediate result from the first local memory and execute the computation logic corresponding to the second thread group in the first fusion computation logic based on the first intermediate result.

[0029] In some embodiments of this disclosure, a first fusion thread group is executed by a stream processor, and a first local memory is allocated to the first fusion thread group by the stream processor.

[0030] In some embodiments of this disclosure, the fusion unit is further configured to: determine the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group, including: determining the inclusion relationship based on the configuration information of the first thread group and the second thread group; or, determining the inclusion relationship based on the running record information of the first thread group and the second thread group.

[0031] In some embodiments of this disclosure, the fusion unit is further configured to: determine a first coordinate range of the output data block of the first thread group in the output tensor of the first operator; determine a second coordinate range of the input data block of the second thread group in the output tensor; and determine an inclusion relationship based on the first coordinate range and the second coordinate range.

[0032] In some embodiments of this disclosure, the fusion unit is further configured to: determine a first coordinate range based on the logical identifier of the first thread group and the output data layout of the first operator, wherein the output data layout of the first operator includes at least one of the following: the shape of the output tensor, the dimensional order of the output tensor, and the memory arrangement format of the output tensor.

[0033] In some embodiments of this disclosure, the fusion unit is further configured to: determine the third coordinate range of the input data block of the second thread group in the input tensor of the second operator based on the logical identifier of the second thread group and the input data layout of the second operator; and map the third coordinate range to the output tensor based on the data flow relationship between the first operator and the second operator to determine the second coordinate range, wherein the input data layout of the second operator includes at least one of the following: the shape of the input tensor, the dimensional order of the input tensor, and the memory arrangement format of the input tensor.

[0034] In some embodiments of this disclosure, the fusion unit is further configured to: obtain a first tag and a second tag from historical execution data, wherein the first tag at least indicates the logical identifier of a first thread group and the range of the output data block of the first thread group, and the second tag at least indicates the logical identifier of a second thread group and the range of the input data block of the second thread group; and determine, based on the first tag and the second tag, the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group.

[0035] In some embodiments of this disclosure, the fusion unit is further configured to: determine a first coordinate range in the output tensor of the first operator based on a first label; determine a third coordinate range in the input tensor of the second operator based on a second label; map the third coordinate range to the output tensor based on the data flow relationship between the first and second operators to determine a second coordinate range in the output tensor of the second thread group; and determine that the range of the output data block of the first thread group includes the range of the input data block of the second thread group based on the fact that the first coordinate range includes the second coordinate range.

[0036] In some embodiments of this disclosure, the fusion unit is further configured to: estimate the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the third thread group in response to the range of the output data block of the third thread group of the third operator including the range of the input data block of the first thread group; and fuse the first fusion calculation logic with the calculation logic of the third thread group in response to the storage capacity being equal to or less than the capacity of the first local memory, so as to update the first fusion calculation logic.

[0037] In some embodiments of this disclosure, the fusion unit is further configured to: estimate the storage capacity required to fuse the current first fusion calculation logic with the calculation logic of the Mth thread group in response to the fact that the range of the output data block of the Mth thread group includes the range of the input data block of the Nth thread group, wherein the current first fusion calculation logic includes the calculation logic of the Nth thread group, and the input of the Nth thread group is the input of the current first fusion calculation logic; and fuse the calculation logic of the Mth thread group with the current first fusion calculation logic in response to the fact that the storage capacity is equal to or less than the capacity of the first local memory, so as to update the first fusion calculation logic, wherein M and N are different and are both positive integers greater than 2.

[0038] In some embodiments of this disclosure, the computing unit is further configured to: execute updated first fusion computing logic by a first fusion thread group, including: executing computing logic corresponding to a third thread group in the updated first fusion computing logic to generate a second intermediate result; and storing the second intermediate result in a first local memory, wherein, based on the second intermediate result read from the first local memory, the computing logic corresponding to the first thread group in the first fusion computing logic is executed.

[0039] In some embodiments of this disclosure, the fusion unit is further configured to: estimate the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the third thread group in response to the range of the output data block of the third thread group including the range of the input data block of the first thread group; and store the result generated by executing the calculation logic of the first fusion calculation logic corresponding to the second thread group in a global memory different from the first local memory in response to the storage capacity being greater than the capacity of the first local memory.

[0040] In some embodiments of this disclosure, a first fusion thread group is executed by a stream processor, and a first local memory is allocated to the first fusion thread group by the stream processor, wherein the global memory can be accessed by multiple stream processors.

[0041] In some embodiments of this disclosure, the fusion unit is further configured to: estimate the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the fourth thread group in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator; and fuse the first fusion calculation logic with the calculation logic of the fourth thread group in response to the storage capacity being equal to or less than the capacity of the first local memory, so as to update the first fusion calculation logic.

[0042] In some embodiments of this disclosure, the computing unit is further configured to: execute the updated first fusion computing logic by the first fusion thread group, including: storing the third intermediate result generated by executing the computing logic corresponding to the second thread group in the first fusion computing logic to the first local memory; and reading the third intermediate result from the first local memory, and executing the computing logic corresponding to the fourth thread group in the first fusion computing logic based on the third intermediate result.

[0043] In some embodiments of this disclosure, the fusion unit is further configured to: estimate the storage capacity required to fuse the current first fusion calculation logic with the calculation logic of the Q thread group in response to the range of the output data block of the P thread group including the range of the input data block of the Q thread group, wherein the current first fusion calculation logic includes the calculation logic of the P thread group and the output of the P thread group is the output of the current first fusion calculation logic; and fuse the calculation logic of the Q thread group with the current first fusion calculation logic in response to the storage capacity being equal to or less than the capacity of the first local memory, to update the first fusion calculation logic, wherein P and Q are different and are both positive integers greater than 2.

[0044] In some embodiments of this disclosure, the fusion unit is further configured to: estimate the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the fourth thread group in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator; and store the result generated by executing the calculation logic of the first fusion calculation logic corresponding to the second thread group in a global memory different from the first local memory in response to the storage capacity being greater than the capacity of the first local memory.

[0045] In some embodiments of this disclosure, the computing unit is further configured to: store a first intermediate result in a first storage space of a first local memory; and after the first intermediate result is read, store a third intermediate result in the first storage space to overwrite the first intermediate result.

[0046] In some embodiments of this disclosure, the fusion unit is further configured to: determine the size of the input data block of the first thread group; determine the data reuse rate based on the ratio between the size of the input data block of the first thread group and the size of the input data block of the second thread group; and, in response to the data reuse rate being greater than or equal to a predetermined threshold, fuse the computational logic of the first thread group and the computational logic of the second thread group to generate the first fused computational logic.

[0047] In some embodiments of this disclosure, the fusion unit is further configured to: in response to the range of the output data block of the fifth thread group of the first operator including the range of the input data block of the sixth thread group of the fifth operator, fuse the computational logic of the fifth thread group and the computational logic of the sixth thread group to generate a second fused computational logic; and the computational unit is further configured to: execute the second fused computational logic by the second fused thread group, including: executing the computational logic corresponding to the fifth thread group in the second fused computational logic to generate a fourth intermediate result; storing the fourth intermediate result in a second local memory allocated to the second fused thread group; and reading the fourth intermediate result from the second local memory and executing the computational logic corresponding to the sixth thread group in the second fused computational logic based on the fourth intermediate result.

[0048] In some embodiments of this disclosure, the first fusion thread group and the second fusion thread group are executed by the same stream processor, and the first fusion thread group and the second fusion thread group are allocated first local memory and second local memory respectively by the same stream processor.

[0049] According to a third aspect of this disclosure, an electronic device is provided, the electronic device including a processor that can be used to implement the thread group-based operator fusion method of the first aspect and any possible implementation thereof.

[0050] According to a fourth aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the thread group-based operator fusion method of the first aspect and any possible implementation thereof.

[0051] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the thread group-based operator fusion method in the first aspect and any possible implementation thereof.

[0052] The operator fusion method, apparatus, electronic device, readable storage medium, and computer program product based on thread groups provided by the embodiments of this disclosure, in response to the range of the output data block of the first thread group of the first operator including the range of the input data block of the second thread group of the second operator, fuses the computational logic of the first thread group and the computational logic of the second thread group to generate fused computational logic. This integrates thread groups that originally belonged to different operators and required independent scheduling and execution into the same fused thread group for sequential execution, thus eliminating the synchronization caused by scheduling switching between operators and improving the continuity of task execution. In addition, during the execution of the fused thread group, the intermediate results generated in the preceding calculation stage are stored in the local memory allocated to the fused thread group, and can be directly read from it in subsequent calculation stages. This mechanism effectively avoids global (memory, video memory) access overhead and reduces the power consumption of data transmission from a higher global access level to a lower local access level. Therefore, the fusion scheme provided by the embodiments of this disclosure, with thread groups as the fusion granularity, improves the continuous utilization of hardware computing resources and enhances the overall execution efficiency and energy efficiency of computing tasks by reducing scheduling and data transmission overhead through logical integration and localized data transmission.

[0053] It should be understood that the description in this section is not intended to identify key or important features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0054] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0055] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a flowchart of an operator fusion method based on thread groups provided according to an exemplary embodiment of the present disclosure; Figure 2 This is a schematic diagram of the process of the operator fusion method based on thread groups provided according to an exemplary embodiment of the present disclosure; Figures 3 to 5 These are flowcharts of the operator fusion method based on thread groups provided according to exemplary embodiments of this disclosure; Figure 6 This is a schematic diagram of the process of the operator fusion method based on thread groups provided according to an exemplary embodiment of the present disclosure; Figure 7 This is a flowchart of an operator fusion method based on thread groups provided according to an exemplary embodiment of the present disclosure; Figures 8 to 10 These are schematic diagrams illustrating the process of the operator fusion method based on thread groups according to exemplary embodiments of this disclosure; Figure 11 This is a block diagram of an operator fusion apparatus based on thread groups provided according to an exemplary embodiment of the present disclosure; Figure 12 This is a schematic block diagram of an electronic device provided according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0056] The various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0057] The term “exemplary” as used herein means “serving as an example, implementation method, or illustration.” Any implementation method described herein as “exemplary” is not necessarily to be construed as superior to or better than other implementation methods.

[0058] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, apparatuses, means, elements, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0059] Some embodiments of this disclosure provide a thread group-based operator fusion method. Figure 1 This is a flowchart of an operator fusion method 1000 based on thread groups provided according to an exemplary embodiment of the present disclosure.

[0060] like Figure 1 As shown, the operator fusion method 1000 based on thread groups includes: Step S1: In response to the fact that the range of the output data block of the first thread group of the first operator includes the range of the input data block of the second thread group of the second operator, the computational logic of the first thread group and the computational logic of the second thread group are merged to generate the first fused computational logic.

[0061] Step S2: The first fusion thread group executes the first fusion calculation logic, wherein the first local memory allocated to the first fusion thread group stores the intermediate results of the first fusion calculation logic.

[0062] Taking the field of AI (Artificial Intelligence) as an example, with the rapid growth in the scale of AI models, the number of operators they contain is enormous and the computations are tedious. Therefore, in the optimization of inference using AI models, the use of fusion technology can save computing power and power consumption and improve hardware utilization. Fusion technology can be understood as the optimization technology used by parallel computing devices or the compiler in such devices. It aims to merge multiple operators that meet specific conditions into a fusion operator. The fusion operator (or fusion kernel function) encapsulates the computational logic of the above multiple operators and completes them continuously in one execution, thereby reducing the global access overhead and task scheduling delay caused by data transfer between operators and improving hardware utilization.

[0063] Specifically, an operator can be understood as an independent and functionally defined computational task unit executed on a parallel computing device such as a GPU (Graphics Processing Unit). In the field of parallel computing, computational tasks can be organized as a data flow graph composed of a large number of operators. Alternatively, an operator can encapsulate specific computational logic to perform predetermined mathematical and / or logical operations on the operator's input data block and produce a corresponding output data block. This computational logic may include convolution, pooling, matrix multiplication, activation functions, etc. Optionally, an operator can be compiled or mapped to one or more kernel functions that execute on hardware. A kernel function is a program instance that is started and executed on a parallel computing device, responsible for scheduling and organizing a large amount of computing resources to complete the computational task defined by its corresponding operator.

[0064] In architectures such as GPUs, a thread is the smallest schedulable unit of execution on a parallel computing device. A thread can independently execute the instruction stream of a kernel function. Multiple threads are organized into thread groups, which are the basic units for task scheduling, resource allocation, and inter-thread cooperation on parallel computing devices. Multiple threads within the same thread group can be scheduled concurrently and cooperate by sharing the local memory allocated to the thread group and executing synchronization instructions, thereby jointly processing a block of data.

[0065] In other words, an operator can execute its computational tasks in parallel by launching one or more thread groups, where each thread group is responsible for processing and computing a corresponding data block, and the complete output of the operator can be composed of the output data blocks computed by all its thread groups. A data block can be understood as a collection of input or output data processed by a single thread group. Optionally, a data block corresponds to a continuous or discontinuous range of coordinates in the operator's overall data tensor, and the complete input or output of the operator can be composed of the data blocks processed by all the thread groups launched by the operator.

[0066] In some operator fusion schemes that focus on operator-level fusion and thread-level fusion, when faced with a computing mode that uses thread groups as the execution unit and data blocks as the read and write carrier, it is difficult to effectively identify and utilize the input-output dependencies formed between thread groups based on data blocks. This results in limited fusion opportunities, high memory access overhead, and restrictions on further improvement of hardware computing performance.

[0067] Taking thread-level fusion as an example, thread-level fusion attempts to analyze and merge the computational logic of individual threads in different operators with finer granularity, aiming to achieve more flexible utilization of data locality. However, in parallel computing architectures, the computational tasks of operators are completed collaboratively by thread groups. The input and output of operators are logically treated as a whole, with each thread processing only a part of the data block. Therefore, thread-level analysis cannot reflect the data block dependencies between thread groups as a whole, making it difficult to establish accurate cross-operator data matching. Fusion analysis is prone to failure or inability to generate correct code.

[0068] Taking operator-level fusion as an example, operator-level fusion requires that the overall output tensor of the predecessor operator and the overall input-output tensor of the successor operator be completely matched in terms of data layout and shape in order to merge them into a single fusion operator. However, in actual parallel computing, the computation task of an operator is completed collaboratively by multiple thread groups, each of which is independently responsible for loading the input and generating the output of a data block. Data dependencies often exhibit locality; for example, the data block output by only one thread group in the predecessor operator may cover the input data block required by one thread group in the subsequent operator. This overall alignment fusion condition in operator-level fusion schemes is too strict, preventing a large number of operator pairs with opportunities for local data reuse from being fused.

[0069] To at least address the aforementioned problems, embodiments of this disclosure provide an operator fusion method based on thread groups, wherein the analysis and execution granularity of the fusion scheme is the thread group and the data block processed by the thread group. Specifically, the operator fusion method 1000, in response to the range of the output data block of the first thread group of the first operator including the range of the input data block of the second thread group of the second operator, fuses the computational logic of the first thread group with the computational logic of the second thread group to generate a first fused computational logic; and the first fused thread group executes the first fused computational logic, wherein a first local memory allocated to the first fused thread group stores the intermediate results of the first fused computational logic.

[0070] For example, the execution of the first fusion computing logic by the first fusion thread group may include: executing the computing logic corresponding to the first thread group in the first fusion computing logic to generate a first intermediate result; storing the first intermediate result in a first local memory allocated to the first fusion thread group; and reading the first intermediate result from the first local memory and executing the computing logic corresponding to the second thread group in the first fusion computing logic based on the first intermediate result.

[0071] Therefore, the thread group-based operator fusion method 1000 uses thread groups as execution units and the data blocks they process as analysis objects to accurately examine the data dependencies between operators. Responding to the fact that the output data block of the first thread group of the first operator includes the input data block of the second thread group of the second operator, the computational logic of the first thread group and the computational logic of the second thread group are fused to generate fused computational logic. By analyzing and matching the input-output dependencies formed between thread groups based on data blocks, it can effectively identify opportunities for reusing local data in operators that are ignored or impossible to achieve by operator-level fusion schemes or thread-level fusion schemes, thus expanding the fusion scenarios. Furthermore, by integrating thread groups that originally belonged to different operators and required independent scheduling and execution into the same fusion thread group for sequential execution, it eliminates the synchronization overhead caused by scheduling switching between operators and improves the continuity of task execution.

[0072] During the execution of a fusion thread group, intermediate results generated in the preceding computation phase are directly stored in the local memory allocated to that fusion thread group, from which subsequent computation phases can directly retrieve data. Local memory is a high-speed storage medium private to each thread group, with access latency far lower than global memory, higher transmission bandwidth, and access restrictions limited to threads within the same thread group. The allocation of local memory is tied to the lifecycle of the thread group; local memory is allocated when the thread group starts and released when the thread group ends. Therefore, data stored in local memory is only valid during the execution of that thread group and cannot be directly accessed by other independently scheduled thread groups. Optionally, a typical implementation of local memory in GPU architecture is a shared cache. Shared caches can be explicitly declared and allocated in kernel functions, their lifecycle is completely bound to the thread group executing them, and access restrictions are limited to threads within the same thread group.

[0073] Specifically, a first fused thread group can be executed by a stream processor, which can also allocate local memory to the first fused thread group. Taking a GPU in a parallel computing architecture as an example, it can include multiple stream processors, each of which can allocate local memory for its scheduled thread groups. For instance, a first stream processor can allocate local memory for its scheduled first fused thread group. Local memory can only be accessed by the thread group within that stream processor, and its access latency is much lower than that of global memory. The first fused thread group, as well as other fused thread groups described below (e.g., a second fused thread group, etc.), can all be executed by a single stream processor, which can allocate corresponding local memory to these fused thread groups. It should be noted that different fused thread groups can be processed by the same stream processor in a time-sharing or concurrent manner with separate allocation of local memory, or they can be processed by different stream processors in parallel with separate allocation of local memory, depending on the scheduling strategy.

[0074] Compared to local memory, global memory (also known as video memory or device memory) is a shared storage medium accessible to all threads in a parallel computing device. Global memory has a large storage capacity and is globally visible, but it has higher access latency, limited transmission bandwidth, and its data lifetime is independent of a single thread group, persisting across different operators and thread groups. Unlike local memory, which can only be accessed by a thread group within a single stream processor, global memory can be accessed by multiple stream processors.

[0075] For example, a first fusion thread group can be executed by a first stream processor among multiple stream processors, and the first stream processor can allocate a first local memory for the first fusion thread group. Furthermore, in the presence of multiple fusion thread groups, each of the multiple stream processors can execute the aforementioned multiple fusion thread groups, and in this embodiment, all multiple stream processors can access global memory.

[0076] In scenarios without operator fusion, each operator is executed by independent thread groups. When data dependencies exist between thread groups of different operators, their scheduling and execution are decoupled because these thread groups belong to different operators, making it impossible to directly share intermediate results through either group's local memory. Therefore, the data pointed to by the dependency must be written to global memory by the preceding thread group and then read from global memory by the subsequent thread group. In this process, global memory becomes the only shared medium for data exchange between different operator thread groups, and each global memory access is accompanied by significant data transmission latency, bandwidth consumption, and power consumption.

[0077] The embodiments of this disclosure merge thread groups with data dependencies and belonging to different operators into a single merged thread group for sequential execution. This allows intermediate data that would otherwise require transit through global memory to be directly transferred through the local memory allocated to the merged thread group. As a result, data transfer completes a closed loop within a high-speed, low-latency storage hierarchy, completely avoiding global memory access and eliminating data transfer overhead and corresponding power consumption. In other words, this mechanism effectively avoids global access overhead, reducing data transfer power consumption from a higher global access level to a lower local access level.

[0078] Figure 2 This is a schematic diagram of the process of the thread group-based operator fusion method 1000 provided according to an exemplary embodiment of the present disclosure.

[0079] like Figure 2 As shown, the following text will take the typical scenario of “3x3 convolution operator A (hereinafter referred to as operator A) followed by “POOL pooling operator B (hereinafter referred to as operator B)” as an example to describe the specific implementation steps of operator fusion method 1000.

[0080] For example, both operator A and operator B are executed in thread groups. For instance, operator A may include multiple thread groups, each responsible for computing a block of output data in the output tensor 1 of operator A, which corresponds to a continuous range of coordinates within the output tensor 1. It should be noted that... Figure 2 The table illustrates the output tensor 1 of operator A as a whole, with the data blocks processed by each thread group of operator A shown as selected areas in the table.

[0081] Thread t_c within a thread group of operator B is responsible for computing a single element at position (x, y) in the output tensor of operator B. For 2×2 max pooling, this output element depends on four data elements within a 2×2 local window of the output tensor 1 of operator A. These four data elements constitute the input data block corresponding to thread t_c. However, the four data elements in the input data block corresponding to thread t_c are not independently computed by any single thread in operator A.

[0082] In practice, as a convolution operator, each element in the output tensor 1 of operator A is obtained by multiplying and adding a 3×3 convolution kernel with a local region of the input feature map. For example, multiple threads within thread group a of operator A collaborate to complete the computation of all elements in the output data block that thread group a is responsible for. In other words, the input data block corresponding to thread t_c can be generated by the overall collaboration of thread group a of operator A, where a single element in the input data block corresponding to thread t_c is part of the overall work of thread group a, rather than the independent output of a single thread.

[0083] Taking thread-level fusion as an example, this approach analyzes data at the single-thread level, attempting to find a single producer thread in operator A for thread t_c that can directly produce all four elements of its required input data block. However, in operators based on thread group collaboration, this thread-level data dependency does not exist. Therefore, the fusion criterion based on a single thread cannot be met, and the compiler or optimizer in a parallel computing architecture cannot prove that fusing operator A with operator B maintains the correct data dependency.

[0084] In the unfused state, operator A and operator B are executed sequentially as independent computational tasks. Operator A starts first, and after all threads, including thread group a, have completed their computations, it writes the complete output tensor 1 to global memory. Then, the kernel execution of operator A ends. Afterward, operator B starts a completely new kernel, and its thread groups reread the complete output tensor 1 from global memory and extract the necessary local window data for computation. This process introduces two global memory accesses, which incur significant overhead, limit fusion opportunities, and restrict further improvements in hardware computational performance.

[0085] To address the issue of the aforementioned thread-level fusion scheme failing due to the inability to establish data matching between individual threads, the implementation of this disclosure analyzes thread group b of operator B. Thread group b is responsible for calculating a specific region (output data block) in the output tensor of operator B. This region may include multiple output elements, each calculated by a thread within thread group b (e.g., thread t_c). To complete the entire calculation of this output data block, thread group b needs to read all the input data, forming an input data block. This input data block is the union of the local input windows required by all threads within thread group b. When the pooling stride is small, these windows overlap, forming a continuous coordinate region corresponding to a continuous region in the output tensor 1 of operator A. Backtracking to operator A, the analysis determines whether the input data block of thread group b of operator B can be completely provided by the output data block calculated by thread group a of operator A. In other words, it determines whether operator A has a convolution thread group whose assigned task is to produce all the input data required by the target thread group of operator B.

[0086] Once the existence of such input-output dependencies between thread groups based on data blocks is confirmed, a fusion operator can be generated. This fusion operator integrates the computational logic of the corresponding thread groups in the aforementioned operators A and B. When the fusion operator starts execution, at least one fusion thread group can be started to work in parallel according to the total amount of data to be processed, and a local memory block is allocated to each started fusion thread group.

[0087] The execution process of each fusion thread group is as follows: The computation logic of the original operator A is executed, and the intermediate results generated are directly stored in the local memory allocated to the fusion thread group. The fusion thread group then continues to execute the computation logic of the original operator B, reading the intermediate results from its local memory for further computation. Through this operation, intermediate data that originally had to be transferred through global memory can now be transferred within the local memory of the fusion thread group, thereby eliminating the corresponding data transmission delay, bandwidth consumption, and power consumption overhead.

[0088] Furthermore, in some embodiments of this disclosure, the operator fusion method 1000 based on thread groups may further include: determining the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group. Alternatively, the inclusion relationship can be determined based on the configuration information of the first and second thread groups; alternatively, the inclusion relationship can be determined based on the runtime record information of the first and second thread groups. These two methods will be described in detail below.

[0089] Figure 3 This is a flowchart of an operator fusion method 1000 based on thread groups provided according to an exemplary embodiment of the present disclosure.

[0090] like Figure 3 As shown, as an option, determining the inclusion relationship based on the configuration information of the first thread group and the second thread group may include: Step S0-1: Determine the first coordinate range of the output data block of the first thread group in the output tensor of the first operator; determine the second coordinate range of the input data block of the second thread group in the output tensor; and determine the inclusion relationship based on the first coordinate range and the second coordinate range.

[0091] In other words, this step is used to determine whether there is a fusionable dependency between the first thread group of the first operator and the second thread group of the second operator, and to generate the first fusion calculation logic when the condition is met. As one implementation path of this disclosure, step S0-1 can be implemented through static compile-time analysis.

[0092] The first thread group, as an execution unit of the first operator, is responsible for executing the computational logic defined by the first operator on the input data block and generating an output data block. This output data block corresponds to a continuous or discontinuous coordinate range in the output tensor of the first operator, such as the first coordinate range.

[0093] Optionally, during the static compilation phase of the parallel computing device, the compiler can determine the first coordinate range based on the logical identifier of the first thread group and the output data layout of the first operator. The output data layout of the first operator may include information such as the shape of the output tensor of the first operator, the dimensional order of the output tensor of the first operator, and the memory arrangement format of the output tensor of the first operator. This information is known at compile time, and the compiler can establish a correspondence between the logical identifier of the first thread group and the coordinate range of the output tensor of the first operator based on this information. This allows for the accurate calculation of the first coordinate range occupied by the output data block produced by the first thread group in the output tensor of the first operator. This eliminates the need to rely on the runtime data of the parallel computing device to determine the first coordinate range of the output data block of the first thread group in the output tensor of the first operator.

[0094] It should be noted that the output tensor of the first operator is a logical container for the computation result of the first operator. It consists of a multi-dimensional array, and its size can be determined by parameters such as the input size of the first operator, the convolution kernel size, the stride, and padding. The logical identifier of the first thread group can be an index that uniquely identifies the first thread group among all thread groups of the first operator. The compiler can use this index and the dimension information of the first thread group to determine the output region that the first thread group is responsible for computing in the output tensor of the first operator. In other words, the compiler can use this index and the dimension information of the first thread group to determine the first coordinate range corresponding to the output data block produced by the first thread group. The output data layout of the first operator describes the mapping relationship between the output tensor of the first operator in logical coordinates and physical storage, and includes at least: the shape of the output tensor of the first operator, the dimensional order of the output tensor of the first operator, and the memory arrangement format of the output tensor of the first operator. The shape of the output tensor of the first operator represents the size of each dimension of the output tensor, such as height H, width W, and channels C. The dimensional order of the output tensor of the first operator represents the arrangement order of each logical dimension of the output tensor in memory. The memory arrangement format of the output tensor of the first operator represents the refined conventions of the storage layout, thereby further refining the storage layout of the output tensor of the first operator, such as whether to use vectorized storage, whether to perform memory alignment, and whether to use tensorized storage format.

[0095] The second thread group is the execution unit of the second operator, and its computation process requires reading an input data block. This input data block has a corresponding coordinate range in the input tensor of the second operator, which is referred to in this disclosure as the third coordinate range. Similar to the determination of the first coordinate range, the compiler can determine the third coordinate range of the input data block of the second thread group in the input tensor of the second operator based on the logical identifier of the second thread group and the input data layout of the second operator. It should be noted that the definition of the input data layout of the second operator is the same as that of the output data layout of the first operator, which may include the shape of the input tensor of the second operator, the dimensional order of the input tensor of the second operator, and the memory arrangement format of the input tensor of the second operator, etc. This disclosure will not elaborate on these aspects here.

[0096] Since the input tensor of the second operator becomes the output tensor of the first operator through simple transformations such as subset taking and transposition, a clear data flow relationship exists between them. This data flow relationship can be understood as data dependencies and execution order constraints between operators. In a system composed of multiple operators, this refers to the logical dependencies and execution order constraints formed through the generation, flow, and consumption of data. This data flow relationship can be represented by edges in a computation graph (data flow graph) and can be fully obtained at compile time through static analysis.

[0097] Therefore, the compiler can map the third coordinate range forward along the data flow relationship to the output tensor space of the first operator, thereby obtaining the second coordinate range corresponding to the input data block of the second thread group of the second operator in the output tensor of the first operator. This mapping process is also static, relying only on the data flow topology of each operator, the data layout of each operator, and the logical identifier of each thread group, without relying on the data of the parallel computing device at runtime.

[0098] After obtaining the first and second coordinate ranges, the compiler performs a set inclusion relationship determination. If the first coordinate range contains the second coordinate range, or in other words, every coordinate point in the second coordinate range lies within the first coordinate range, it indicates that all input data required by the second thread group of the second operator can be directly provided by the output data blocks produced by the first thread group of the first operator, and there is an input-output dependency relationship between the two based on data blocks. Therefore, in response to the first coordinate range containing the second coordinate range, the computational logic of the first thread group and the computational logic of the second thread group are merged to generate the first fused computational logic.

[0099] The compiler can compile the first fusion computation logic into a kernel function, which is then executed by the first fusion thread group. During execution, computation logic that originally belonged to different operators and required independent scheduling is integrated into the same fusion thread group and executed sequentially. Intermediate results generated in the preceding computation stage are directly stored in the local memory allocated to the fusion thread group, and subsequent computation stages can directly read from this local memory and continue computation.

[0100] Figure 4 This is a flowchart of an operator fusion method 1000 based on thread groups provided according to an exemplary embodiment of the present disclosure.

[0101] like Figure 4 As shown, as an alternative, the thread group-based operator fusion method 1000 may also include: Step S0-2: Obtain a first label and a second label from historical execution data, wherein the first label indicates at least the logical identifier of the first thread group and the range of the output data block of the first thread group, and the second label indicates at least the logical identifier of the second thread group and the range of the input data block of the second thread group; and based on the first label and the second label, determine the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group.

[0102] Step S0-2 identifies the data dependency relationship between the first thread group and the second thread group based on the historical execution data generated during the operation of the parallel computing device, and determines whether the two meet the fusion conditions. It should be noted that, as another implementation path of this disclosure, step S0-2 can be independent of the analysis path of the aforementioned step S0-1, or they can complement each other to adapt to the needs of different optimization scenarios.

[0103] Specifically, during one or more actual executions of an operator, the parallel computing device can record the execution information of each thread group and organize it into persistent tag data to form historical execution data. A first tag is associated with the first thread group of the first operator and records at least the logical identifier of the first thread group and the range of output data blocks actually produced by the first thread group in the executed computation. A second tag is associated with the second thread group of the second operator and records at least the logical identifier of the second thread group and the range of input data blocks actually consumed by the second thread group in the executed computation.

[0104] Optionally, determining the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group based on the first label and the second label may include: determining a first coordinate range of the output data block of the first thread group in the output tensor of the first operator based on the first label; determining a third coordinate range of the input data block of the second thread group in the input tensor of the second operator based on the second label; mapping the third coordinate range to the output tensor based on the data flow relationship between the first operator and the second operator to determine a second coordinate range of the input data block of the second thread group in the output tensor; and determining that the range of the output data block of the first thread group includes the range of the input data block of the second thread group based on the first coordinate range including the second coordinate range.

[0105] For example, when it is necessary to perform fusion optimization on the first operator and the second operator, the optimizer of the parallel computing device (e.g., compiler, runtime optimization system, or other dedicated profiling tool) can read the above tags from historical execution data and perform the following determination process: Based on the information recorded by the first tag, determine the first coordinate range occupied by the output data block of the first thread group in the output tensor of the first operator; Based on the information recorded by the second tag, determine the third coordinate range occupied by the input data block of the second thread group in the input tensor of the second operator; According to the data flow relationship between the first operator and the second operator, map the third coordinate range to the output tensor space of the first operator, thereby obtaining the second coordinate range corresponding to the input data block of the second thread group in the output tensor of the first operator, wherein the data flow relationship can be obtained statically from the computation graph or from the tensor dependency relationship recorded in historical execution; Perform set inclusion relationship determination: In response to the first coordinate range completely covering the second coordinate range, determine that the range of the output data block of the first thread group includes the range of the input data block of the second thread group, and there is an input-output dependency relationship based on data blocks between the two, satisfying the fusion condition.

[0106] This public announcement was passed. Figure 3 The static compile-time decision path shown is... Figure 4 The dynamic runtime profiling and decision-making paths shown construct a complementary and collaborative thread group-level operator fusion decision-making technology system. The two paths have different focuses in terms of information source, applicable scenarios, implementation timing, and optimization characteristics. They can be deployed independently or organically combined to adapt to the needs of different compilation optimization stages and hardware deployment environments.

[0107] Static compile-time path determination takes the known thread group logical identifiers, operator data layouts, and data flow relationships as input, and determines the path through coordinate range calculations and containment relationships. Therefore, it does not require analyzing historical execution data or any dynamic analysis, and does not introduce runtime overhead. This path is suitable for scenarios where operator execution behavior is fixed, data layout information is complete, and thread group task division rules are clear.

[0108] The dynamic runtime profiling and path determination method takes the thread group labels recorded in historical execution data as input and determines the inclusion relationship based on the actual observed range of production / consumption data blocks. This method does not rely on compile-time complete information and can effectively handle complex situations that are difficult to handle with static analysis.

[0109] Furthermore, in embodiments of this disclosure, two threads belonging to different operators that have a data block inclusion relationship are merged to form a fusion operator with a single fusion thread group as the execution carrier. Based on this, some embodiments of this disclosure further provide a recursively expandable fusion computation chain construction mechanism, enabling fusion optimization to span multiple operators and multiple thread groups, forming a longer sequence of continuous computation tasks.

[0110] Specifically, when the range of the input data block of a merged thread group can be contained within the range of the output data block of a thread group of another upstream operator, the upstream thread group has the condition to further merge with the merged thread group; similarly, when the range of the output data block of a merged thread group can contain the range of the input data block of a thread group of another downstream operator, the downstream thread group also has the condition to further merge. By recursively applying the inclusion relationship determination criterion, the first fusion calculation logic of the first fusion thread group can be updated, gradually expanding it into a fusion calculation logic that includes the calculation logic of multiple original operators.

[0111] Figure 5 This is a flowchart of an operator fusion method 1000 based on thread groups provided according to an exemplary embodiment of the present disclosure. Figure 6 This is a schematic diagram of the process of the thread group-based operator fusion method 1000 provided according to an exemplary embodiment of the present disclosure.

[0112] like Figure 5 and Figure 6 As shown, in some embodiments of this disclosure, the operator fusion method 1000 based on thread groups may further include: Step S1-1: In response to the fact that the range of the output data block of the third thread group of the third operator includes the range of the input data block of the first thread group, estimate the storage capacity required after fusing the first fusion computing logic with the computing logic of the third thread group; and in response to the storage capacity being equal to or less than the capacity of the first local memory, fuse the first fusion computing logic with the computing logic of the third thread group to update the first fusion computing logic.

[0113] Combination Figure 1 and Figure 6 In step S1, in response to the fact that the output data block range of the first thread group a of the first operator A includes the input data block range of the second thread group b of the second operator B, the computational logic of the two has been merged to generate a first fused computational logic. This first fused computational logic is executed by the first fused thread group, which internally executes the computational tasks of the first thread group a and the second thread group b sequentially. The first fused computational logic can be considered a higher-level computational unit, whose input data block is the same as that of the first thread group a, and whose output data block is the same as that of the second thread group b.

[0114] Building upon this, step S1-1 further provides a mechanism for forward expansion fusion. Specifically, it can be first determined whether the range of the output data block of the third thread group c of the third operator C (which can be understood as the output tensor 3) can include the range of the input data block of the first fusion calculation logic, or in other words, whether the range of the output data block of the third thread group c of the third operator C can include the range of the input data block of the first thread group a. If the range of the output data block of the third thread group c of the third operator C can include the range of the input data block of the first fusion calculation logic, it indicates that the data block produced by the third thread group c is sufficient to directly provide all the input data required by the first fusion thread group.

[0115] In response to the aforementioned inclusion relationship, a data block-based input-output dependency exists between the third thread group c and the first fusion thread group, enabling further fusion. However, after incorporating the computational logic of the third thread group c into the first fusion computational logic, the number of intermediate results that the first fusion thread group needs to maintain simultaneously during execution will increase, leading to a rise in the total occupancy of the first local memory. Therefore, step S1-1 estimates the storage capacity of the local memory after fusion expansion and compares it with the hardware capacity of the first local memory.

[0116] For example, in some embodiments of this disclosure, in response to the range of the output data block of the third thread group of the third operator including the range of the input data block of the first thread group, the storage capacity required to merge the first fusion computing logic and the computing logic of the third thread group is estimated; and in response to the storage capacity being greater than the capacity of the first local memory, the result generated by executing the computing logic of the second thread group in the first fusion computing logic is stored in a global memory different from the first local memory.

[0117] For example, in some other embodiments of this disclosure, in response to the range of the output data block of the third thread group of the third operator including the range of the input data block of the first thread group, the storage capacity required to merge the first fusion computing logic with the computing logic of the third thread group is estimated; and in response to the storage capacity being equal to or less than the capacity of the first local memory, the first fusion computing logic is merged with the computing logic of the third thread group to update the first fusion computing logic.

[0118] Specifically, in some embodiments of this disclosure, forward expansion fusion may fail to execute due to insufficient local memory capacity. For example, incorporating the computational logic of the third thread group c would cause local memory to exceed its limit, making it impossible to support the normal execution of the fused thread group. Therefore, the fusion expansion operation is not performed, and only the existing first fused thread group is maintained, executing the first fused computational logic. In other words, in response to the estimated storage capacity required after fusing the first fused computational logic with the computational logic of the third thread group c being greater than the capacity of the first local memory, the results generated by executing the computational logic corresponding to the second thread group in the first fused computational logic can be stored in a global memory different from the first local memory for use by other operators or other fusion operators. The results generated by the computational logic corresponding to the second thread group are the output data blocks of the first fused thread group, which are the same as the output data blocks of the second thread group b.

[0119] In some other embodiments of this disclosure, in response to the estimated storage capacity required after the first fusion computing logic and the computing logic of the third thread group c are fused together being equal to or less than the capacity of the first local memory, a fusion expansion operation can be performed to incorporate the computing logic of the third thread group c into the first fusion computing logic, thereby updating the first fusion computing logic.

[0120] With this update, the first fusion computing logic now includes the computing logic of three thread groups (thread group a, thread group b, and thread group c), which are executed sequentially by the same fusion thread group. The input data block of this fusion thread group is the same as the input data block of thread group c, and the output data block of this fusion thread group is still the same as the output data block of thread group b.

[0121] Based on this, if the output data block range of a thread group in a higher-level operator can include the input data block range of the current fusion thread group, then the judgment and expansion logic in step S1-1 can be repeatedly applied to integrate the higher-level thread groups one by one into the current fusion computation logic. This forward recursive expansion uses the judgment criterion that the output data block range of the thread group in the upstream operator includes the input data block range of the thread group in the downstream operator as a unified link interface, connecting the thread groups that originally belonged to different operators into a long sequence of computation tasks that are executed sequentially within a single fusion thread group. By using the thread group as the fusion granularity, through logical integration and localized data transfer, the continuous utilization rate of hardware computing resources is improved, and the overall execution efficiency and energy efficiency of the computation task are enhanced, while reducing scheduling and data transfer overhead.

[0122] For example, in response to the range of the output data block of the Mth thread group encompassing the range of the input data block of the Nth thread group, the storage capacity required to merge the current first fusion computation logic with the computation logic of the Mth thread group is estimated, wherein the current first fusion computation logic includes the computation logic of the Nth thread group, and the input of the Nth thread group is the input of the current first fusion computation logic; and in response to the storage capacity being equal to or less than the capacity of the first local memory, the computation logic of the Mth thread group is merged with the current first fusion computation logic to update the first fusion computation logic, wherein M and N are different and both are positive integers greater than 2. In the computation logic composed of the multiple thread groups that have been merged, the Nth thread group is the starting executor of the entire fusion chain, and the input data it requires is the input data of the entire fusion computation logic. In other words, the Nth thread group, as the upstream thread group of the fusion chain, has the input of the current first fusion computation logic.

[0123] In this embodiment, the Mth thread group and the Nth thread group belong to two different operators, and these two operators are different from the first and second operators mentioned above. In other words, the operator to which the Mth thread group belongs is not the same operator as the operator to which the Nth thread group belongs, nor is it the same as the first or second operator.

[0124] It should be noted that in the above-described forward-expanding fusion implementation, the "first fusion thread group" mentioned is different from the aforementioned basic fusion implementation (e.g., Figure 1 In the implementation method, the "first fusion thread group" refers to the same execution entity, but the computational logic it contains has been expanded. Specifically, in the basic fusion stage, the first fusion thread group corresponds to the first fusion computational logic, which may only include the computational tasks of the first thread group of the first operator and the second thread group of the second operator. In the implementation method of forward expansion fusion, in response to the third thread group of the third operator satisfying the fusion condition, the computational logic of the third thread group can be merged with the original first fusion computational logic, thereby updating the first fusion computational logic. Therefore, the computational logic executed by the updated first fusion thread group actually sequentially includes the thread groups of the third, first, and second operators, totaling three thread groups. The "first fusion thread group" in this implementation method is consistent in name with the "first fusion thread group" in the basic fusion implementation method to reflect that it is a continuation of the same execution entity in the fusion chain expansion process, but the scope of the fusion computational logic it carries has been dynamically expanded according to the judgment criteria and storage capacity constraints.

[0125] In this embodiment, step S2, where the first fusion thread group executes the first fusion calculation logic, can be updated to step S2-1, where the first fusion thread group executes the updated first fusion calculation logic. Step S2-1 may include: The updated first fusion computing logic corresponding to the third thread group is executed to generate a second intermediate result; and the second intermediate result is stored in the first local memory, wherein, based on the second intermediate result read from the first local memory, the computing logic corresponding to the first thread group in the first fusion computing logic is executed.

[0126] Specifically, refer to Figure 5 and Figure 6 After successfully merging the computation logic of the third thread group c into the first fusion computation logic in step S1-1, the updated first fusion computation logic contains three computation stages, and their execution order and data flow are as follows: The fusion thread group first executes the computation logic corresponding to the third thread group c of the third operator C, generating a second intermediate result, which is the output data block of the third thread group c. This second intermediate result is written to the first local memory allocated to the fusion thread group. The fusion thread group reads the second intermediate result from the first local memory, uses it as input, executes the computation logic corresponding to the first thread group a, generating a first intermediate result, which is the output data block of the first thread group a. This first intermediate result is written to the first local memory. The fusion thread group reads the first intermediate result from the first local memory, executes the computation logic corresponding to the second thread group b, and outputs the computation result.

[0127] In this way, the generation and consumption of all intermediate results are completed within the local memory of the same fused thread group, without any access to global memory, completely eliminating read / write latency and bandwidth consumption introduced by cross-operator data transfer. After forward expansion fusion, the newly integrated upstream thread group (e.g., the third thread group c) naturally becomes the first stage of the execution sequence, and its calculation results are directly supplied to subsequent stages, conforming to the positive dependency relationship in the data flow graph, without the need for additional data rearrangement or synchronization operations.

[0128] Figure 7 This is a flowchart of an operator fusion method 1000 based on thread groups provided according to an exemplary embodiment of the present disclosure. Figure 8 This is a schematic diagram of the process of the thread group-based operator fusion method 1000 provided according to an exemplary embodiment of the present disclosure.

[0129] like Figure 7 and Figure 8 As shown, in some embodiments of this disclosure, the operator fusion method 1000 based on thread groups may further include: Steps S1-2: In response to the fact that the range of the output data block of the second thread group includes the range of the input data block of the fourth thread group of the fourth operator, estimate the storage capacity required after fusing the first fusion computing logic with the computing logic of the fourth thread group; and in response to the fact that the storage capacity is equal to or less than the capacity of the first local memory, fuse the first fusion computing logic with the computing logic of the fourth thread group to update the first fusion computing logic.

[0130] Combination Figure 1 and Figure 7 In step S1, in response to the fact that the output data block range of the first thread group a of the first operator A includes the input data block range of the second thread group b of the second operator B, the computational logic of the two has been merged to generate a first fused computational logic. This first fused computational logic is executed by the first fused thread group, which internally executes the computational tasks of the first thread group a and the second thread group b sequentially. The first fused computational logic can be considered a higher-level computational unit, whose input data block is the same as that of the first thread group a, and whose output data block is the same as that of the second thread group b.

[0131] Building upon this, steps S1-2 further provide a mechanism for backward-forward expansion fusion. Specifically, it can be first determined whether the range of the output data block of the first fusion calculation logic can include the range of the output data block of the fourth thread group d of the fourth operator D, or in other words, whether the range of the output data block of the second thread group (i.e., output tensor 2) can include the range of the output data block of the fourth thread group d of the fourth operator D. If the range of the output data block of the second thread group can include the range of the output data block of the fourth thread group d of the fourth operator D, it indicates that the data block produced by the first fusion thread group is sufficient to directly provide all the input data required by the fourth thread group d of the fourth operator D.

[0132] In response to the aforementioned inclusion relationship, a data block-based input-output dependency exists between the first fusion thread group and the fourth thread group d, enabling further fusion. However, after incorporating the computational logic of the fourth thread group d into the first fusion computational logic, the number of intermediate results that need to survive simultaneously during the execution of the first fusion thread group will increase, leading to an increase in the total occupancy of the first local memory. Therefore, steps S1-2 can estimate the storage capacity of the local memory after fusion expansion and compare it with the hardware capacity of the first local memory.

[0133] For example, in some embodiments of this disclosure, in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator, the storage capacity required to merge the first fusion computing logic with the computing logic of the fourth thread group is estimated; and in response to the storage capacity being greater than the capacity of the first local memory, the result generated by executing the computing logic of the first fusion computing logic corresponding to the second thread group is stored in a global memory different from the first local memory.

[0134] For example, in some other embodiments of this disclosure, in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator, the storage capacity required to merge the first fusion computing logic with the computing logic of the fourth thread group is estimated; and in response to the storage capacity being equal to or less than the capacity of the first local memory, the first fusion computing logic is merged with the computing logic of the fourth thread group to update the first fusion computing logic.

[0135] Specifically, in some embodiments of this disclosure, backward expansion fusion may fail to execute due to insufficient local memory capacity. For example, incorporating the computational logic of the fourth thread group d would cause local memory to exceed its limit, making it impossible to support the normal execution of the fused thread group. Therefore, the fusion expansion operation is not performed, and only the existing first fused thread group is maintained, executing the first fused computational logic. In other words, in response to the estimated storage capacity required after fusing the first fused computational logic with the computational logic of the fourth thread group d being greater than the capacity of the first local memory, the results generated by executing the computational logic corresponding to the second thread group in the first fused computational logic can be stored in a global memory different from the first local memory for use by other operators or other fusion operators. The results generated by the computational logic corresponding to the second thread group are the output data blocks of the first fused thread group, which are the same as the output data blocks of the second thread group b.

[0136] In some other embodiments of this disclosure, in response to the estimated storage capacity required after the first fusion computing logic and the computing logic of the fourth thread group d are fused being equal to or less than the capacity of the first local memory, a fusion expansion operation can be performed to incorporate the computing logic of the fourth thread group d into the first fusion computing logic, thereby updating the first fusion computing logic.

[0137] With this update, the first fusion computing logic now includes the computing logic of three thread groups (thread group a, thread group b, and thread group d), which are executed sequentially by the same fusion thread group. The input data block of this fusion thread group is still the same as the input data block of thread group a, and the output data block of this fusion thread group is the same as the output data block of thread group d.

[0138] Based on this, if the output data block range of the current fusion thread group can encompass the input data block range of a thread group in a downstream operator, then the judgment and expansion logic in steps S1-2 can be repeatedly applied to integrate the downstream thread groups one by one into the current fusion computation logic. This backward recursive expansion uses the judgment criterion that the output data block range of the thread group in the upstream operator includes the input data block range of the thread group in the downstream operator as a unified link interface, chaining thread groups originally belonging to different operators into a long sequence of computation tasks executed sequentially within a single fusion thread group. Using thread groups as the fusion granularity, through logical integration and localized data transfer, the continuous utilization of hardware computing resources is improved, and the overall execution efficiency and energy efficiency of the computation tasks are enhanced, while reducing scheduling and data transfer overhead.

[0139] For example, in response to the range of the output data block of thread group P including the range of the input data block of thread group Q, the storage capacity required to merge the current first fusion computation logic with the computation logic of thread group Q is estimated, wherein the current first fusion computation logic includes the computation logic of thread group P, and the output of thread group P is the output of the current first fusion computation logic; and in response to the storage capacity being equal to or less than the capacity of the first local memory, the computation logic of thread group Q is merged with the current first fusion computation logic to update the first fusion computation logic, wherein P and Q are different and are both positive integers greater than 2. In the computation logic composed of multiple currently merged thread groups, thread group P is the final executor of the entire fusion chain, and the result it calculates is the final output of the entire fusion computation logic. In other words, thread group P, as the downstream thread group of the fusion chain, outputs the output of the current first fusion computation logic.

[0140] In this embodiment, the P-th thread group and the Q-th thread group belong to two different operators, and these two operators are different from the first and second operators mentioned above. In other words, the operator to which the M-th thread group belongs is not the same operator as the operator to which the N-th thread group belongs, nor is it the same as the first or second operator.

[0141] In this embodiment, step S2, where the first fusion thread group executes the first fusion calculation logic, can be updated to step S2-2, where the first fusion thread group executes the updated first fusion calculation logic. Step S2-2 may include: The third intermediate result generated by the computation logic corresponding to the second thread group in the updated first fusion computation logic is stored in the first local memory; the third intermediate result is read from the first local memory, and the computation logic corresponding to the fourth thread group in the first fusion computation logic is executed based on the third intermediate result.

[0142] Specifically, refer to Figure 7and Figure 8 After successfully merging the computation logic of the fourth thread group d into the first fusion computation logic in step S1-2, the updated first fusion computation logic contains three computation stages, and their execution order and data flow are as follows: The fusion thread group first executes the computation logic corresponding to the first thread group a, generating a first intermediate result, which is the output data block of the first thread group a. This first intermediate result is written to the first local memory. The fusion thread group reads the first intermediate result from the first local memory, executes the computation logic corresponding to the second thread group b, and generates a third intermediate result. The third intermediate result is stored in the first local memory. The third intermediate result is then read from the first local memory, and based on this third intermediate result, the computation logic corresponding to the fourth thread group d in the first fusion computation logic is executed.

[0143] In this way, the generation and consumption of all intermediate results are completed within the local memory of the same fused thread group, without any access to the global memory, completely eliminating read / write latency and bandwidth consumption introduced by cross-operator data transfer. After backward expansion and fusion, the newly integrated downstream thread group (e.g., the fourth thread group d) naturally becomes the last stage of the execution sequence, and its calculation results can be stored in a global memory different from the first local memory. This conforms to the positive dependency relationship in the data flow graph and requires no additional data rearrangement or synchronization operations.

[0144] It should be noted that in the above-described backward expansion fusion implementation, the "first fusion thread group" mentioned is different from the aforementioned basic fusion implementation (e.g., Figure 1 In the implementation method, the "first fusion thread group" refers to the same execution entity, but the computational logic it contains has been expanded. Specifically, in the basic fusion stage, the first fusion thread group corresponds to the first fusion computational logic, which may only contain the computational tasks of the first thread group of the first operator and the second thread group of the second operator. In the implementation method of backward fusion, in response to the fourth thread group of the fourth operator satisfying the fusion condition, the computational logic of the fourth thread group can be merged with the original first fusion computational logic, thereby updating the first fusion computational logic. Therefore, the computational logic executed by the updated first fusion thread group actually sequentially includes the thread groups of the first, second, and fourth operators, totaling three thread groups. The "first fusion thread group" in this implementation method is consistent in name with the "first fusion thread group" in the basic fusion implementation method to reflect that it is a continuation of the same execution entity in the fusion chain expansion process, but the scope of the fusion computational logic it carries has been dynamically expanded according to the judgment criteria and storage capacity constraints.

[0145] Figure 9This is a schematic diagram of the process of the thread group-based operator fusion method 1000 provided according to an exemplary embodiment of the present disclosure.

[0146] like Figure 9 As shown, in some embodiments of this disclosure, the operator fusion method 1000 based on thread groups may include the forward expansion fusion step and the backward expansion fusion step described above. By recursively applying the inclusion relationship determination criterion, the first fusion calculation logic of the first fusion thread group is updated, and the fusion operator is gradually expanded into a fusion calculation chain containing multiple original operators and multiple original thread groups. The specific process will not be described in detail here.

[0147] Optionally, in the first local memory, at least two of the multiple intermediate results generated by executing the updated first fused computing logic may be stored in a storage space with at least partial address overlap, wherein the multiple intermediate results include the first intermediate result, and the storage lifetimes of the at least two intermediate results do not overlap.

[0148] In other words, the space reuse mechanism of the local memory allocated to the fusion thread group supports time-sharing multiplexing. Intermediate results can be generated, transferred, and consumed in the first local memory, and then their storage space can be reused or released by subsequent computations. The storage lifecycle can be understood as the time interval from when an intermediate result is written to local memory until it is last read. In the fusion computation chain executed by the same fusion thread group, the storage lifecycles of intermediate results generated in different computation stages are often serial and do not overlap. For example, in the forward scaling fusion scenario, the second intermediate result generated by the third thread group c is no longer used after being read by the first thread group a; while the first intermediate result generated by the first thread group a has not yet been generated or has just been generated. Therefore, the storage lifecycles of the second intermediate result and the first intermediate result do not overlap. Similarly, in the backward scaling fusion scenario, the first intermediate result generated by the first thread group a is no longer used after being read by the second thread group b, while the lifecycle of the third intermediate result generated by the second thread group b has not yet started or has just started.

[0149] In some embodiments of this disclosure, leveraging the aforementioned non-overlapping lifecycle characteristic, multiple intermediate results with non-overlapping lifecycles are allocated to the same local memory address space through compile-time static lifecycle analysis or runtime dynamic allocation strategies. A subsequent intermediate result can directly overwrite the storage area occupied by a previous intermediate result, without needing to reserve dedicated storage space for each intermediate result.

[0150] For example, taking multiple intermediate results including a first intermediate result and a third intermediate result as an example, the first intermediate result can be stored in the first storage space of the first local memory; and after the first intermediate result is read, the third intermediate result is stored in the first storage space to overwrite the first intermediate result.

[0151] Alternatively, taking multiple intermediate results including a second intermediate result and a first intermediate result as an example, the second intermediate result can be stored in the first storage space of the first local memory; and after the second intermediate result is read, the first intermediate result can be stored in the first storage space to overwrite the second intermediate result.

[0152] It should be noted that the first, second, and third intermediate results exemplified here can be the first, second, and third intermediate results described in the foregoing embodiments (for example, in a forward-expanding fusion scenario, the second intermediate result can be the intermediate result generated by the third thread group, and the first intermediate result can be the intermediate result generated by the first thread group, etc.). However, the above naming is only to distinguish different intermediate results and is not a limitation on the number, generation order, or specific correspondence of intermediate results. The overwrite operation is not limited to the implementation of two or three intermediate results; any subsequent intermediate result can overwrite a previously read intermediate result. These intermediate results can all be generated by the same fusion thread group and stored in the local memory allocated to that same fusion thread group.

[0153] In this way, the actual local memory usage no longer depends on the number of intermediate results, but on the size of the largest simultaneous surviving intermediate result. Since the lifecycles of each intermediate result in the fused computing chain are serial, this number of "simultaneously surviving" results is usually 1 or 2. Therefore, the local memory usage is compressed to a constant level, which is essentially decoupled from the length of the fused chain, making the expansion of the fused chain not subject to the physical constraints of local memory capacity.

[0154] Furthermore, in some embodiments of this disclosure, step S1, in response to the range of the output data block of the first thread group of the first operator including the range of the input data block of the second thread group of the second operator, and merging the computational logic of the first thread group and the computational logic of the second thread group to generate the first fused computational logic, may further include: determining the size of the input data block of the first thread group; determining the data reuse rate based on the ratio between the size of the input data block of the first thread group and the size of the input data block of the second thread group; and in response to the data reuse rate being greater than or equal to a predetermined threshold, merging the computational logic of the first thread group and the computational logic of the second thread group to generate the first fused computational logic.

[0155] As described above, the embodiments of this disclosure use the inclusion of the input data block range of the thread group of the upstream operator within the thread group of the downstream operator as the criterion for determining whether two thread groups can be merged, which ensures the correctness of the data dependency relationship. However, in some computing scenarios, there may be significant overlap in computation between two thread groups that satisfy the inclusion relationship. In other words, the first thread group needs to load and process a large input data block to generate its output data block, while the second thread group only consumes a small sub-region from it. In this case, although fusion is feasible, the benefits of data reuse may be offset by the excessive overhead of repeated computation, and the overall benefit is not significant.

[0156] To address the aforementioned issues, the embodiments of this disclosure further introduce a quantitative metric—data reuse rate—to evaluate the actual benefits of fusion operations. Specifically, for a first thread group and a second thread group that satisfy an inclusion relationship, the input data block size of the first thread group reflects the total amount of raw data that the first thread group needs to load and process to produce its output data block; the input data block size of the second thread group reflects the total amount of data that the second thread group needs to consume from the output data block of the first thread group to complete its computational task. Since the output data block of the first thread group is generated from its input data block through computation, a definite mapping relationship exists between the two. With fixed operator types and operator parameters, the proportional relationship between the input data block sizes of the first thread group and the second thread group can characterize the trade-off between the computational cost incurred by the first thread group to support the computation of the second thread group and the data reuse benefits obtained by the second thread group.

[0157] Optionally, the ratio of the input data block size of the second thread group to the input data block size of the first thread group is defined as the data reuse rate. A higher data reuse rate indicates that a higher proportion of the input data processed by the first thread group is directly reused by the second thread group, resulting in more significant fusion benefits. Conversely, a low data reuse rate indicates that the first thread group needs to process a large amount of input data to produce a small output data block, while the second thread group only consumes a very small portion. The data locality benefits brought by fusion may not offset the significant computational overhead of the first thread group itself. Therefore, this embodiment introduces a predetermined threshold to quantify the data reuse rate. If the data reuse rate is greater than or equal to the predetermined threshold, it is determined that the fusion of the first and second thread groups has positive benefits, and a fusion operation is performed. If the data reuse rate is less than the predetermined threshold, even if the data blocks of the first and second thread groups are contained within each other, fusion may not be performed to avoid situations where the benefits are less than expected.

[0158] Figure 10This is a schematic diagram of the process of the thread group-based operator fusion method 1000 provided according to an exemplary embodiment of the present disclosure.

[0159] like Figure 10 As shown, in some embodiments of this disclosure, the operator fusion method 1000 based on thread groups may further include: in response to the range of the output data block of the fifth thread group of the first operator including the range of the input data block of the sixth thread group of the fifth operator, fusing the computational logic of the fifth thread group with the computational logic of the sixth thread group to generate a second fused computational logic; and the execution of the second fused computational logic by the second fused thread group, including: executing the computational logic corresponding to the fifth thread group in the second fused computational logic to generate a fourth intermediate result; storing the fourth intermediate result in a second local memory allocated to the second fused thread group; and reading the fourth intermediate result from the second local memory and executing the computational logic corresponding to the sixth thread group in the second fused computational logic based on the fourth intermediate result.

[0160] In other words, the foregoing embodiments of this disclosure have detailed how to achieve thread-group-level operator fusion for a specific thread group pair—the first thread group a of the first operator A and the second thread group b of the second operator B—through steps such as inclusion relationship determination, fusion logic generation, and fusion thread group execution. However, in actual computational graphs, there are a large number of fusionable thread group pairs distributed across different operators and data flow paths, without direct dependencies or conflicts between them. To fully explore the potential for global fusion optimization, the embodiments of this disclosure further provide the ability to construct parallel and independent fusion instances.

[0161] In this embodiment, the fifth thread group and the sixth thread group of the first operator have an input-output dependency relationship based on data blocks. Therefore, the computational logic of the fifth thread group and the computational logic of the sixth thread group can be merged to generate a second fused computational logic. The second fused computational logic is another fused computational logic generated independently of the first fused computational logic. The two are logically completely equivalent and do not interfere with each other. The second fused thread group is the execution entity that executes the second fused computational logic. It can be scheduled in parallel with the first fused thread group and executed independently. The second fused thread group can be allocated an independent second local memory to store intermediate results (e.g., a fourth intermediate result) generated during its execution.

[0162] Specifically, the second fusion thread group can execute the computational logic corresponding to the fifth thread group to generate a fourth intermediate result; store the fourth intermediate result in the second local memory allocated to the second fusion thread group; read the fourth intermediate result from the second local memory and execute the computational logic of the sixth thread group. It should be noted that the second fusion computational logic and its execution entity, the second fusion thread group, constructed in this embodiment are fully compatible with the aforementioned technical features of this disclosure. In other words, the second fusion computational logic is not an isolated basic fusion instance, but can be fully incorporated into the fusion optimization technology system constructed in this disclosure. For example, the fusion expansion mechanism, local memory space reuse mechanism, and data reuse rate determination mechanism described above are all adapted to the second fusion computational logic and its execution entity, the second fusion thread group.

[0163] Optionally, the first fusion thread group and the second fusion thread group are executed by the same stream processor, and the same stream processor allocates the first fusion thread group and the second fusion thread group to the first fusion thread group and the second fusion thread group, respectively.

[0164] For example, the same stream processor can schedule the first fusion thread group and the second fusion thread group sequentially or simultaneously using time-division multiplexing or thread-level parallelism. When allocating local memory, the same stream processor can independently allocate a local memory space for each fusion thread group it schedules; for example, allocating the first local memory for the first fusion thread group and the second local memory for the second fusion thread group. The local memories of different fusion thread groups can be physically or logically isolated from each other, preventing data conflicts. Since the computational logic of each fusion thread group is independent, and its intermediate results are stored only in its respective local memory, no additional synchronization overhead is required when the same stream processor executes multiple fusion thread groups, and the efficiency of local data transfer within each fusion thread group is not affected. This implementation further improves the utilization of the stream processor, allowing multiple independent fusion computing tasks to share the same stream processor resources while maintaining their respective data locality advantages.

[0165] Therefore, according to at least one embodiment of this disclosure, the range of the output data block of the first thread group in response to the first operator includes the range of the input data block of the second thread group of the second operator. The computational logic of the first thread group and the computational logic of the second thread group are merged to generate fused computational logic. This integrates thread groups that originally belonged to different operators and required independent scheduling and execution into a single fused thread group for sequential execution, thus eliminating the synchronization caused by scheduling switching between operators and improving the continuity of task execution. Furthermore, during the execution of the fused thread group, intermediate results generated in the preceding computation stage are stored in the local memory allocated to the fused thread group, from which subsequent computation stages can directly read. This mechanism effectively avoids global access overhead, reducing the power consumption of data transmission from a higher global access level to a lower local access level. Therefore, the fusion scheme provided by the embodiments of this disclosure, using thread groups as the fusion granularity, improves the continuous utilization of hardware computing resources and enhances the overall execution efficiency and energy efficiency of computing tasks by reducing scheduling and data transmission overhead through logical integration and localized data transfer.

[0166] Figure 11 This is a block diagram of an operator fusion apparatus based on thread groups provided according to an exemplary embodiment of the present disclosure.

[0167] like Figure 11 As shown, some embodiments of this disclosure provide a thread group-based operator fusion apparatus 2000. The thread group-based operator fusion apparatus 2000 may include a fusion unit 100 and a calculation unit 200, wherein the fusion unit 100 is configured to fuse the calculation logic of the first thread group with the calculation logic of the second thread group in response to the range of the output data block of the first thread group of a first operator encompassing the range of the input data block of the second thread group of a second operator, generating first fused calculation logic; and the calculation unit 200 is configured to execute the first fused calculation logic by the first fused thread group, wherein a first local memory allocated to the first fused thread group stores intermediate results of the first fused calculation logic.

[0168] In architectures such as GPUs, a thread is the smallest schedulable unit of execution on a parallel computing device. A thread can independently execute the instruction stream of a kernel function. Multiple threads are organized into thread groups, which are the basic units for task scheduling, resource allocation, and inter-thread cooperation on parallel computing devices. Multiple threads within the same thread group can be scheduled concurrently and cooperate by sharing the local memory allocated to the thread group and executing synchronization instructions, thereby jointly processing a block of data.

[0169] In other words, an operator can execute its computational tasks in parallel by launching one or more thread groups, where each thread group is responsible for processing and computing a corresponding data block, and the complete output of the operator can be composed of the output data blocks computed by all its thread groups. A data block can be understood as a collection of input or output data processed by a single thread group. Optionally, a data block corresponds to a continuous or discontinuous range of coordinates in the operator's overall data tensor, and the complete input or output of the operator can be composed of the data blocks processed by all the thread groups launched by the operator.

[0170] In some operator fusion schemes that focus on operator-level fusion and thread-level fusion, when faced with a computing mode that uses thread groups as the execution unit and data blocks as the read and write carrier, it is difficult to effectively identify and utilize the input-output dependencies formed between thread groups based on data blocks. This results in limited fusion opportunities, high memory access overhead, and restrictions on further improvement of hardware computing performance.

[0171] To at least address the aforementioned issues, embodiments of this disclosure provide an operator fusion apparatus 2000 based on thread groups, where the analysis and execution granularity of the fusion scheme is the thread group and the data blocks processed by the thread group. Specifically, the range of the output data block of the first thread group responding to the first operator includes the range of the input data block of the second thread group of the second operator. The computational logic of the first thread group and the computational logic of the second thread group are fused to generate fused computational logic. This integrates thread groups that originally belonged to different operators and required independent scheduling and execution into the same fused thread group for sequential execution, thereby eliminating the synchronization caused by scheduling switching between operators and improving the continuity of task execution. Furthermore, during the execution of the fused thread group, the intermediate results generated in the preceding calculation stage are stored in the local memory allocated to the fused thread group, from which subsequent calculation stages can directly read. This mechanism effectively avoids global access overhead and reduces the power consumption of data transmission from a higher global access level to a lower local access level. Therefore, the fusion scheme provided by the embodiments of this disclosure, with thread groups as the fusion granularity, improves the continuous utilization of hardware computing resources and enhances the overall execution efficiency and energy efficiency of computing tasks by reducing scheduling and data transmission overhead through logical integration and localized data transmission.

[0172] For example, the fusion unit 100 is further configured to: execute the computation logic corresponding to the first thread group in the first fusion computation logic to generate a first intermediate result; store the first intermediate result in a first local memory allocated to the first fusion thread group; and read the first intermediate result from the first local memory and execute the computation logic corresponding to the second thread group in the first fusion computation logic based on the first intermediate result.

[0173] Optionally, the first fusion thread group is executed by the stream processor, and the stream processor allocates first local memory for the first fusion thread group.

[0174] Furthermore, embodiments of this disclosure also provide static compile-time determination paths and dynamic runtime profiling determination paths for constructing a thread group-level operator fusion determination technology system. For example, the fusion unit 100 is further configured to: determine the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group. Determining the inclusion relationship may include: determining the inclusion relationship based on the configuration information of the first thread group and the second thread group; or, determining the inclusion relationship based on the runtime record information of the first thread group and the second thread group.

[0175] Specifically, in some embodiments of this disclosure, the fusion unit 100 is further configured to: determine a first coordinate range of the output data block of the first thread group in the output tensor of the first operator; determine a second coordinate range of the input data block of the second thread group in the output tensor; and determine the above-mentioned inclusion relationship based on the first coordinate range and the second coordinate range.

[0176] Based on this, as an option, a first coordinate range can be determined according to the logical identifier of the first thread group and the output data layout of the first operator, wherein the output data layout of the first operator includes at least one of the following: the shape of the output tensor, the dimensional order of the output tensor, and the memory arrangement format of the output tensor. Alternatively, a third coordinate range in the input tensor of the second operator can be determined according to the logical identifier of the second thread group and the input data layout of the second operator; and the third coordinate range can be mapped to the output tensor based on the data flow relationship between the first and second operators to determine the second coordinate range, wherein the input data layout of the second operator includes at least one of the following: the shape of the input tensor, the dimensional order of the input tensor, and the memory arrangement format of the input tensor.

[0177] Optionally, during the static compilation phase of the parallel computing device, the compiler can determine the first coordinate range based on the logical identifier of the first thread group and the output data layout of the first operator. The output data layout of the first operator may include information such as the shape of the output tensor of the first operator, the dimensional order of the output tensor of the first operator, and the memory arrangement format of the output tensor of the first operator. This information is known at compile time, and the compiler can establish a correspondence between the logical identifier of the first thread group and the coordinate range of the output tensor of the first operator based on this information. This allows for the accurate calculation of the first coordinate range occupied by the output data block produced by the first thread group in the output tensor of the first operator. This eliminates the need to rely on the runtime data of the parallel computing device to determine the first coordinate range of the output data block of the first thread group in the output tensor of the first operator.

[0178] It should be noted that the output tensor of the first operator is a logical container for the computation result of the first operator. It consists of a multi-dimensional array, and its size can be determined by parameters such as the input size of the first operator, the convolution kernel size, the stride, and padding. The logical identifier of the first thread group can be an index that uniquely identifies the first thread group among all thread groups of the first operator. The compiler can use this index and the dimension information of the first thread group to determine the output region that the first thread group is responsible for computing in the output tensor of the first operator. In other words, the compiler can use this index and the dimension information of the first thread group to determine the first coordinate range corresponding to the output data block produced by the first thread group. The output data layout of the first operator describes the mapping relationship between the output tensor of the first operator in logical coordinates and physical storage, and includes at least: the shape of the output tensor of the first operator, the dimensional order of the output tensor of the first operator, and the memory arrangement format of the output tensor of the first operator. The shape of the output tensor of the first operator represents the size of each dimension of the output tensor, such as height H, width W, and channels C. The dimensional order of the output tensor of the first operator represents the arrangement order of each logical dimension of the output tensor in memory. The memory arrangement format of the output tensor of the first operator represents the refined conventions of the storage layout, thereby further refining the storage layout of the output tensor of the first operator, such as whether to use vectorized storage, whether to perform memory alignment, and whether to use tensorized storage format.

[0179] The second thread group is the execution unit of the second operator, and its calculation process requires reading an input data block. This input data block has a corresponding coordinate range in the input tensor of the second operator, which is referred to in this disclosure as the third coordinate range. Similar to the determination of the first coordinate range, the compiler can determine the third coordinate range of the input data block of the second thread group in the input tensor of the second operator based on the logical identifier of the second thread group and the input data layout of the second operator.

[0180] Since the input tensor of the second operator becomes the output tensor of the first operator through simple transformations such as subset taking and transposition, a clear data flow relationship exists between them. This data flow relationship can be understood as data dependencies and execution order constraints between operators. In a system composed of multiple operators, this refers to the logical dependencies and execution order constraints formed through the generation, flow, and consumption of data. This data flow relationship can be represented by edges in a computation graph (data flow graph) and can be fully obtained at compile time through static analysis.

[0181] Therefore, the compiler can map the third coordinate range forward along the data flow relationship to the output tensor space of the first operator, thereby obtaining the second coordinate range corresponding to the input data block of the second thread group of the second operator in the output tensor of the first operator. This mapping process is also static, relying only on the data flow topology of each operator, the data layout of each operator, and the logical identifier of each thread group, without relying on the data of the parallel computing device at runtime.

[0182] After obtaining the first and second coordinate ranges, the compiler performs a set inclusion relationship determination. If the first coordinate range contains the second coordinate range, or in other words, every coordinate point in the second coordinate range lies within the first coordinate range, it indicates that all input data required by the second thread group of the second operator can be directly provided by the output data blocks produced by the first thread group of the first operator, and there is an input-output dependency relationship between the two based on data blocks. Therefore, in response to the first coordinate range containing the second coordinate range, the computational logic of the first thread group and the computational logic of the second thread group are merged to generate the first fused computational logic.

[0183] The compiler can compile the first fusion computation logic into a kernel function, which is then executed by the first fusion thread group. During execution, computation logic that originally belonged to different operators and required independent scheduling is integrated into the same fusion thread group and executed sequentially. Intermediate results generated in the preceding computation stage are directly stored in the local memory allocated to the fusion thread group, and subsequent computation stages can directly read from this local memory and continue computation.

[0184] In other embodiments of this disclosure, the fusion unit 100 is further configured to: obtain a first tag and a second tag from historical execution data, wherein the first tag at least indicates the logical identifier of a first thread group and the range of the output data block of the first thread group, and the second tag at least indicates the logical identifier of a second thread group and the range of the input data block of the second thread group; and determine, based on the first tag and the second tag, the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group.

[0185] Based on this, as an option, the fusion unit 100 can also be configured to: determine a first coordinate range in the output tensor of the first operator based on a first label; determine a third coordinate range in the input tensor of the second operator based on a second label; map the third coordinate range to the output tensor based on the data flow relationship between the first and second operators to determine a second coordinate range in the output tensor of the second thread group; and determine that the range of the output data block of the first thread group includes the range of the input data block of the second thread group based on the first coordinate range including the second coordinate range.

[0186] This implementation identifies the data dependency between the first thread group and the second thread group based on historical execution data generated during the operation of the parallel computing device, and determines whether the two meet the fusion conditions. It should be noted that, as another implementation path of this disclosure, it can be independent of the aforementioned static compile-time analysis path, or it can complement the other, to adapt to the needs of different optimization scenarios.

[0187] Specifically, during one or more actual executions of an operator, the parallel computing device can record the execution information of each thread group and organize it into persistent tag data to form historical execution data. A first tag is associated with the first thread group of the first operator and records at least the logical identifier of the first thread group and the range of output data blocks actually produced by the first thread group in the executed computation. A second tag is associated with the second thread group of the second operator and records at least the logical identifier of the second thread group and the range of input data blocks actually consumed by the second thread group in the executed computation.

[0188] Optionally, determining the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group based on the first label and the second label may include: determining a first coordinate range of the output data block of the first thread group in the output tensor of the first operator based on the first label; determining a third coordinate range of the input data block of the second thread group in the input tensor of the second operator based on the second label; mapping the third coordinate range to the output tensor based on the data flow relationship between the first operator and the second operator to determine a second coordinate range of the input data block of the second thread group in the output tensor; and determining that the range of the output data block of the first thread group includes the range of the input data block of the second thread group based on the first coordinate range including the second coordinate range.

[0189] For example, when it is necessary to perform fusion optimization on the first operator and the second operator, the optimizer of the parallel computing device (e.g., compiler, runtime optimization system, or other dedicated profiling tool) can read the above tags from historical execution data and perform the following determination process: Based on the information recorded by the first tag, determine the first coordinate range occupied by the output data block of the first thread group in the output tensor of the first operator; Based on the information recorded by the second tag, determine the third coordinate range occupied by the input data block of the second thread group in the input tensor of the second operator; According to the data flow relationship between the first operator and the second operator, map the third coordinate range to the output tensor space of the first operator, thereby obtaining the second coordinate range corresponding to the input data block of the second thread group in the output tensor of the first operator, wherein the data flow relationship can be obtained statically from the computation graph or from the tensor dependency relationship recorded in historical execution; Perform set inclusion relationship determination: In response to the first coordinate range completely covering the second coordinate range, determine that the range of the output data block of the first thread group includes the range of the input data block of the second thread group, and there is an input-output dependency relationship based on data blocks between the two, satisfying the fusion condition.

[0190] Furthermore, in embodiments of this disclosure, two threads belonging to different operators that have a data block inclusion relationship are merged to form a fusion operator with a single fusion thread group as the execution carrier. Based on this, some embodiments of this disclosure further provide a recursively expandable fusion computation chain construction mechanism, enabling fusion optimization to span multiple operators and multiple thread groups, forming a longer sequence of continuous computation tasks.

[0191] Specifically, when the input data block range of a merged thread group can be contained within the output data block range of a thread group of another upstream operator, the upstream thread group is ready to be further merged with the merged thread group. Similarly, when the output data block range of a merged thread group can contain the input data block range of a thread group of another downstream operator, the downstream thread group is also ready to be further merged. By recursively applying the inclusion relationship determination criterion, the first fusion calculation logic of the first fusion thread group can be updated, gradually expanding the fusion operator into a fusion calculation chain containing multiple original operators and multiple original thread groups.

[0192] Alternatively, in some embodiments of this disclosure, the fusion unit 100 may also be configured to: estimate the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the third thread group in response to the range of the output data block of the third thread group of the third operator including the range of the input data block of the first thread group; and fuse the first fusion calculation logic with the calculation logic of the third thread group in response to the storage capacity being equal to or less than the capacity of the first local memory, so as to update the first fusion calculation logic.

[0193] Based on this, the computing unit 200 can also be configured to: execute the updated first fusion computing logic by the first fusion thread group, including: executing the computing logic corresponding to the third thread group in the updated first fusion computing logic to generate a second intermediate result; and storing the second intermediate result in the first local memory, wherein, based on the second intermediate result read from the first local memory, the computing logic corresponding to the first thread group in the first fusion computing logic is executed.

[0194] Furthermore, the fusion unit 100 may also be configured to: in response to the range of the output data block of the Mth thread group including the range of the input data block of the Nth thread group, estimate the storage capacity required to fuse the current first fusion calculation logic with the calculation logic of the Mth thread group, wherein the current first fusion calculation logic includes the calculation logic of the Nth thread group, and the input of the Nth thread group is the input of the current first fusion calculation logic; and in response to the storage capacity being equal to or less than the capacity of the first local memory, fuse the calculation logic of the Mth thread group with the current first fusion calculation logic to update the first fusion calculation logic, wherein M and N are different and are both positive integers greater than 2.

[0195] Alternatively, the fusion unit 100 may also be configured to: estimate the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the third thread group in response to the range of the output data block of the third thread group including the range of the input data block of the first thread group; and store the result of executing the calculation logic of the first fusion calculation logic corresponding to the second thread group in a global memory different from the first local memory in response to the storage capacity being greater than the capacity of the first local memory.

[0196] Based on this, if the output data block range of a thread group in a higher-level operator can include the input data block range of the current fusion thread group, the above judgment and expansion logic can be repeated to integrate the higher-level thread groups one by one into the current fusion computation logic. This forward recursive expansion application uses the judgment criterion that the output data block range of the thread group in the upstream operator includes the input data block range of the thread group in the downstream operator as a unified link interface, and strings together the thread groups that originally belonged to different operators into a long sequence of computation tasks that are executed sequentially within a single fusion thread group. With the thread group as the fusion granularity, through logical integration and localized data transfer, the continuous utilization rate of hardware computing resources is improved while reducing scheduling and data transfer overhead, and the overall execution efficiency and energy efficiency of the computation task are enhanced.

[0197] For example, the fusion unit 100 may also be configured to: in response to the range of the output data block of the P-th thread group including the range of the input data block of the Q-th thread group, estimate the storage capacity required to fuse the current first fusion calculation logic with the calculation logic of the Q-th thread group, wherein the current first fusion calculation logic includes the calculation logic of the P-th thread group, and the output of the P-th thread group is the output of the current first fusion calculation logic; and in response to the storage capacity being equal to or less than the capacity of the first local memory, fuse the calculation logic of the Q-th thread group with the current first fusion calculation logic to update the first fusion calculation logic, wherein P and Q are different and are both positive integers greater than 2.

[0198] Furthermore, in some embodiments of this disclosure, the fusion unit 100 may also be configured to: estimate the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the fourth thread group in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator; and fuse the first fusion calculation logic with the calculation logic of the fourth thread group in response to the storage capacity being equal to or less than the capacity of the first local memory, so as to update the first fusion calculation logic.

[0199] Based on this, the computing unit 200 can also be configured to: execute the updated first fusion computing logic by the first fusion thread group, including: storing the third intermediate result generated by executing the computing logic corresponding to the second thread group in the first fusion computing logic to the first local memory; and reading the third intermediate result from the first local memory, and executing the computing logic corresponding to the fourth thread group in the first fusion computing logic based on the third intermediate result.

[0200] Furthermore, the fusion unit 100 may also be configured to: in response to the range of the output data block of the second thread group including the range of the input data block of the fourth thread group of the fourth operator, estimate the storage capacity required after fusing the first fusion calculation logic with the calculation logic of the fourth thread group; and in response to the storage capacity being greater than the capacity of the first local memory, store the result generated by executing the calculation logic of the first fusion calculation logic corresponding to the second thread group in a global memory different from the first local memory.

[0201] Unlike local memory, which can only be accessed by a thread group within a single stream processor, global memory can be accessed by multiple stream processors. For example, a first fused thread group can be executed by a first stream processor among multiple stream processors, and the first stream processor can allocate first local memory for the first fused thread group. Furthermore, in the presence of multiple fused thread groups, each of the multiple fused thread groups can be executed by multiple stream processors, in which case all multiple stream processors can access global memory.

[0202] Based on this, if the output data block range of the current fusion thread group can encompass the input data block range of a thread group in a downstream operator, the above-mentioned judgment and expansion logic can be repeatedly applied to integrate the downstream thread groups one by one into the current fusion computation logic. This backward recursive expansion, using the judgment criterion that the output data block range of the thread group in the upstream operator includes the input data block range of the thread group in the downstream operator as a unified link interface, strings together thread groups originally belonging to different operators into a long sequence of computation tasks executed sequentially within a single fusion thread group. By using thread groups as the fusion granularity, through logical integration and localized data transfer, the continuous utilization of hardware computing resources is improved, and the overall execution efficiency and energy efficiency of the computation tasks are enhanced, while reducing scheduling and data transfer overhead.

[0203] The thread-group-based operator fusion device 2000 can be configured to execute the forward expansion fusion step and the backward expansion fusion step described above. By recursively applying the inclusion relationship determination criterion, it updates the first fusion calculation logic of the first fusion thread group, gradually expanding the fusion operator into a fusion calculation chain containing multiple original operators and multiple original thread groups. The specific process is not detailed here. In the fusion thread group, the generation and consumption of all intermediate results are completed within the local memory of the fusion thread group, without involving access to global memory, thus completely eliminating read / write latency and bandwidth consumption introduced by cross-operator data transfer.

[0204] In some embodiments of this disclosure, the computing unit 200 may also be configured to: store a first intermediate result in a first storage space of a first local memory; and after the first intermediate result is read, store a third intermediate result in the first storage space to overwrite the first intermediate result.

[0205] Storage lifetime can be understood as the time interval from when an intermediate result is written to local memory until it is last read. In the fusion computation chain executed by the same fusion thread group, the storage lifetimes of intermediate results generated at different computation stages are often serial and non-overlapping. In other words, the space reuse mechanism of local memory allocated to the fusion thread group supports time-sharing multiplexing. Intermediate results can be created, transferred, and consumed in the first local memory, and then their storage space can be reused or released by subsequent computations.

[0206] Leveraging the aforementioned non-overlapping lifecycle characteristic, multiple intermediate results with non-overlapping lifecycles can be allocated to the same local memory address space through compile-time static lifecycle analysis or runtime dynamic allocation strategies. A later intermediate result can directly overwrite the storage area occupied by a previous intermediate result, without needing to reserve dedicated storage space for each intermediate result. Thus, the actual local memory usage no longer depends on the number of intermediate results, but rather on the size of the largest simultaneously surviving intermediate result. Since the lifecycles of intermediate results in the fused computation chain are serial, this number of "simultaneously surviving" results is typically 1 or 2. Therefore, the local memory usage is compressed to a constant level, essentially decoupled from the fused chain length, making the expansion of the fused chain unconstrained by the physical limitations of local memory capacity.

[0207] In some embodiments of this disclosure, the fusion unit 100 is further configured to: determine the size of the input data block of the first thread group; determine the data reuse rate based on the ratio between the size of the input data block of the first thread group and the size of the input data block of the second thread group; and, in response to the data reuse rate being greater than or equal to a predetermined threshold, fuse the computational logic of the first thread group and the computational logic of the second thread group to generate the first fused computational logic.

[0208] As described above, the embodiments of this disclosure use the inclusion of the input data block range of the thread group of the upstream operator within the thread group of the downstream operator as the criterion for determining whether two thread groups can be merged, which ensures the correctness of the data dependency relationship. However, in some computing scenarios, there may be significant overlap in computation between two thread groups that satisfy the inclusion relationship. In other words, the first thread group needs to load and process a large input data block to generate its output data block, while the second thread group only consumes a small sub-region from it. In this case, although fusion is feasible, the benefits of data reuse may be offset by the excessive overhead of repeated computation, and the overall benefit is not significant.

[0209] To address the aforementioned issues, the embodiments of this disclosure further introduce a quantitative metric—data reuse rate—to evaluate the actual benefits of fusion operations. Specifically, for a first thread group and a second thread group that satisfy an inclusion relationship, the input data block size of the first thread group reflects the total amount of raw data that the first thread group needs to load and process to produce its output data block; the input data block size of the second thread group reflects the total amount of data that the second thread group needs to consume from the output data block of the first thread group to complete its computational task. Since the output data block of the first thread group is generated from its input data block through computation, a definite mapping relationship exists between the two. With fixed operator types and operator parameters, the proportional relationship between the input data block sizes of the first thread group and the second thread group can characterize the trade-off between the computational cost incurred by the first thread group to support the computation of the second thread group and the data reuse benefits obtained by the second thread group.

[0210] Optionally, the ratio of the input data block size of the second thread group to the input data block size of the first thread group is defined as the data reuse rate. A higher data reuse rate indicates that a higher proportion of the input data processed by the first thread group is directly reused by the second thread group, resulting in more significant fusion benefits. Conversely, a low data reuse rate indicates that the first thread group needs to process a large amount of input data to produce a small output data block, while the second thread group only consumes a very small portion. The data locality benefits brought by fusion may not offset the significant computational overhead of the first thread group itself. Therefore, this embodiment introduces a predetermined threshold to quantify the data reuse rate. If the data reuse rate is greater than or equal to the predetermined threshold, it is determined that the fusion of the first and second thread groups has positive benefits, and a fusion operation is performed. If the data reuse rate is less than the predetermined threshold, even if the data blocks of the first and second thread groups are contained within each other, fusion may not be performed to avoid situations where the benefits are less than expected.

[0211] In some embodiments of this disclosure, the fusion unit 100 is further configured to: in response to the range of the output data block of the fifth thread group of the first operator including the range of the input data block of the sixth thread group of the fifth operator, fuse the computational logic of the fifth thread group and the computational logic of the sixth thread group to generate a second fused computational logic; and the computation unit 200 is further configured to: execute the second fused computational logic by the second fused thread group, including: executing the computational logic corresponding to the fifth thread group in the second fused computational logic to generate a fourth intermediate result; storing the fourth intermediate result in a second local memory allocated to the second fused thread group; and reading the fourth intermediate result from the second local memory and executing the computational logic corresponding to the sixth thread group in the second fused computational logic based on the fourth intermediate result.

[0212] Furthermore, the first fusion thread group and the fourth fusion thread group can be executed by the same stream processor, and the first local memory and the second local memory can be allocated to the first fusion thread group and the fourth fusion thread group respectively by the same stream processor.

[0213] The foregoing embodiments of this disclosure have detailed how to achieve thread-group-level operator fusion for a specific thread group pair—the first thread group a of the first operator A and the second thread group b of the second operator B—through steps such as inclusion relationship determination, fusion logic generation, and fusion thread group execution. However, in actual computational graphs, there are a large number of fusionable thread group pairs distributed across different operators and data flow paths, without direct dependencies or conflicts between them. To fully explore the potential for global fusion optimization, the embodiments of this disclosure further provide the ability to construct parallel and independent fusion instances.

[0214] In this embodiment, the fifth thread group of the first operator and the sixth thread group of the fifth operator have an input-output dependency relationship based on data blocks. The second fusion computing logic is another fusion computing logic generated independently of the first fusion computing logic; the two are logically completely equivalent and do not interfere with each other. The second fusion thread group is the execution entity that executes the second fusion computing logic; it can be scheduled in parallel with the first fusion thread group and executed independently. The second fusion thread group can be allocated an independent second local memory to store intermediate results (e.g., a fourth intermediate result) generated during its execution.

[0215] Furthermore, the second fusion computing logic and its execution entity, the second fusion thread group, constructed in this embodiment are fully compatible with the aforementioned technical features of this disclosure. In other words, the second fusion computing logic is not an isolated basic fusion instance, but can be fully incorporated into the fusion optimization technology system constructed in this disclosure. Mechanisms such as fusion expansion mechanisms, local memory space reuse mechanisms, and data reuse rate determination mechanisms are all adapted to the second fusion computing logic and its execution entity, the second fusion thread group.

[0216] Therefore, according to at least one embodiment of this disclosure, the range of the output data block of the first thread group in response to the first operator includes the range of the input data block of the second thread group of the second operator. The computational logic of the first thread group and the computational logic of the second thread group are merged to generate fused computational logic. This integrates thread groups that originally belonged to different operators and required independent scheduling and execution into a single fused thread group for sequential execution, thus eliminating the synchronization caused by scheduling switching between operators and improving the continuity of task execution. Furthermore, during the execution of the fused thread group, intermediate results generated in the preceding computation stage are stored in the local memory allocated to the fused thread group, from which subsequent computation stages can directly read. This mechanism effectively avoids global access overhead, reducing the power consumption of data transmission from a higher global access level to a lower local access level. Therefore, the fusion scheme provided by the embodiments of this disclosure, using thread groups as the fusion granularity, improves the continuous utilization of hardware computing resources and enhances the overall execution efficiency and energy efficiency of computing tasks by reducing scheduling and data transmission overhead through logical integration and localized data transfer.

[0217] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0218] Figure 12 A schematic block diagram of an example electronic device 3000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobility methods, such as personal digital processing, cellular phones, smartphones, wearable devices, and other similar computing methods. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0219] like Figure 12 As shown, the electronic device 3000 includes a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 3000. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0220] Multiple components in electronic device 3000 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 3000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0221] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various means and processes described above, such as the thread group-based operator fusion method. For example, in some embodiments, the thread group-based operator fusion method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 3000 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of the thread group-based operator fusion method described above can be performed. Alternatively, in other embodiments, computing unit 301 may be configured to perform a thread group-based operator fusion method by any other suitable means (e.g., by means of firmware).

[0222] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input method, and at least one output method, and transferring data and instructions to the storage system, the at least one input method, and the at least one output method.

[0223] Program code for implementing the apparatus of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing method, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0224] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, method, or apparatus. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, methods, or apparatuses, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0225] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display method for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing method (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of methods can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0226] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0227] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0228] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, apparatuses, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0229] Various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technological improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0230] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0231] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An operator fusion method based on thread groups, comprising: The range of the output data block of the first thread group in response to the first operator includes the range of the input data block of the second thread group of the second operator. The computational logic of the first thread group and the computational logic of the second thread group are fused to generate the first fused computational logic. The range of the output data block of the first thread group is the first coordinate range of the output data block of the first thread group in the output tensor of the first operator, and the range of the input data block of the second thread group is the second coordinate range of the input data block of the second thread group in the output tensor. as well as A first fusion thread group is executed by a single stream processor. The first fusion thread group is used to execute the first fusion computation logic, wherein the intermediate results of the first fusion computation logic are stored in a first local memory allocated by the single stream processor for the first fusion thread group. The first thread group and the second thread group belong to different operators and are independently scheduled and executed before fusion. After fusion, they are integrated into the same first fusion thread group and executed sequentially.

2. The method according to claim 1, wherein, The first fusion calculation logic is executed by the first fusion thread group, including: Execute the computation logic corresponding to the first thread group in the first fusion computation logic to generate a first intermediate result; Store the first intermediate result in the first local memory; and The first intermediate result is read from the first local memory, and the calculation logic corresponding to the second thread group in the first fusion calculation logic is executed based on the first intermediate result.

3. The method according to claim 1, wherein, The method further includes: Determining the inclusion relationship between the range of the output data block of the first thread group and the range of the input data block of the second thread group includes: The inclusion relationship is determined based on the configuration information of the first thread group and the second thread group; or... The inclusion relationship is determined based on the execution record information of the first thread group and the second thread group.

4. The method according to claim 1, wherein, The method further includes determining the first coordinate range by: Based on the logical identifier of the first thread group and the output data layout of the first operator, the first coordinate range is determined. The output data layout of the first operator includes at least one of the following: the shape of the output tensor, the dimensional order of the output tensor, and the memory arrangement format of the output tensor.

5. The method according to claim 4, wherein, The method further includes determining the second coordinate range by: Based on the logical identifier of the second thread group and the input data layout of the second operator, determine the third coordinate range of the input data block of the second thread group in the input tensor of the second operator; as well as Based on the data flow relationship between the first operator and the second operator, the third coordinate range is mapped to the output tensor to determine the second coordinate range. The input data layout of the second operator includes at least one of the following: the shape of the input tensor, the dimensional order of the input tensor, and the memory arrangement format of the input tensor.

6. The method according to claim 3, wherein, Determining the inclusion relationship based on the execution record information of the first thread group and the second thread group includes: Obtain a first label and a second label from historical execution data, wherein the first label at least indicates the logical identifier of the first thread group and the range of the output data block of the first thread group, and the second label at least indicates the logical identifier of the second thread group and the range of the input data block of the second thread group; and The inclusion relationship is determined based on the first tag and the second tag.

7. The method according to claim 6, wherein, Determining the inclusion relationship based on the first tag and the second tag includes: Based on the first label, determine the first coordinate range; Based on the second label, determine the third coordinate range of the input data block of the second thread group in the input tensor of the second operator; Based on the data flow relationship between the first operator and the second operator, the third coordinate range is mapped to the output tensor to determine the second coordinate range.

8. The method according to claim 1, wherein, The method further includes: The range of the output data block of the third thread group in response to the third operator includes the range of the input data block of the first thread group; the storage capacity required after fusing the first fusion computation logic with the computation logic of the third thread group is estimated; and In response to the storage capacity being equal to or less than the capacity of the first local memory, the first fused computing logic is merged with the computing logic of the third thread group to update the first fused computing logic.

9. The method according to claim 8, wherein, The method further includes: The updated first fusion calculation logic is executed by the first fusion thread group, including: Execute the updated computation logic corresponding to the third thread group in the first fusion computation logic to generate a second intermediate result; and Store the second intermediate result in the first local memory. Specifically, based on the second intermediate result read from the first local memory, the computation logic corresponding to the first thread group in the first fusion computation logic is executed.

10. The method according to claim 2, wherein, The method further includes: In response to the fact that the range of the output data block of the Mth thread group includes the range of the input data block of the Nth thread group, the storage capacity required to fuse the current first fusion computation logic with the computation logic of the Mth thread group is estimated, wherein the current first fusion computation logic includes the computation logic of the Nth thread group, and the input of the Nth thread group is the input of the current first fusion computation logic; and In response to the storage capacity being equal to or less than the capacity of the first local memory, the computational logic of the Mth thread group is merged with the current first fusion computational logic to update the first fusion computational logic, wherein M and N are different and are both positive integers greater than 2.

11. The method according to claim 2, wherein, The method further includes: The range of the output data block of the third thread group in response to the third operator includes the range of the input data block of the first thread group; the storage capacity required after fusing the first fusion computation logic with the computation logic of the third thread group is estimated; and In response to the storage capacity being greater than the capacity of the first local memory, the result of executing the computation logic corresponding to the second thread group in the first fusion computing logic is stored in a global memory that is different from the first local memory.

12. The method according to claim 11, wherein, The method further includes: The first fusion thread group is executed by the stream processor, and the stream processor allocates the first local memory to the first fusion thread group. The global memory is accessed by multiple stream processors.

13. The method according to claim 2, wherein, The method further includes: In response to the fact that the range of the output data block of the second thread group includes the range of the input data block of the fourth thread group of the fourth operator, the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the fourth thread group is estimated; and In response to the storage capacity being equal to or less than the capacity of the first local memory, the first fused computing logic is merged with the computing logic of the fourth thread group to update the first fused computing logic.

14. The method according to claim 13, wherein, The method further includes: The updated first fusion calculation logic is executed by the first fusion thread group, including: The third intermediate result generated by the computation logic corresponding to the second thread group in the updated first fusion computation logic is stored in the first local memory; and The third intermediate result is read from the first local memory, and the calculation logic corresponding to the fourth thread group in the first fusion calculation logic is executed based on the third intermediate result.

15. The method according to claim 2, wherein, The method further includes: In response to the fact that the range of the output data block of the Pth thread group includes the range of the input data block of the Qth thread group, the storage capacity required to fuse the current first fusion computation logic with the computation logic of the Qth thread group is estimated, wherein the current first fusion computation logic includes the computation logic of the Pth thread group, and the output of the Pth thread group is the output of the current first fusion computation logic; and In response to the storage capacity being equal to or less than the capacity of the first local memory, the computational logic of the Qth thread group is merged with the current first fusion computational logic to update the first fusion computational logic, wherein P and Q are different and are both positive integers greater than 2.

16. The method according to claim 2, wherein, The method further includes: In response to the fact that the range of the output data block of the second thread group includes the range of the input data block of the fourth thread group of the fourth operator, the storage capacity required to fuse the first fusion calculation logic with the calculation logic of the fourth thread group is estimated; and In response to the storage capacity being greater than the capacity of the first local memory, the result of executing the computation logic corresponding to the second thread group in the first fusion computing logic is stored in a global memory that is different from the first local memory.

17. The method according to claim 14, wherein, The method further includes: Store the first intermediate result in the first storage space of the first local memory; and After the first intermediate result is read, the third intermediate result is stored in the first storage space to overwrite the first intermediate result.

18. The method according to claim 2, wherein, The step of fusing the computational logic of the first thread group with the computational logic of the second thread group to generate the first fused computational logic includes: Determine the size of the input data block for the first thread group; The data reuse rate is determined based on the ratio between the size of the input data block of the first thread group and the size of the input data block of the second thread group; and In response to the data reuse rate being greater than or equal to a predetermined threshold, the computational logic of the first thread group and the computational logic of the second thread group are merged to generate the first fused computational logic.

19. The method according to claim 2, wherein, The method further includes: In response to the fact that the range of the output data block of the fifth thread group of the first operator includes the range of the input data block of the sixth thread group of the fifth operator, the computational logic of the fifth thread group and the computational logic of the sixth thread group are fused to generate a second fused computational logic; and The second fusion calculation logic is executed by the second fusion thread group, including: The calculation logic corresponding to the fifth thread group in the second fusion calculation logic is executed to generate a fourth intermediate result; Store the fourth intermediate result in a second local memory allocated to the second fusion thread group; and The fourth intermediate result is read from the second local memory, and the calculation logic corresponding to the sixth thread group in the second fusion calculation logic is executed based on the fourth intermediate result.

20. The method according to claim 19, wherein, The method further includes: The first fusion thread group and the second fusion thread group are executed by the same stream processor, and the same stream processor allocates the first local memory and the second local memory to the first fusion thread group and the second fusion thread group, respectively.

21. An operator fusion device based on thread groups, comprising: A fusion unit is configured to, in response to the fact that the range of the output data block of the first thread group of the first operator includes the range of the input data block of the second thread group of the second operator, fuse the computational logic of the first thread group and the computational logic of the second thread group to generate a first fusion computational logic, wherein the range of the output data block of the first thread group is a first coordinate range of the output data block of the first thread group in the output tensor of the first operator, and the range of the input data block of the second thread group is a second coordinate range of the input data block of the second thread group in the output tensor; as well as The computing unit is configured to execute a first fusion thread group by a single stream processor. The first fusion thread group executes the first fusion computing logic, wherein intermediate results of the first fusion computing logic are stored in a first local memory allocated by the single stream processor for the first fusion thread group. The first thread group and the second thread group belong to different operators and are independently scheduled and executed before fusion. After fusion, they are integrated into the same first fusion thread group and executed sequentially.

22. An electronic device, comprising: A processor and a memory, wherein the memory stores at least one instruction, at least one program, a code set, or an instruction set, wherein the at least one instruction, the at least one program, the code set, or the instruction set is loaded and executed by the processor to implement the fusion method as described in any one of claims 1 to 20.

23. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the fusion method of any one of claims 1 to 20.

24. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the fusion method according to any one of claims 1 to 20.

Images