Compiling processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a compilation processing method and device, electronic equipment and storage medium. The method comprises: obtaining a computing kernel and hardware configuration information in a program to be compiled; determining a target processing unit for performing subsequent operation and a target storage space corresponding to output data; determining a local storage space associated with the second processing unit as the target storage space, and converting a first encoding layout corresponding to the output data into a second encoding layout matching a storage layout of the target storage space; determining storage mapping information corresponding to the output data according to the target storage space and the second encoding layout corresponding to the output data; and generating a data operation compilation result corresponding to the computing kernel according to the storage mapping information. In the presence of operator fusion, the present disclosure automatically completes storage space selection, encoding layout conversion and storage mapping information generation, so that the output data of the matrix multiplication operation is directly stored in the local storage space and directly consumed by the subsequent processing unit.

Description

Technical Field

[0001] This disclosure relates to the field of compilation technology, and in particular to a compilation processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In dedicated accelerators for AI (Artificial Intelligence) inference, the computing units are equipped with multi-level on-chip storage, which may include data local memory (DLM).

[0003] When AI compilers (such as Triton, XLA, etc.) are performing compilation, if the output of matrix multiplication is consumed by subsequent operators, it is necessary to manually write fusion operators or use frameworks such as TVM (Tensor Virtual Machine) to specify layout transformation strategies to handle the storage of output tensors.

[0004] However, in operator fusion scenarios, existing compilation frameworks typically allocate the output of matrix multiplication operations to the shared memory space by default. This causes the vector operation unit to need to read the data from the shared memory space first when consuming the result, which increases on-chip data transfer overhead and access latency. Summary of the Invention

[0005] This disclosure provides a compilation processing method, apparatus, electronic device, and storage medium that can automatically allocate matrix multiplication output data to local storage space associated with subsequent consumption processing units and complete automatic encoding layout conversion, thereby reducing on-chip data transfer overhead and access latency, and improving local storage utilization and fusion operator execution efficiency.

[0006] According to one aspect of this disclosure, a compilation processing method is provided, comprising:

[0007] Obtain the computing kernel and hardware configuration information in the program to be compiled. The computing kernel includes matrix multiplication and subsequent operations that consume the output data of the matrix multiplication. The hardware configuration information includes a first processing unit that performs the matrix multiplication, at least one second processing unit, and local storage space associated with each of the second processing units.

[0008] Based on the data dependencies between operations in the computing kernel and the hardware configuration information, the target processing unit for executing the subsequent operations and the target storage space corresponding to the output data are determined.

[0009] When the target processing unit for performing the subsequent operations is the second processing unit, the local storage space associated with the second processing unit is determined as the target storage space, and the first encoding layout corresponding to the output data is converted into a second encoding layout that matches the storage layout of the target storage space.

[0010] Based on the target storage space and the second encoding layout corresponding to the output data, determine the storage mapping information corresponding to the output data;

[0011] The data operation compilation result corresponding to the computing kernel is generated based on the storage mapping information. The data operation compilation result includes data writing information for writing the output data to the target storage space and data reading information for reading the output data from the target storage space.

[0012] Optionally, determining the target processing unit for executing the subsequent operations and the target storage space corresponding to the output data based on the data dependencies between operations in the computing kernel and the hardware configuration information includes:

[0013] The subsequent operations for consuming the output data are determined based on the data dependencies.

[0014] Based on the operation type of the subsequent operation and the hardware configuration information, a target processing unit for performing the subsequent operation is determined from the at least one second processing unit;

[0015] The local storage space associated with the determined second processing unit is determined as the target storage space corresponding to the output data.

[0016] Optionally, the hardware configuration information also includes at least one third processing unit, at least one candidate storage space other than the local storage space, and the data access relationship between the third processing unit and the candidate storage space, wherein no direct data access path is set between the third processing unit and the local storage space;

[0017] The step of determining the target processing unit for executing the subsequent operations and the target storage space corresponding to the output data based on the data dependencies between operations in the computing kernel and the hardware configuration information further includes:

[0018] When the target processing unit for performing the subsequent operation is the third processing unit, according to the data access relationship between the target processing unit for performing the subsequent operation and each of the candidate storage spaces, a candidate storage space with a direct data access path between it and the third processing unit is selected from the at least one candidate storage space, and the selected candidate storage space is determined as the target storage space corresponding to the output data.

[0019] Optionally, the first processing unit is a tensor processing unit;

[0020] The second processing unit is an execution unit that performs at least one of vector operations and scalar operations;

[0021] The local storage space is the data local storage space bound to the execution unit;

[0022] A data reading path is set between the execution unit and the corresponding local data storage space.

[0023] Optionally, the matrix multiplication operation is a matrix multiplication operation or a matrix multiplication accumulation operation;

[0024] The execution unit includes a vector register, and the data read path includes a data transfer path between the local data storage space and the vector register;

[0025] The subsequent operations include at least one of vector addition, vector multiplication, and scalar operations.

[0026] Optionally, the output data is tensor data; the first encoding layout is a block encoding layout corresponding to the block calculation method of the matrix multiplication operation, and the first encoding layout represents the block arrangement relationship of each data element in the output data;

[0027] The target storage space adopts a row-major order storage layout;

[0028] The second encoding layout represents the mapping relationship between each data element in the output data and the storage location in the target storage space.

[0029] Optionally, the second encoding layout is used to establish a mapping relationship between each data element and the corresponding storage location in the target storage space according to the row index and column index of each data element in the output data, so that data elements with the same row index are mapped to consecutive storage locations in column index order.

[0030] Optionally, the target processing unit includes a plurality of second processing units, and the step of determining the storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data includes:

[0031] Based on the size of the output data in the row dimension and the number of target processing units, the output data is divided along the row dimension into multiple data regions corresponding to the multiple target processing units respectively;

[0032] The local storage space associated with each of the multiple target processing units is determined as multiple target storage spaces;

[0033] Establish a mapping relationship between each of the data regions and each of the target storage spaces;

[0034] Based on the second encoding layout and the mapping relationship, the target storage space and storage location information corresponding to the data elements in each data area are determined.

[0035] Optionally, dividing the output data into multiple data regions corresponding to the multiple target processing units along the row dimension based on the size of the output data in the row dimension and the number of target processing units includes:

[0036] Based on the size of the output data in the row dimension and the number of target processing units, determine the row index range corresponding to each target processing unit;

[0037] The output data is divided according to the index range of each row to obtain multiple data regions.

[0038] Optionally, determining the target storage space and storage location information corresponding to the data elements in each of the data regions based on the second encoding layout and the mapping relationship includes:

[0039] Based on the mapping relationship between each data region and each target storage space, the target storage space identifier corresponding to each data region is determined;

[0040] Based on the second encoding layout, determine the relative storage offset of the data elements in each data area in the corresponding target storage space;

[0041] Based on the target storage space identifier and the storage relative offset, the first processing unit generates data writing information for writing each of the data regions to the corresponding target storage space.

[0042] Optionally, the computation kernel includes at least two matrix multiplication operations, and the method further includes:

[0043] For each of the matrix multiplication operations, the survival interval of the corresponding output data is determined, and the survival interval is determined at least based on the generation position of the corresponding output data and the position of the last consumption of the output data;

[0044] Based on the survival interval corresponding to each output data, a storage area in the corresponding target storage space is allocated to each output data.

[0045] Optionally, allocating a storage area in the corresponding target storage space for each output data according to the liveness interval corresponding to each output data includes:

[0046] When the live intervals corresponding to any two output data do not overlap, the two output data reuse at least a portion of the storage area in the target storage space;

[0047] When the live intervals corresponding to any two output data overlap, non-overlapping storage areas are allocated to the two output data.

[0048] Optionally, allocating a storage region in the corresponding target storage space for the output data includes:

[0049] The corresponding storage space requirements are determined based on the data size of the output data.

[0050] At least one candidate free storage region is determined based on the address range of the allocated storage regions in the target storage space;

[0051] From the at least one candidate free storage area, determine a free storage area whose size meets the storage space requirement, and allocate the storage area corresponding to the output data in the determined free storage area;

[0052] If no free storage area with a size that meets the storage space requirement exists in at least one candidate free storage area, the storage area corresponding to the output data shall be allocated outside the allocated storage area in the target storage space.

[0053] Optionally, the storage mapping information includes the target storage space identifier and the relative storage offset;

[0054] The step of determining the storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data includes:

[0055] Determine the initial relative offset of the output data in the target storage space;

[0056] Based on the second encoding layout and the index position of each data element in the output data, determine the element local offset of each data element relative to the initial relative offset;

[0057] Based on the initial relative offset and the element local offset, the storage relative offset of each data element in the target storage space is determined.

[0058] Optionally, the method further includes:

[0059] When compiling the subsequent operations, the target storage space where the input data corresponding to the subsequent operations is located is determined;

[0060] The target storage space identifier and the relative storage offset are passed to the data read operation corresponding to the subsequent operation;

[0061] Based on the target storage space identifier and the relative storage offset, a data read instruction corresponding to the subsequent operation is generated.

[0062] Optionally, the method further includes:

[0063] When the compiler intermediate representation does not include an explicit data write operation corresponding to the data read operation, the data read operation is generated as a memory read instruction with volatile memory access semantics.

[0064] During the compilation optimization process, the memory read instruction is not deleted because it lacks a corresponding explicit data write operation.

[0065] According to one aspect of this disclosure, a compilation processing apparatus is provided, comprising: an acquisition module, configured to acquire a computing kernel and hardware configuration information in a program to be compiled, wherein the computing kernel includes matrix multiplication and subsequent operations consuming the output data of the matrix multiplication, and the hardware configuration information includes a first processing unit for performing the matrix multiplication, at least one second processing unit, and local storage spaces respectively associated with each of the second processing units; a storage space determination module, configured to determine a target processing unit for performing the subsequent operations and a target storage space corresponding to the output data based on the data dependencies between operations in the computing kernel and the hardware configuration information; and a layout conversion module, configured to, in order to perform the subsequent operations on the target processing unit... When the unit is the second processing unit, the local storage space associated with the second processing unit is determined as the target storage space, and the first encoding layout corresponding to the output data is converted into a second encoding layout that matches the storage layout of the target storage space; the mapping information determination module is used to determine the storage mapping information corresponding to the output data according to the target storage space and the second encoding layout corresponding to the output data; the compilation result generation module is used to generate the data operation compilation result corresponding to the computing kernel according to the storage mapping information, the data operation compilation result including data writing information for writing the output data to the target storage space and data reading information for reading the output data from the target storage space.

[0066] According to one aspect of this disclosure, an electronic device is proposed, the electronic device including a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for implementing connection communication between the processor and the memory, wherein the program is executed by the processor to implement the compilation processing method described above.

[0067] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores one or more programs, which can be executed by one or more processors to implement the compilation processing method described above.

[0068] This disclosure proposes a compilation processing method, apparatus, electronic device, and storage medium. The method includes: acquiring a computational kernel and hardware configuration information in a program to be compiled; the computational kernel includes matrix multiplication operations and subsequent operations consuming the output data of the matrix multiplication operations; the hardware configuration information includes a first processing unit performing the matrix multiplication operations, at least one second processing unit, and local storage spaces associated with each second processing unit; determining the target processing unit for performing subsequent operations and the target storage space corresponding to the output data based on the data dependencies between operations in the computational kernel and the hardware configuration information; when the target processing unit for performing subsequent operations is a second processing unit, determining the local storage space associated with the second processing unit as the target storage space, and converting the first encoding layout corresponding to the output data into a second encoding layout matching the storage layout of the target storage space; determining storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data; and generating a data operation compilation result corresponding to the computational kernel based on the storage mapping information, the data operation compilation result including data write information for writing output data to the target storage space and data read information for reading output data from the target storage space. This disclosure performs static data dependency analysis on the computational kernel during compilation to automatically identify the subsequent operations and execution units of the matrix multiplication output data. When it is determined that the subsequent operation will be executed by a second processing unit, the local storage space associated with the second processing unit is automatically identified as the target storage space for the output data. The original first encoding layout of the output data, oriented towards the shared space, is converted to a second encoding layout that matches the storage characteristics of the local storage space, thereby generating a data operation compilation result containing precise storage mapping information. This allows the matrix multiplication output data to be directly read and consumed by subsequent processing units from the local storage space, avoiding the transfer of result data in shared storage. This reduces on-chip data transfer overhead and access latency while improving local storage utilization and fusion operator execution efficiency.

[0069] Furthermore, subsequent operations on the output data are determined through data dependencies, and the target processing unit is determined from at least one second processing unit according to the operation type of the subsequent operations. The local storage space associated with the target processing unit is then determined as the target storage space, thereby establishing a direct correspondence between the output data and the target processing unit and its associated local storage space during compilation.

[0070] Furthermore, when the target processing unit for subsequent operations is the third processing unit, a candidate storage space with a direct data access path between it and the third processing unit is selected as the target storage space. This expands the applicability of the compilation processing method to heterogeneous hardware configurations and avoids allocating the output data to a storage space without a direct data access path.

[0071] Furthermore, the first processing unit can be a tensor processing unit, and the second processing unit can be an execution unit that performs vector and scalar operations. The local storage space can be a data local storage space bound to the execution unit. A data reading path is set between the execution unit and the corresponding data local storage space, thereby enabling the data operation compilation results to directly correspond to the physical hardware resources of the heterogeneous accelerator.

[0072] Furthermore, by covering matrix multiplication and accumulation operations and explicitly defining the data reading path, including the data transfer path between the local data storage space and the vector register, the fusion result after the tensor processing unit performs matrix multiplication and accumulation can be directly output to the local data storage space. The execution unit can then directly load data from the local data storage space into the vector register to perform subsequent operations, reducing computational overhead.

[0073] Furthermore, by converting the output data from a block-based encoding layout corresponding to the block-based computation method of matrix multiplication to a second encoding layout that matches the row-major order storage layout of the target storage space, the second encoding layout represents the mapping relationship between each data element and the storage location in the target storage space. This makes the storage format of the output data match the reading characteristics of the target processing unit, eliminating the cost of manual layout adaptation.

[0074] Furthermore, by establishing a mapping relationship based on the row and column indices of each data element through the second encoding layout, data elements with the same row index are mapped to consecutive storage locations in order of column index, thereby improving the spatial locality when the target processing unit reads the output data sequentially.

[0075] Furthermore, based on the size of the output data in the row dimension and the number of target processing units, the output data is divided into multiple data regions along the row dimension, each corresponding to a multiple target processing unit. A mapping relationship is established between each data region and each target storage space, thereby achieving a one-to-one correspondence between each data region and the corresponding target storage space when multiple processing units consume data in parallel, avoiding resource contention caused by multiple processing units accessing the same storage region.

[0076] Furthermore, the row index range corresponding to each target processing unit is determined based on the size of the output data in the row dimension and the number of target processing units, and the output data is divided according to each row index range. This allows the compiler to automatically deduce the data boundaries responsible for each target processing unit based on the matrix shape and parallel configuration, without having to manually specify the block size or processing unit mapping relationship in the computing kernel.

[0077] Furthermore, the target storage space identifier corresponding to each data region is determined according to the mapping relationship between each data region and each target storage space, and the relative storage offset of each data element in the corresponding target storage space is determined according to the second encoding layout, thereby generating data writing information, so that the data operation compilation result can directly guide the first processing unit to write each data region into the corresponding target storage space according to the precise storage location information.

[0078] Furthermore, for at least two matrix multiplication operations included in the computing kernel, the corresponding survival interval is determined according to the generation position of each output data and the position of the last consumed output data, and the corresponding storage area in the target storage space is allocated to each output data according to each survival interval, thereby providing a lifecycle-based fine-grained storage plan for the intermediate output data of multiple matrix multiplication operations.

[0079] Furthermore, when the survival intervals corresponding to any two output data do not overlap, the two output data reuse at least a portion of the storage area in the target storage space. When the survival intervals corresponding to any two output data overlap, non-overlapping storage areas are allocated to the two output data. Thus, under the limited capacity of the target storage space, by reusing the storage area when the survival intervals do not overlap, more matrix multiplication operations can be supported to use the target storage space simultaneously under the limited capacity.

[0080] Furthermore, based on the data size of the output data, the corresponding storage space requirements are determined. First, at least one candidate free storage region is determined from the address range of the allocated storage regions in the target storage space. From the at least one candidate free storage region, a free storage region whose size meets the storage space requirements is determined and allocated, so as to achieve fine-grained allocation of the target storage space and improve the utilization efficiency of limited storage space.

[0081] Furthermore, by determining the initial relative offset of the output data in the target storage space, and based on the second encoding layout and the index position of each data element in the output data, the element local offset of each data element relative to the initial relative offset is determined, thereby determining the storage relative offset of each data element in the target storage space, thus achieving element-level precise addressing and providing directly usable address calculation parameters for generating data reading information.

[0082] Furthermore, during the compilation process of subsequent operations, the target storage space identifier and the relative storage offset are passed to the data read operation corresponding to the subsequent operation, and the data read instruction corresponding to the subsequent operation is generated based on the target storage space identifier and the relative storage offset. This establishes an address information transmission channel between the matrix multiplication operation side and the subsequent operation side, ensuring that the target processing unit can correctly locate and read the output data based on the storage mapping information.

[0083] Furthermore, when the compiler's intermediate representation does not include an explicit data write operation corresponding to the data read operation, a memory read instruction with volatile memory access semantics will be generated through the data read operation. During the compilation optimization process, the memory read instruction will not be deleted based on the lack of a corresponding explicit data write operation. This prevents the loading operation of the target storage space from being mistakenly identified as dead code and deleted during the compilation optimization stage, ensuring that the target processing unit can correctly read the output data in the target storage space and guaranteeing the correctness of the compilation results of data operations.

[0084] Other features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description or may be learned by practicing the disclosure. The objectives and other advantages of this disclosure may be realized and obtained by means of the structures particularly pointed out in the description, claims and drawings. Attached Figure Description

[0085] The accompanying drawings are provided to further understand the technical solutions of this disclosure and constitute a part of the specification. They are used together with the embodiments of this disclosure to explain the technical solutions of this disclosure and do not constitute a limitation on the technical solutions of this disclosure.

[0086] Figure 1 This is an architecture diagram of the compilation processing method applied in the embodiments of this disclosure;

[0087] Figure 2 This is a main flowchart of the compilation process of one embodiment of this disclosure;

[0088] Figure 3 This is a schematic diagram of step S220 of an embodiment of the present disclosure;

[0089] Figure 4 This is a schematic diagram of step S220 of another embodiment of the present disclosure;

[0090] Figure 5 This is a schematic diagram of step S230 of an embodiment of the present disclosure;

[0091] Figure 6 This is a schematic diagram of step S230 of another embodiment of the present disclosure;

[0092] Figure 7 This is a schematic diagram of step S250 of an embodiment of the present disclosure;

[0093] Figure 8 This is a schematic diagram of the compilation process of one embodiment of this disclosure;

[0094] Figure 9 This is a schematic diagram of the structure of a compilation processing apparatus proposed in one embodiment of this disclosure;

[0095] Figure 10 This is a schematic diagram of the structure of an electronic device proposed in one embodiment of the present disclosure. Detailed Implementation

[0096] To make the objectives, technical solutions, and advantages of this disclosure clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and are not intended to limit the scope of this disclosure.

[0097] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations:

[0098] Artificial Intelligence Compiler (AI Compiler): A specialized compiler designed for AI computational workloads, used to convert AI model descriptions (such as computation graphs) into executable code for target hardware. Unlike general-purpose compilers, AI compilers typically use tensors as the basic data type, computation graphs as intermediate representations, and perform specific optimizations for typical AI operators such as matrix multiplication and convolution, including operator fusion, automatic tuning, and memory layout optimization.

[0099] The program to be compiled: The program to be compiled in the AI ​​compiler refers to the computation graph structure inside the deep learning model, which describes how multidimensional tensors flow through each operator node. Essentially, it is a topological blueprint of data flow.

[0100] A computational kernel is a basic computational unit received and processed by the compiler, typically corresponding to a complete computational logic defined using a high-level programming interface. In this embodiment, the computational kernel includes at least matrix multiplication and subsequent operations that consume the output data of the matrix multiplication, serving as the basic object for the compiler to perform data dependency analysis, memory allocation, and code generation.

[0101] Matrix multiplication is a mathematical operation that involves multiplying two two-dimensional matrices to obtain a result matrix. In artificial intelligence accelerators, this operation can be performed by a dedicated tensor processing unit. In the embodiments of this application, matrix multiplication can be a matrix multiplication accumulation operation (such as A×B+C, A×B+C×D, etc.), that is, after the matrix multiplication is completed, the result is fused with a third matrix or accumulator.

[0102] Hardware configuration information refers to a set of static parameters describing the hardware architecture characteristics of the target accelerator, used to guide the compiler in resource allocation and code generation during compilation. In the embodiments of the application, the hardware configuration information includes at least a first processing unit that performs matrix multiplication, a second processing unit that performs vector / scalar operations, the local storage space associated with the second processing unit, other candidate storage spaces, and the data access path relationships between each processing unit and the storage space.

[0103] Local Memory (DLM): This is a dedicated data storage space integrated within a high-performance embedded processor, primarily used to store frequently accessed critical data. The core characteristic of DLM is its extremely low access latency, enabling the processor to perform read and write operations with zero wait cycles, thereby effectively improving the system's real-time processing capabilities.

[0104] Encoding layout refers to the arrangement of tensor data in storage space, defining the mapping rules between tensor logical indices, such as [row] and [col], and physical storage addresses. On parallel processors such as GPUs (Graphics Processing Units), different encoding layouts affect the memory access continuity of thread bundles, thereby affecting bandwidth utilization and overall execution performance.

[0105] Block-based storage layout: A layout method that divides tensors into several fixed-size tiles and organizes them for storage. Tensors are stored contiguously in small tiles, with elements within a tile being adjacent in physical memory, and elements between tiles skipping physical memory locations. This block-based storage layout can improve data locality, allowing multiple threads within the same thread bundle to reuse the same small tile of data, thereby improving computational efficiency. However, it sacrifices memory access continuity during the write phase.

[0106] Row-major order (RMO): A layout that arranges tensors contiguously in row-major order. For a two-dimensional matrix, elements in the same row are stored contiguously at physical memory addresses; for a multi-dimensional tensor, elements in the innermost dimension are stored contiguously. RMO ensures that the target addresses accessed by adjacent threads within the same thread bundle are contiguous, matching the memory merging mechanism of GPU hardware and thus improving read / write bandwidth utilization.

[0107] Processing unit: A processing unit in computer hardware is a functional unit that performs computational operations. In heterogeneous AI accelerator architectures, processing units are divided into different types according to their functions, including tensor processing engines dedicated to performing tensor operations such as matrix multiplication and convolution, and execution units dedicated to performing vector and scalar operations. Different types of processing units have different instruction sets, register files, and memory access interfaces. In the embodiments of this application, the first processing unit is a tensor processing engine, the second processing unit is an execution unit, and the third processing unit is another type of processing unit.

[0108] Data access relationships refer to the mapping between the physical connections and access permissions between processing units and storage spaces at various levels in a heterogeneous computing architecture. This mapping characterizes whether a specific processing unit can directly perform read or write operations on a specific storage space. If a direct data path exists between the processing unit and the storage space, the processing unit can directly access the storage space; otherwise, it cannot directly access it and must access it through other storage layers or processing units.

[0109] Vector addition is a mathematical operation that involves adding corresponding elements of two vectors (one-dimensional arrays) of the same dimension to generate a result vector. At the hardware implementation level, vector addition is typically executed in parallel by a processing unit supporting vector instruction sets via vector registers. In this embodiment, vector addition is performed by a second processing unit as a subsequent operation on the output data of the matrix multiplication operation.

[0110] Vector multiplication is a mathematical operation that involves multiplying corresponding elements of two vectors (one-dimensional arrays) of the same dimension to produce a result vector. In hardware implementation, vector multiplication is typically performed in parallel by a processing unit supporting vector instruction sets (SVCs) using vector registers.

[0111] Consumption (being consumed): In the computation graph constructed by the compiler, this refers to the output tensor of one operator being used as input data by another operator. The consumed output tensor is an intermediate result and needs to reside in the storage space until all consumption operations are completed; the unconsumed output tensor is the final result and needs to be exported to external storage.

[0112] Warp: The basic execution scheduling unit in a GPU, consisting of a fixed number (usually 32) of threads. Threads within the same warp execute the same sequence of instructions in lockstep mode, but process their own independent data. Warp is the basic granularity for GPU memory coalescing.

[0113] Volatile memory access semantics refers to setting corresponding memory read operations as memory accesses that need to be actually executed in the compiler. This prevents the compilation optimization process from deleting the memory read operation as an invalid or redundant memory access due to the lack of an explicit data write operation corresponding to the memory read operation, or the inability to determine the data source of the memory read operation based on data dependency analysis in the intermediate representation.

[0114] In related technologies, when compilers process computation kernels containing matrix multiplication operations, they typically allocate the output data of matrix multiplication operations to shared memory by default. This forces subsequent vector operation units to read the output data from shared memory, increasing on-chip data read paths and access latency. Furthermore, directly storing the output data of matrix multiplication operations in the local memory associated with vector operation units for direct consumption often requires manually writing fusion operators or manually specifying storage allocation strategies, making it impossible to automatically determine the storage strategy based on the data dependencies between operations in the computation kernel.

[0115] Based on this, this disclosure proposes a compilation processing method, apparatus, electronic device, and storage medium. By performing static data dependency analysis on the computational kernel during the compilation process, it automatically identifies subsequent consumption operations of the matrix multiplication output data and, based on this, determines the target processing unit and corresponding target storage space for executing subsequent operations, in conjunction with hardware configuration information. Simultaneously, it converts the first encoding layout corresponding to the output data into a second encoding layout that conforms to the storage characteristics of the local storage space, thereby determining the storage mapping information and generating the corresponding data operation compilation results. This allows the output data of the matrix multiplication operation to be directly written into the local storage space by the target processing unit and directly read and executed from the local storage space by subsequent operations, reducing access overhead and latency, achieving automatic adaptation to heterogeneous storage spaces, and reducing manual costs.

[0116] Figure 1 This is a schematic diagram of the system architecture of the compilation processing method according to an embodiment of the present disclosure. It includes: object terminal 110, network 120, gateway 130 and compilation server 140.

[0117] The object terminal 110 is a device used by the object to input source code to be compiled. The source code to be compiled includes a matrix calculation kernel, which includes matrix multiplication operations and subsequent operations on the output data of the matrix multiplication operations. The object terminal 110 can take various forms, such as a desktop computer, a laptop computer, or a workstation. The object terminal 110 communicates with the compilation server 140 via network 120 to transmit the source code to be compiled and the compilation request.

[0118] Gateway 130 enables network interconnection at the transport layer, acting as a converter between different communication protocols or data formats. The source code to be compiled and the compilation request sent by object terminal 110 to compilation server 140 are forwarded through gateway 130; the compilation results returned by compilation server 140 to object terminal 110 are also forwarded through gateway 130.

[0119] The compilation server 140 is a computer system that runs an artificial intelligence compiler and executes compilation processing methods. The compilation server 140 obtains the computational kernel from the source code to be compiled and the hardware configuration information of the target hardware; the target hardware includes a first processing unit, at least one second processing unit, and local storage spaces associated with each second processing unit. Based on the data dependencies between operations in the computational kernel and the hardware configuration information, the compilation server 140 determines the target processing unit for executing subsequent operations and the target storage space corresponding to the output data, performs compilation processing on the computational kernel, and generates corresponding data operation compilation results. The data operation compilation results are used to instruct the target processing unit to store the output data of the matrix multiplication operation into the target storage space and to read the output data from the target storage space to execute subsequent operations. The compilation server 140 then sends the data operation compilation results to the target hardware for execution.

[0120] Overall Implementation of the Compilation Processing Method in the Embodiments of this Disclosure

[0121] This disclosure provides a compilation processing method, applied to a compilation processing method apparatus, with reference to... Figure 2 The compilation and processing methods include:

[0122] Step S210: Obtain the computing kernel and hardware configuration information in the program to be compiled. The computing kernel includes matrix multiplication operations and subsequent operations that consume the output data of matrix multiplication operations.

[0123] Step S220: Based on the data dependencies between operations in the computing kernel and the hardware configuration information, determine the target processing unit for executing subsequent operations and the target storage space corresponding to the output data.

[0124] In step S230, when the target processing unit for performing subsequent operations is the second processing unit, the local storage space associated with the second processing unit is determined as the target storage space, and the first encoding layout corresponding to the output data is converted into a second encoding layout that matches the storage layout of the target storage space.

[0125] Step S240: Determine the storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data.

[0126] Step S250: Generate the data operation compilation result corresponding to the computing kernel based on the storage mapping information. The data operation compilation result includes data writing information for writing output data to the target storage space and data reading information for reading output data from the target storage space.

[0127] In step S210, the hardware configuration information includes a first processing unit that performs matrix multiplication, at least one second processing unit, and local storage space associated with each second processing unit.

[0128] Optionally, the program to be compiled can be written in a high-level programming language, in which the computing kernel is defined through a specific interface. The source code file to be compiled can be read from the target terminal's file system, or the source code data stream transmitted over a network by the target terminal can be received to obtain the program to be compiled.

[0129] In step S220, data dependencies refer to the relationships formed between operations in the computation kernel due to data transfer, such as the relationship between the output data of a matrix multiplication operation and subsequent operations that consume that output data. Data dependencies can be obtained by traversing the input and output tensors of each operation in the computation kernel and establishing mapping relationships.

[0130] Further, the subsequent operations of the consumed output data are determined based on the data dependencies; the target processing unit for performing the subsequent operations is determined from at least one second processing unit based on the operation type and hardware configuration information of the subsequent operations; and the local storage space associated with the determined second processing unit is determined as the target storage space corresponding to the output data.

[0131] Specifically, data dependencies can be determined by constructing a data flow graph of the computation kernel, representing each operation as a node in the graph, and the data transfer relationships between operations as edges between nodes. By traversing the data flow graph and identifying the subsequent operation nodes to which the output data of the matrix multiplication operation flows, the subsequent operations consuming that output data are determined. After determining the subsequent operations, the processing units supporting those subsequent operations are identified as target processing units by querying the functional scope of each processing unit in the hardware configuration information. Furthermore, the storage space associated with the target processing unit is identified as the target storage space by querying the association between the processing unit and the storage space in the hardware configuration information.

[0132] For example, if the compiler determines, based on data dependencies, that the output data of a matrix multiplication operation is consumed by an addition operation, then it determines that the addition operation is the subsequent operation. Finally, if the type of the addition operation is determined to be a vector operation, then based on the operation type and hardware configuration information, a second processing unit is selected from the second processing units as the target processing unit to perform the subsequent operation.

[0133] In this embodiment, the subsequent operations of the output data are determined by the data dependency relationship, and the target processing unit is determined from at least one second processing unit according to the operation type of the subsequent operation. Then, the local storage space associated with the target processing unit is determined as the target storage space, thereby establishing a direct correspondence between the output data and the target processing unit and its associated local storage space during the compilation period.

[0134] Furthermore, the hardware configuration information also includes at least one third processing unit, at least one candidate storage space other than the local storage space, and the data access relationship between the third processing unit and the candidate storage space. No direct data access path is set between the third processing unit and the local storage space.

[0135] When the target processing unit for performing subsequent operations is the third processing unit, the candidate storage space with a direct data access path between the target processing unit and each candidate storage space is selected from at least one candidate storage space according to the data access relationship between the target processing unit and each candidate storage space, and the selected candidate storage space is determined as the target storage space corresponding to the output data.

[0136] In one implementation, the first processing unit may be a Tensor Processing Engine (TPE) for performing matrix multiplication operations. The output data of the matrix multiplication operation is shared storage space by default, or transferred to other storage levels via shared storage space. The second processing unit may be an Execution Unit (EU) for performing vector or scalar operations, associated with the local data storage space, and provided with a direct data access path. The third processing unit may be a Special Function Unit (SFU), which does not have a direct data access path with the local data storage space, but can exchange data through shared storage space or global memory.

[0137] Candidate storage spaces may include shared storage space and global memory. Shared storage space is an on-chip shared cache that can be accessed by multiple processing units within the computing unit. Global memory is a large-capacity off-chip storage located outside the computing unit, and each processing unit accesses the global content through the storage controller.

[0138] Please refer to Figure 3This illustration shows a hardware architecture in a parallel computing cluster provided by an exemplary embodiment of this application, which includes computing units (CUs) within a block processing cluster (BPC). Each computing unit includes a memory controller (MC) for enabling read / write interactions between the computing unit and external global memory. The computing unit also includes an Intra-CUInterconnect network that connects to the memory controller, the Execution Units (EUs), the Tensor Processing Engine (TPE), and the Static Random Access Memory Interface (SRAM IF) to transmit routing data and control signals. Specifically, the EUs are used for scalar or vector operations; the data-local memory provides ultra-high bandwidth private data access; the tensor processing engines are used for intensive computations such as matrix multiplication and addition or graphics sampling; and the SRAM IF is used for high-bandwidth data interaction with shared memory (SMEM) or scratchpad memory (SPM) outside the block processing cluster. In addition, the computing unit also includes a control engine (CE) for BPC-level instruction scheduling, task distribution, and state management; an interface description specification for declaring the interfaces within the computing unit and between the computing unit and external modules; and a hub interface (hub IF) as the network from the BPC to the global interconnection network, responsible for communicating with other BPCs or higher-level controllers.

[0139] In addition to the computing units, the block processing cluster also includes global memory (GMEM, Global Memory Bundle) as well as shared memory and note storage, which are delayed on-chip shared memory pools that can be read by all computing units within the block processing cluster.

[0140] Specifically, if the target processing unit for subsequent operations does not belong to the second processing unit, the target processing unit is determined as the third processing unit. The compiler traverses each candidate storage space in the hardware configuration information and sequentially queries whether there is a data access relationship between the third processing unit and each candidate storage space. The compiler then selects candidate storage spaces with direct data access paths from at least one candidate storage space and determines the selected candidate storage space as the target storage space corresponding to the output data.

[0141] In this embodiment, when the target processing unit for subsequent operations does not belong to the second processing unit, the target storage space is determined based on the data access relationship between the third processing unit and each candidate storage space. This expands the applicability of the compilation processing method to heterogeneous hardware configurations and avoids allocating output data to storage spaces without direct data access paths.

[0142] Furthermore, the first processing unit is a tensor processing unit; the second processing unit is an execution unit that performs at least one of vector operations and scalar operations. The local storage space is a data local storage space bound to the execution unit. A data read path is established between the execution unit and the corresponding data local storage space.

[0143] For example, the first processing unit performs a matrix multiplication operation A×B for the operation A×B+C, obtaining the output data of the matrix multiplication operation. This output data is consumed by the subsequent operation +C, which is a vector addition operation performed by the second processing unit. According to the hardware configuration information, the local data storage space bound to the second processing unit is determined as the target storage space corresponding to the output data. The first processing unit stores the output data of the matrix multiplication operation in the local data storage space, and the execution unit reads the output data from the local data storage space through the data read path and performs the vector addition operation.

[0144] This embodiment provides possible specific processing unit types for each processing unit, and sets a data reading path between the execution unit and the corresponding local data storage space, thereby enabling the data operation compilation results to directly correspond to the physical hardware resources of the heterogeneous accelerator.

[0145] Furthermore, matrix multiplication can be either a matrix multiplication operation or a matrix multiplication accumulation operation. For example, a matrix multiplication operation can be A×B or A×B+C×D, etc.

[0146] The execution unit includes a vector register, and the data read path includes the data transfer path between the local data storage space and the vector register.

[0147] The subsequent operations include at least one of vector addition, vector multiplication, and scalar operations. For example, if the subsequent operation is vector addition, the matrix multiplication operation is A×B, and the subsequent operation is (A×B)+C; if the subsequent operation is vector multiplication, the matrix multiplication operation is A×B, and the subsequent operation is (A×B)⊙D; if the subsequent operation is scalar operation, the matrix multiplication operation is A×B, and the subsequent operation can be (A×B)×3.

[0148] In one possible implementation, the candidate operation for matrix multiplication is vector addition. The initialization operation of the matrix multiplication operation in the compiler's intermediate representation is obtained, and the initial value of the accumulator associated with the matrix multiplication operation is determined based on the initialization operation. If the initial value is not zero, the target processing unit corresponding to the output data of the matrix multiplication operation is determined as the second processing unit, and the target storage space is the local data storage space.

[0149] In another possible implementation, the storage attribute information of the output data corresponding to the matrix multiplication operation is obtained. Based on the storage attribute information, it is determined whether the output data is allocated to global memory. If the output data is not allocated to global memory, the target processing unit corresponding to the output data of the matrix multiplication operation is determined to be the second processing unit, and the target storage space is the local data storage space.

[0150] Please refer to Figure 4 After obtaining the output data of the matrix multiplication operation, the initial value of the accumulator associated with the matrix multiplication operation is obtained, and then it is determined whether the initial value of the accumulator is 0. If not, the target storage space is determined to be the local data storage space. If so, the storage attribute information of the output data is further obtained, and it is determined whether the output data is stored in the external storage space based on the storage attribute information. If so, the target storage space is determined not to be the local data storage space. If not, the target storage space is determined to be the local data storage space.

[0151] In this embodiment, by covering matrix multiplication and accumulation operations and explicitly defining the data reading path as including the data transmission path between the local data storage space and the vector register, the fusion result after the tensor processing unit performs matrix multiplication and accumulation can be directly output to the local data storage space. The execution unit can directly load data from the local data storage space into the vector register to perform subsequent operations, thereby reducing computational overhead.

[0152] In step S230, the first encoding layout is the encoding layout corresponding to the output data before conversion, which is used to characterize the arrangement relationship of each data element in the data. It can be obtained by parsing the output layout attribute configured by the matrix multiplication operation in the calculation kernel.

[0153] The second encoding layout can refer to an encoding layout that matches the storage layout of the target storage space, used to characterize the mapping relationship between each element in the output data and the storage location in the target storage space.

[0154] Optionally, it can also be derived based on the storage layout characteristics of the target storage space, the matrix shape of the output data, and the parallel configuration.

[0155] For example, if the target processing unit for performing subsequent vector addition operations is determined to be the second processing unit, the compiler can generate the second encoding layout by deriving and calculating through linear layout based on the storage layout type adopted by the target storage space, the matrix shape of the output data, and the number of target processing units. If the second encoding layout is a row-major order storage layout, the second encoding layout will divide the output data in the row dimension so that each data element is mapped to a contiguous storage location in the target storage space in row-major order.

[0156] Furthermore, the output data is tensor data. The first encoding layout can be a block encoding layout corresponding to the block computation method of matrix multiplication, and the first encoding layout represents the block arrangement relationship of each data element in the output data.

[0157] If the target storage space adopts a row-major order storage layout, then the second encoding layout can represent the mapping relationship between each data element in the output data and the storage location in the target storage space.

[0158] Please refer to Figure 5 This diagram illustrates the data layout and access patterns of the same M×N two-dimensional matrix in shared storage and local storage. The first encoding layout corresponds to the block-based computation method of matrix multiplication, representing the block arrangement of data elements in the output data. The second encoding layout represents the mapping relationship between each data element in the output data and its storage location in the local storage space, which uses a row-major order storage layout. In the first encoding layout, the original matrix is ​​logically divided into a 2-row × 3-column tile grid (corresponding to tiles 0 to 5 in the diagram), with data within each tile strictly following row-major order for continuous storage. The arrows in the diagram illustrate a zigzag data access pattern. Taking tile 0 as an example, the arrow first reads from left to right along the first row, then folds back to the next row and continues reading to the right. This access method helps optimize the bandwidth utilization of shared storage.

[0159] Refer to the above Figure 5 In the second encoding layout, the mapping relationship between each data element in the output data and its storage location in the target storage space is represented. The M×N two-dimensional matrix is ​​divided along its row dimension into two data regions corresponding to the two target processing units. Each target processing unit's associated local storage space stores its corresponding data region. One local storage space stores the top M / 2 rows of the matrix, with a size of (M / 2)×N, while the other local storage space stores the bottom M / 2 rows, also with a size of (M / 2)×N. Taking one of the local storage spaces as an example, its internal data is still stored in row-major order. The arrows in the diagram indicate the global continuous access process from the top left corner to the bottom right corner.

[0160] In this embodiment, the output data is converted from a block encoding layout corresponding to the block calculation method of matrix multiplication to a second encoding layout that matches the row-major order storage layout of the target storage space. The second encoding layout represents the mapping relationship between each data element and the storage location in the target storage space. This makes the storage format of the output data match the reading characteristics of the target processing unit, eliminating the cost of manual layout adaptation.

[0161] Furthermore, the second encoding layout is used to establish a mapping relationship between each data element and its corresponding storage location in the target storage space according to the row index and column index of each data element in the output data, so that data elements with the same row index are mapped to consecutive storage locations in column index order.

[0162] Specifically, the matrix shape of the output data is parsed to determine the row index range and column index range of each data element in the output data. Based on the row-major layout adopted by the target storage space, the compiler establishes a mapping relationship between the data elements and the corresponding storage locations in the target storage space according to the row index and column index of each data element in the output data. For data elements with the same row index, the compiler allocates contiguous storage addresses according to the column index order, so that data elements with the same row index should be set to contiguous storage locations according to the column index order.

[0163] For example, the output data is a 2x3 matrix containing 6 data elements. The row index of the data elements can be determined to be 0 or 1, and the column index can be 0, 1 or 2. The storage location can be calculated as: storage location = base address + row index × number of columns + column index. The data element with row index 0 has column indices of 0, 1, and 2, respectively. The corresponding storage locations are base address, base address + 1, and base address + 2, respectively. The storage locations should be set consecutively according to the column index order.

[0164] Optionally, a mapping relationship between each data element and its corresponding storage location in the target storage space is established according to the row and column indices of each data element in the output data. The compiler groups data elements with the same index and allocates consecutive offsets within each group in ascending order of index, so that data elements with the same row index are mapped to consecutive storage locations in column index order.

[0165] Please refer to Figure 6 This demonstrates the correspondence between data elements and their storage locations in the target storage space. The output data is a 3x4 matrix. The index of the matrix is ​​traversed to obtain the row and column indices of each element. Then, based on the second encoding layout, taking a data element with row index 0 and column indices 0, 1, 2, and 3 as an example, offsets of 0, 1, 2, and 3 are assigned to each data element in sequence, and the elements are mapped to consecutive storage locations in the target storage space starting from the base address according to the column index order.

[0166] In this embodiment, a mapping relationship is established according to the row index and column index of each data element through the second encoding layout, so that data elements with the same row index are mapped to consecutive storage locations in the order of column index, thereby improving the spatial locality when the target processing unit reads the output data sequentially.

[0167] In step S240, the local storage spaces associated with multiple target processing units are identified as multiple target storage spaces. A mapping relationship is established between each data region and each target storage space to determine the target storage space identifier corresponding to each data region. Based on the encoding layout and this mapping relationship, the compiler calculates the storage location information of data elements in each data region to determine the target storage space and storage location information corresponding to each data element, which serves as storage mapping information.

[0168] Optionally, the storage mapping information corresponding to the output data is determined by determining the starting relative offset of the output data in the target storage space. Based on the encoding layout and the index position of each data element in the output data, the relative position of each data element is determined by calculating the element-local offset of each data element relative to the starting relative offset. Based on the starting relative offset and the element-local offset, the storage relative offset of each data element in the target storage space is determined as the storage mapping information.

[0169] Furthermore, the target processing unit includes multiple second processing units. Based on the size of the output data along the row dimension and the number of target processing units, the output data is divided along the row dimension into multiple data regions, each corresponding to one of the multiple target processing units. For example, in the case of multiple second processing units, for a 4x3 matrix, row 0 and row 1 correspond to the first second processing unit, and row 2 and row 3 correspond to the second second processing unit.

[0170] Subsequently, the local storage spaces associated with multiple target processing units are identified as multiple target storage spaces, and a mapping relationship is established between each data region and each target storage space. Based on the second encoding layout and the mapping relationship, the target storage space and storage location information corresponding to the data elements in each data region are determined.

[0171] For example, if the output data is an M×N matrix and the target processing unit includes a second processing unit, then the size of the output data in the row dimension is M, the number of target processing units is 2, and the output data is divided into 2 data regions along the row dimension, each data region including M / 2 rows. Refer to the above appendix. Figure 5The local storage spaces associated with the two target processing units are identified as the two target storage spaces. Then, a mapping relationship is established between each data area and each target storage space, so that the upper M / 2 row corresponds to the first target storage space and the lower M / 2 row corresponds to the second target storage space. Based on the second encoding layout and the mapping relationship, the target storage space and storage location information corresponding to the data elements in each data area are determined.

[0172] In this embodiment, based on the size of the output data in the row dimension and the number of target processing units, the output data is divided into multiple data regions along the row dimension, each corresponding to a multiple target processing unit. A mapping relationship is established between each data region and each target storage space, thereby realizing a one-to-one correspondence between each data region and the corresponding target storage space when multiple processing units consume in parallel, avoiding resource contention caused by multiple processing units accessing the same storage region.

[0173] Optionally, based on the size of the output data in the row dimension and the number of target processing units, the row index range corresponding to each target processing unit is determined, and the output data is divided according to each row index range to obtain multiple data regions.

[0174] After determining the row dimension size of the output data, based on the row dimension size and the number of target processing units, the output data is divided into multiple data regions corresponding to each target processing unit by calculating the row index range for each target processing unit. Then, by querying the binding relationship between processing units and storage spaces in the hardware configuration information, the local storage spaces associated with each target processing unit are determined as multiple target storage spaces.

[0175] For example, the output data is a 4×N matrix, and the target processing unit includes two second processing units. The size of the output data in the row dimension can be determined to be 4, and the number of target processing units is 2. Subsequently, the output data is divided into two data regions along the row dimension. The row index range of the first data region is 0 to 1, and the row index range of the second data region is 2 to 3. The local storage space associated with the two target processing units is determined as the two target storage spaces.

[0176] In some embodiments, the row index range corresponding to each target processing unit is determined by dividing the size of the output data in the row dimension by the number of target processing units to obtain the number of rows corresponding to each target processing unit, based on the size of the output data in the row dimension and the number of target processing units.

[0177] In this embodiment, the row index range corresponding to each target processing unit is determined based on the size of the output data in the row dimension and the number of target processing units. The output data is then divided according to the row index range, so that the compiler can automatically deduce the data boundary of each target processing unit based on the matrix shape and parallel configuration, without having to manually specify the block size or processing unit mapping relationship in the computing kernel.

[0178] Optionally, the target storage space identifier corresponding to each data region is determined based on the mapping relationship between each data region and each target storage space. The target storage space identifier is used to characterize the target storage space corresponding to each data region.

[0179] Based on the second encoding layout, the relative storage offset of data elements in each data area in the corresponding target storage space is determined, and based on the target storage space identifier and the relative storage offset, the first processing unit generates data writing information for writing each data area to the corresponding target storage space.

[0180] Specifically, based on the second encoding layout, the relative position of the data elements in each data area in the corresponding target storage space is calculated, and the relative storage offset of the data elements in each data area in the corresponding target storage space is determined.

[0181] For example, the output data exists in two data regions. Based on the mapping relationship, the target storage space identifier corresponding to the first data region is determined to be DLM0, and the target storage space identifier corresponding to the second data region is determined to be DLM1. Subsequently, the relative storage offset of each data element in the corresponding target storage space is calculated according to the second encoding layout. Then, based on the target storage space identifiers DLM0 and DLM1 and the relative storage offset of each data element, data writing information is generated for the first processing unit to write the first data region to DLM0 and the second data region to DLM1.

[0182] In this embodiment, the target storage space identifier corresponding to each data region is determined according to the mapping relationship between each data region and each target storage space, and the relative storage offset of each data element in the corresponding target storage space is determined according to the second encoding layout, thereby generating data writing information, so that the data operation compilation result can directly guide the first processing unit to write each data region into the corresponding target storage space according to the precise storage location information.

[0183] Furthermore, the storage mapping information includes the target storage space identifier and the relative storage offset.

[0184] The starting relative offset of the output data in the target storage space is determined, and based on the second encoding layout and the index position of each data element in the output data, the element-local offset of each data element relative to the starting relative offset is determined. Finally, based on the starting relative offset and the element-local offset, the storage relative offset of each data element in the target storage space is determined.

[0185] Specifically, the output data undergoes liveness interval analysis to determine its liveness range, which characterizes the operational scope of the output data from its generation to its last consumption. Subsequently, based on the liveness range, the starting relative offset of the output data in the target storage space is determined. This starting relative offset is the address offset relative to the base address of the target storage space.

[0186] Furthermore, based on the aforementioned embodiments, according to the second encoding layout and the index position of each element in the output data, the element local offset of each data element relative to the starting relative offset is determined. Finally, according to the starting relative offset and the element local offset, the storage relative offset of each data element relative to the target storage space base address is determined.

[0187] The storage space occupied by each output result in the target storage space is related to the size of the output result.

[0188] In this embodiment, by determining the initial relative offset of the output data in the target storage space, and by determining the element local offset of each data element relative to the initial relative offset based on the second encoding layout and the index position of each data element in the output data, the storage relative offset of each data element in the target storage space is determined, thereby achieving element-level precise addressing and providing directly usable address calculation parameters for generating data reading information.

[0189] In step S250, the data reading information is used to read the output data from the target storage space and then use the set of parameters required for subsequent computation.

[0190] Optionally, the data reading information includes the target storage space identifier, the starting address for reading, the data size, and the reading mode, etc.

[0191] Furthermore, during the compilation process for subsequent operations, the target storage space where the input data corresponding to the subsequent operation is located is determined. Then, the target storage space identifier and the relative storage offset are passed to the data read operation corresponding to the subsequent operation. Finally, based on the target storage space identifier and the relative storage offset, the data read instruction corresponding to the subsequent operation is generated.

[0192] Optionally, the data read instructions are assembly instructions.

[0193] Optionally, "transfer" refers to the process of passing the target storage space identifier and storage offset from the storage mapping information at the compiler front end to the back end data read operation node. This can be achieved through the compiler's internal symbol set or the attribute fields of intermediate representation nodes.

[0194] Specifically, during the compilation process for subsequent operations, the target storage space corresponding to the input data of the operation is first determined. The data read operation node corresponding to the subsequent operation is located in the intermediate representation, and the target storage space identifier and relative storage offset are set as additional attributes of this node to complete the transfer. Finally, in the code generation stage, the compiler traverses this data read operation node, extracts the target storage space identifier and relative storage offset from the attribute fields, and maps these parameters to storage identifier information and address information in the load instruction according to the instruction set specification, generating the corresponding data read instruction.

[0195] For example, the storage mapping information indicates that the target storage space is identified as DLM1, the starting storage relative offset is 0x1A0, the operation node corresponding to its input data is located in the intermediate representation node of the subsequent operation, and the attributes "space_id=DLM_1、base_offset=0x1A0" are set for the node. During the code generation stage, the attributes of the above node are read, and the data reading instruction is generated as "VLD DLM_1, 0x1A0, V2", indicating that vector data is loaded from offset 0x1A0 into vector register V2.

[0196] In this embodiment, when compiling subsequent operations, the target storage space identifier and the relative storage offset are passed to the data read operation corresponding to the subsequent operation, and the data read instruction corresponding to the subsequent operation is generated according to the target storage space identifier and the relative storage offset. This establishes an address information transmission channel between the matrix multiplication operation side and the subsequent operation side, ensuring that the target processing unit can correctly locate and read the output data according to the storage mapping information.

[0197] Furthermore, when the compiler's intermediate representation does not include an explicit data write operation corresponding to a data read operation, the data read operation is generated as a memory read instruction with volatile memory access semantics. During compilation optimization, the memory read instruction is not removed based on the lack of a corresponding explicit data write operation.

[0198] In one implementation, after the first processing unit performs a matrix multiplication operation, it writes the output data of the matrix multiplication operation into the target storage space according to the storage mapping information determined during the compilation stage; the second processing unit reads the corresponding data from the target storage space when performing subsequent operations. Since the data writing process of the first processing unit can be formed during the hardware instruction generation stage or during the compilation process for the first processing unit, there may not necessarily be an explicit data writing operation in the compiler intermediate representation corresponding to the data reading operation.

[0199] In the above situation, if data read instructions are generated according to ordinary memory read semantics, the compilation optimization process may determine the data read operation as a read operation lacking a valid data source or a redundant read operation that can be eliminated based on the data flow relationship in the current intermediate representation, and thus delete it, causing the second processing unit to be unable to obtain the output data generated by the first processing unit from the target storage space when performing subsequent operations.

[0200] Therefore, data read operations can be compiled according to volatile memory access semantics, so that data read operations are not deleted due to the lack of corresponding explicit data write operations in the current intermediate representation, thereby preserving the actual read operations on the target storage space in the generated target program.

[0201] In one implementation, volatile memory access semantics can be achieved by setting volatile semantics for memory read instructions. Memory read instructions with volatile semantics are treated as read operations with actual memory access requirements during subsequent compilation optimization. The compiler retains the memory read operation and generates the corresponding actual memory read instruction when generating the object code.

[0202] Specifically, during compilation, the corresponding memory read instruction node is first generated, and the context information of the current computing kernel is checked to determine whether the data in the target storage space accessed by the memory read instruction originates from a matrix multiplication operation, and whether the write operation of the output data of the matrix multiplication operation is not reflected as an explicit data write operation in the intermediate representation. If so, the corresponding flag bit in the memory read instruction is set to true.

[0203] During subsequent compilation optimization, the intermediate representation in the compiler is traversed, and the flags of each memory access instruction are checked. If a memory read instruction's flag is found to be true, it is not considered dead code that can be deleted, and is thus mapped to the actual memory load machine instruction during the code generation phase. For example, this flag may be a volatile attribute.

[0204] In this embodiment, when the compiler intermediate representation does not include an explicit data write operation corresponding to the data read operation, a memory read instruction with volatile memory access semantics will be generated through the data read operation. During the compilation optimization process, the memory read instruction will not be deleted based on the lack of a corresponding explicit data write operation, thereby preventing the loading operation of the target storage space from being mistakenly judged as dead code and deleted during the compilation optimization stage. This ensures that the target processing unit can correctly read the output data in the target storage space and guarantees the correctness of the compilation result of the data operation.

[0205] Furthermore, the computation kernel includes at least two matrix multiplication operations. For each matrix multiplication operation, the survival interval of the corresponding output data is determined, which is determined at least based on the generation location of the corresponding output data and the location of the last consumed output data. And based on the survival interval corresponding to each output data, a storage area in the corresponding target storage space is allocated to each output data.

[0206] When a computation kernel includes multiple matrix multiplication operations, each matrix multiplication produces output data which is stored in the target storage space (such as DLM). If each output data occupies a dedicated storage area, it will lead to a large amount of local storage space being occupied. Therefore, by performing liveness interval analysis, when the liveness intervals of two output data do not overlap, it indicates that these two output data will not be used simultaneously. Thus, these two output data can reuse the same storage area, thereby checking the total demand of the target storage space and improving the utilization of storage resources.

[0207] Optionally, when the live intervals corresponding to any two output data do not overlap, the two output data reuse at least a portion of the storage area in the target storage space; when the live intervals corresponding to any two output data overlap, non-overlapping storage areas are allocated to the two output data.

[0208] Specifically, if the current output data's survival range does not overlap with the survival range of the allocated output data, the current output data is allocated to the storage area occupied by the allocated output data, and the two share the same starting relative offset.

[0209] If the current output data's survival interval overlaps with the survival interval of the allocated output data, a free area outside the allocated area in the current target storage space that does not intersect with the survival interval of the allocated output data is selected for the current output data, and its initial relative offset is recorded.

[0210] Optionally, if the current output data's liveness interval does not overlap with the liveness intervals of multiple allocated output data, the storage area corresponding to the output data is determined based on the size of the output data.

[0211] refer to Figure 7The same computational kernel contains two matrix multiplication operations. DotOp1 (the first output data of the first matrix multiplication operation) is the result of A×B1, and DotOp2 (the second output data of the second matrix multiplication operation) is the result of A×B2. Both DotOp1 and DotOp2 are consumed by subsequent operations, and their corresponding target processing unit is the second processing unit. Survival interval analysis is performed on both, determining that the survival interval for DotOp1 is [0, 10), and the survival interval for DotOp2 is [50, 60). Their survival intervals do not overlap. Therefore, DotOp1 and DotOp2 can reuse the same storage area, meaning they can be allocated the same starting offset address. In the diagram, the starting relative offsets of the storage areas for DotOp1 and DotOp2 are both 0, thus achieving local storage space reuse.

[0212] Optionally, the corresponding storage space requirement is determined based on the data size of the output data. Then, at least one candidate free storage region is determined based on the address range of the allocated storage regions in the target storage space; and from the at least one candidate free storage region, a free storage region whose size meets the storage space requirement is determined, and the storage region corresponding to the output data is allocated in the determined free storage region.

[0213] If there is no free storage area with a size that meets the storage space requirements in at least one candidate free storage area, the storage area corresponding to the output data shall be allocated outside the allocated storage area in the target storage space.

[0214] Since the target storage space has a limited capacity and there may be unused gaps in the allocated areas, we can first determine the storage space requirements of the output data, and then find a free storage area of ​​matching size for reuse, making use of the free space between the allocated areas and reducing space waste.

[0215] In one embodiment, the compiler determines the corresponding storage space requirement based on the size of the current output data. Then, the compiler sequentially scans each candidate storage region in the candidate storage region list in ascending address order, subtracting the starting relative offset from the end offset of the current candidate region to obtain the size of the free storage region. If the size of the first candidate free storage region meets the storage space requirement, the compiler allocates the storage region corresponding to the output data within that candidate storage region, using the starting relative offset of the candidate region as the starting relative offset for the output data, and deducts the allocated storage space from the candidate region.

[0216] For example, the address range of the allocated storage regions in the target storage space is [0x00, 0x1F] and [0x40, 0x5F]. Candidate free storage regions include [0x20, 0x3F] (size 0x20) and [0x60, 0xFF] (size 0xA0). The storage space requirement for the output data is 0x30. The compiler scans in address order. The first candidate [0x20, 0x3F] has a size of 0x20, which is less than 0x30, so it does not meet the requirement. The compiler then continues scanning the second candidate [0x60, 0xFF], whose size is 0xA0, which is not less than 0x30, thus meeting the requirement. The compiler allocates a storage region in this area, with a starting relative offset of 0x60 and a length of 0x30. This candidate region is then updated to [0x90, 0xFF] with a size of 0x70.

[0217] In another embodiment, after determining the storage space requirement, the compiler iterates through all candidate free storage regions, calculates the size of each candidate region, and determines the difference between its size and the storage space requirement. Then, from all candidate free storage regions whose size is not less than the storage space requirement, the candidate storage region with the smallest difference is selected and allocated to the output data.

[0218] Optionally, if the target storage space has an upper limit address constraint, such as the maximum addressable range specified by the hardware, the compiler must ensure that the end offset of the newly allocated region does not exceed the maximum addressable range when allocating addresses.

[0219] In this embodiment, the corresponding storage space requirement is determined based on the data size of the output data. First, at least one candidate free storage area is determined from the address range of the allocated storage area in the target storage space. From the at least one candidate free storage area, a free storage area whose size meets the storage space requirement is determined and allocated, so as to achieve fine allocation of the target storage space and improve the utilization efficiency of the limited storage space.

[0220] Please refer to the solutions provided in the above embodiments. Figure 8 The compilation process includes the following steps.

[0221] Step S801: Determine the target processing unit to perform subsequent operations.

[0222] Based on the data dependencies between operations in the computation kernel, the subsequent operations on the output data of the matrix multiplication operation are determined. Based on the operation type and hardware configuration information of the subsequent operations, the target processing unit for executing the subsequent operations is determined from at least one second processing unit. When the target processing unit for executing the subsequent operations is a second processing unit, the local storage space associated with the second processing unit is determined as the target storage space corresponding to the output data.

[0223] Step S802: Perform address allocation in the target storage space.

[0224] The storage space requirements are determined based on the size of the output data. The survival interval of the output data is determined based on the definition location of the output data in the source code to be compiled and the location of subsequent operations consuming the output data. The survival interval is determined by the location where the output data is generated and the location of the last consumed output data. Based on the survival interval corresponding to each output data, a corresponding storage area in the target storage space is allocated for the output data of each matrix multiplication operation, and the initial relative offset of the output data in the target storage space is determined.

[0225] Step S803, encoding layout conversion.

[0226] When the target processing unit for performing subsequent operations is the second processing unit, the first encoding layout corresponding to the output data is converted into a second encoding layout that matches the storage layout of the target storage space.

[0227] Step S804: Transfer of storage mapping information.

[0228] Based on the target storage space and the second encoding layout corresponding to the output data, the storage mapping information corresponding to the output data is determined. The storage mapping information includes the target storage space identifier and the relative storage offset.

[0229] Specifically, the starting relative offset of the output data in the target storage space is determined; based on the second encoding layout and the index position of each data element in the output data, the element local offset of each data element relative to the starting relative offset is determined; based on the starting relative offset and the element local offset, the storage relative offset of each data element in the target storage space is determined.

[0230] When compiling subsequent operations, the target storage space where the input data for the subsequent operations is located is determined, and the target storage space identifier and the relative storage offset are passed to the data reading operation corresponding to the subsequent operations.

[0231] Step S805, setting volatile memory access semantics.

[0232] When the compiler's intermediate representation does not include an explicit data write operation corresponding to a data read operation, the data read operation is generated as a memory read instruction with volatile memory access semantics. During compilation optimization, memory read instructions are not removed based on the lack of a corresponding explicit data write operation.

[0233] Memory read instructions with volatile memory access semantics are treated as read operations with actual memory access requirements during subsequent compilation optimization. The compiler retains the memory read operation and generates the corresponding actual memory read instruction when generating the target code.

[0234] This disclosure performs static data dependency analysis on the computational kernel during compilation to automatically identify the subsequent operations and execution units of the matrix multiplication output data. When it is determined that the subsequent operation will be executed by a second processing unit, the local storage space associated with the second processing unit is automatically identified as the target storage space for the output data. The original first encoding layout of the output data, oriented towards the shared space, is converted to a second encoding layout that matches the storage characteristics of the local storage space, thereby generating a data operation compilation result containing precise storage mapping information. This allows the matrix multiplication output data to be directly read and consumed by subsequent processing units from the local storage space, avoiding the transfer of result data in shared storage. This reduces on-chip data transfer overhead and access latency while improving local storage utilization and fusion operator execution efficiency.

[0235] Furthermore, subsequent operations on the output data are determined through data dependencies, and the target processing unit is determined from at least one second processing unit according to the operation type of the subsequent operations. The local storage space associated with the target processing unit is then determined as the target storage space, thereby establishing a direct correspondence between the output data and the target processing unit and its associated local storage space during compilation.

[0236] Furthermore, when the target processing unit for subsequent operations is the third processing unit, a candidate storage space with a direct data access path between it and the third processing unit is selected as the target storage space. This expands the applicability of the compilation processing method to heterogeneous hardware configurations and avoids allocating the output data to a storage space without a direct data access path.

[0237] Furthermore, the first processing unit can be a tensor processing unit, and the second processing unit can be an execution unit that performs vector and scalar operations. The local storage space can be a data local storage space bound to the execution unit. A data reading path is set between the execution unit and the corresponding data local storage space, thereby enabling the data operation compilation results to directly correspond to the physical hardware resources of the heterogeneous accelerator.

[0238] Furthermore, by covering matrix multiplication and accumulation operations and explicitly defining the data reading path, including the data transfer path between the local data storage space and the vector register, the fusion result after the tensor processing unit performs matrix multiplication and accumulation can be directly output to the local data storage space. The execution unit can then directly load data from the local data storage space into the vector register to perform subsequent operations, reducing computational overhead.

[0239] Furthermore, by converting the output data from a block-based encoding layout corresponding to the block-based computation method of matrix multiplication to a second encoding layout that matches the row-major order storage layout of the target storage space, the second encoding layout represents the mapping relationship between each data element and the storage location in the target storage space. This makes the storage format of the output data match the reading characteristics of the target processing unit, eliminating the cost of manual layout adaptation.

[0240] Furthermore, by establishing a mapping relationship based on the row and column indices of each data element through the second encoding layout, data elements with the same row index are mapped to consecutive storage locations in order of column index, thereby improving the spatial locality when the target processing unit reads the output data sequentially.

[0241] Furthermore, based on the size of the output data in the row dimension and the number of target processing units, the output data is divided into multiple data regions along the row dimension, each corresponding to a multiple target processing unit. A mapping relationship is established between each data region and each target storage space, thereby achieving a one-to-one correspondence between each data region and the corresponding target storage space when multiple processing units consume data in parallel, avoiding resource contention caused by multiple processing units accessing the same storage region.

[0242] Furthermore, the row index range corresponding to each target processing unit is determined based on the size of the output data in the row dimension and the number of target processing units, and the output data is divided according to each row index range. This allows the compiler to automatically deduce the data boundaries responsible for each target processing unit based on the matrix shape and parallel configuration, without having to manually specify the block size or processing unit mapping relationship in the computing kernel.

[0243] Furthermore, the target storage space identifier corresponding to each data region is determined according to the mapping relationship between each data region and each target storage space, and the relative storage offset of each data element in the corresponding target storage space is determined according to the second encoding layout, thereby generating data writing information, so that the data operation compilation result can directly guide the first processing unit to write each data region into the corresponding target storage space according to the precise storage location information.

[0244] Furthermore, for at least two matrix multiplication operations included in the computing kernel, the corresponding survival interval is determined according to the generation position of each output data and the position of the last consumed output data, and the corresponding storage area in the target storage space is allocated to each output data according to each survival interval, thereby providing a lifecycle-based fine-grained storage plan for the intermediate output data of multiple matrix multiplication operations.

[0245] Furthermore, when the survival intervals corresponding to any two output data do not overlap, the two output data reuse at least a portion of the storage area in the target storage space. When the survival intervals corresponding to any two output data overlap, non-overlapping storage areas are allocated to the two output data. Thus, under the limited capacity of the target storage space, by reusing the storage area when the survival intervals do not overlap, more matrix multiplication operations can be supported to use the target storage space simultaneously under the limited capacity.

[0246] Furthermore, based on the data size of the output data, the corresponding storage space requirements are determined. First, at least one candidate free storage region is determined from the address range of the allocated storage regions in the target storage space. From the at least one candidate free storage region, a free storage region whose size meets the storage space requirements is determined and allocated, so as to achieve fine-grained allocation of the target storage space and improve the utilization efficiency of limited storage space.

[0247] Furthermore, by determining the initial relative offset of the output data in the target storage space, and based on the second encoding layout and the index position of each data element in the output data, the element local offset of each data element relative to the initial relative offset is determined, thereby determining the storage relative offset of each data element in the target storage space, thus achieving element-level precise addressing and providing directly usable address calculation parameters for generating data reading information.

[0248] Furthermore, during the compilation process of subsequent operations, the target storage space identifier and the relative storage offset are passed to the data read operation corresponding to the subsequent operation, and the data read instruction corresponding to the subsequent operation is generated based on the target storage space identifier and the relative storage offset. This establishes an address information transmission channel between the matrix multiplication operation side and the subsequent operation side, ensuring that the target processing unit can correctly locate and read the output data based on the storage mapping information.

[0249] Furthermore, when the compiler's intermediate representation does not include an explicit data write operation corresponding to the data read operation, a memory read instruction with volatile memory access semantics will be generated through the data read operation. During the compilation optimization process, the memory read instruction will not be deleted based on the lack of a corresponding explicit data write operation. This prevents the loading operation of the target storage space from being mistakenly identified as dead code and deleted during the compilation optimization stage, ensuring that the target processing unit can correctly read the output data in the target storage space and guaranteeing the correctness of the compilation results of data operations.

[0250] Description of apparatus and devices according to embodiments of this disclosure

[0251] See Figure 9This disclosure also provides a compilation processing apparatus 900, including an acquisition module 910, a storage space determination module 920, a layout conversion module 930, a mapping information determination module 940, and a compilation result generation module 950.

[0252] The acquisition module 910 is used to acquire the computing kernel and hardware configuration information in the program to be compiled. The computing kernel includes matrix multiplication operation and subsequent operations that consume the output data of the matrix multiplication operation. The hardware configuration information includes a first processing unit that performs the matrix multiplication operation, at least one second processing unit, and local storage space associated with each second processing unit.

[0253] The storage space determination module 920 is used to determine the target processing unit for executing the subsequent operation and the target storage space corresponding to the output data based on the data dependency relationship between each operation in the computing kernel and the hardware configuration information.

[0254] The layout conversion module 930 is used to determine the local storage space associated with the second processing unit as the target storage space when the target processing unit for performing the subsequent operation is the second processing unit, and to convert the first encoding layout corresponding to the output data into a second encoding layout that matches the storage layout of the target storage space.

[0255] The mapping information determination module 940 is used to determine the storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data;

[0256] The compilation result generation module 950 is used to generate the data operation compilation result corresponding to the computing kernel based on the storage mapping information. The data operation compilation result includes data writing information for writing the output data to the target storage space and data reading information for reading the output data from the target storage space.

[0257] Optionally, the storage space determination module 920 is used for:

[0258] The subsequent operations for consuming the output data are determined based on the data dependencies.

[0259] Based on the operation type of the subsequent operation and the hardware configuration information, a target processing unit for performing the subsequent operation is determined from the at least one second processing unit;

[0260] The local storage space associated with the determined second processing unit is determined as the target storage space corresponding to the output data.

[0261] Optionally, the hardware configuration information also includes at least one third processing unit, at least one candidate storage space other than the local storage space, and the data access relationship between the third processing unit and the candidate storage space, wherein no direct data access path is set between the third processing unit and the local storage space;

[0262] The storage space determination module 920 is further configured to, when the target processing unit for performing the subsequent operation is a third processing unit, filter out candidate storage spaces from the at least one candidate storage space that have a direct data access path between the third processing unit and each candidate storage space according to the data access relationship between the target processing unit for performing the subsequent operation and each candidate storage space, and determine the filtered candidate storage space as the target storage space corresponding to the output data.

[0263] Optionally, the first processing unit is a tensor processing unit;

[0264] The second processing unit is an execution unit that performs at least one of vector operations and scalar operations;

[0265] The local storage space is the data local storage space bound to the execution unit;

[0266] A data reading path is set between the execution unit and the corresponding local data storage space.

[0267] Optionally, the matrix multiplication operation is a matrix multiplication operation or a matrix multiplication accumulation operation;

[0268] The execution unit includes a vector register, and the data read path includes a data transfer path between the local data storage space and the vector register;

[0269] The subsequent operations include at least one of vector addition, vector multiplication, and scalar operations.

[0270] Optionally, the output data is tensor data; the first encoding layout is a block encoding layout corresponding to the block calculation method of the matrix multiplication operation, and the first encoding layout represents the block arrangement relationship of each data element in the output data;

[0271] The target storage space adopts a row-major order storage layout;

[0272] The second encoding layout represents the mapping relationship between each data element in the output data and the storage location in the target storage space.

[0273] Optionally, the second encoding layout is used to establish a mapping relationship between each data element and the corresponding storage location in the target storage space according to the row index and column index of each data element in the output data, so that data elements with the same row index are mapped to consecutive storage locations in column index order.

[0274] Optionally, the target processing unit includes a plurality of the second processing units, and the mapping information determination module 940 is used for:

[0275] Based on the size of the output data in the row dimension and the number of target processing units, the output data is divided along the row dimension into multiple data regions corresponding to the multiple target processing units respectively;

[0276] The local storage space associated with each of the multiple target processing units is determined as multiple target storage spaces;

[0277] Establish a mapping relationship between each of the data regions and each of the target storage spaces;

[0278] Based on the second encoding layout and the mapping relationship, the target storage space and storage location information corresponding to the data elements in each data area are determined.

[0279] Optionally, the mapping information determination module 940, based on the size of the output data in the row dimension and the number of target processing units, is used to:

[0280] Based on the size of the output data in the row dimension and the number of target processing units, determine the row index range corresponding to each target processing unit;

[0281] The output data is divided according to the index range of each row to obtain multiple data regions.

[0282] Optionally, the mapping information determination module 940 is used to:

[0283] Based on the mapping relationship between each data region and each target storage space, the target storage space identifier corresponding to each data region is determined;

[0284] Based on the second encoding layout, determine the relative storage offset of the data elements in each data area in the corresponding target storage space;

[0285] Based on the target storage space identifier and the storage relative offset, the first processing unit generates data writing information for writing each of the data regions to the corresponding target storage space.

[0286] Optionally, the computational kernel includes at least two matrix multiplication operations, and the apparatus further includes:

[0287] The survival interval determination module is used to determine the survival interval of the corresponding output data for each of the matrix multiplication operations. The survival interval is determined at least based on the generation position of the corresponding output data and the position of the last consumption of the output data.

[0288] The storage area allocation module is used to allocate a storage area in the corresponding target storage space to each output data according to the liveness interval corresponding to each output data.

[0289] Optionally, the storage area allocation module is used for:

[0290] When the live intervals corresponding to any two output data do not overlap, the two output data reuse at least a portion of the storage area in the target storage space;

[0291] When the live intervals corresponding to any two output data overlap, non-overlapping storage areas are allocated to the two output data.

[0292] Optionally, the storage area allocation module is used for:

[0293] The corresponding storage space requirements are determined based on the data size of the output data.

[0294] At least one candidate free storage region is determined based on the address range of the allocated storage regions in the target storage space;

[0295] From the at least one candidate free storage area, determine a free storage area whose size meets the storage space requirement, and allocate the storage area corresponding to the output data in the determined free storage area;

[0296] If no free storage area with a size that meets the storage space requirement exists in at least one candidate free storage area, the storage area corresponding to the output data shall be allocated outside the allocated storage area in the target storage space.

[0297] Optionally, the storage mapping information includes the target storage space identifier and the relative storage offset;

[0298] The mapping information determination module 940 is used for:

[0299] Determine the initial relative offset of the output data in the target storage space;

[0300] Based on the second encoding layout and the index position of each data element in the output data, determine the element local offset of each data element relative to the initial relative offset;

[0301] Based on the initial relative offset and the element local offset, the storage relative offset of each data element in the target storage space is determined.

[0302] Optionally, the device further includes an instruction generation module for:

[0303] When compiling the subsequent operations, the target storage space where the input data corresponding to the subsequent operations is located is determined;

[0304] The target storage space identifier and the relative storage offset are passed to the data read operation corresponding to the subsequent operation;

[0305] Based on the target storage space identifier and the relative storage offset, a data read instruction corresponding to the subsequent operation is generated.

[0306] Optionally, the instruction generation module is further configured to:

[0307] When the compiler intermediate representation does not include an explicit data write operation corresponding to the data read operation, the data read operation is generated as a memory read instruction with volatile memory access semantics.

[0308] During the compilation optimization process, the memory read instruction is not deleted because it lacks a corresponding explicit data write operation.

[0309] The compiler processing apparatus 900 disclosed herein is used to execute the compiler processing method as described in the above embodiments. Its specific processing procedure is the same as that of the compiler processing method described in the above embodiments, and will not be repeated here.

[0310] This disclosure also provides an electronic device 1000, comprising:

[0311] At least one processor, and,

[0312] A memory that is communicatively connected to at least one processor; wherein,

[0313] The memory stores instructions that are executed by at least one processor to cause the at least one processor to perform the method as described in any of the above embodiments of this application when executing the instructions.

[0314] The following is combined with Figure 10 The hardware structure of the electronic device is described in detail. The electronic device includes: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050.

[0315] The processor 1010 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure.

[0316] The memory 1020 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1020 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010 using the compilation processing method of the embodiments of this disclosure.

[0317] The input / output interface 1030 is used to implement information input and output;

[0318] The communication interface 1040 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and the bus 1050 is used to transmit information between various components of the device (such as processor 1010, memory 1020, input / output interface 1030 and communication interface 1040).

[0319] The processor 1010, memory 1020, input / output interface 1030 and communication interface 1040 are connected to each other within the device via bus 1050.

[0320] This application also provides a computer-readable storage medium that stores one or more programs, which can be executed by one or more processors to implement the compilation processing method of the above embodiments, which will not be described again here.

[0321] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “including,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatuses.

[0322] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0323] It should be understood that in the description of the embodiments of this disclosure, "multiple" means two or more, "greater than", "less than", "exceeding" etc. are understood to exclude the number itself, and "above", "below", "within" etc. are understood to include the number itself.

[0324] In the several embodiments provided in this disclosure, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0325] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0326] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0327] It should also be understood that the various implementation methods provided in this disclosure can be combined arbitrarily to achieve different technical effects.

[0328] The above is a detailed description of the embodiments of this disclosure. However, this disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this disclosure. All such equivalent modifications or substitutions are included within the scope defined by the claims of this disclosure.

Claims

1. A compilation processing method, characterized in that, The method includes: Obtain the computing kernel and hardware configuration information in the program to be compiled. The computing kernel includes matrix multiplication and subsequent operations that consume the output data of the matrix multiplication. The hardware configuration information includes a first processing unit that performs the matrix multiplication, at least one second processing unit, and local storage space associated with each of the second processing units. Based on the data dependencies between operations in the computing kernel and the hardware configuration information, the target processing unit for executing the subsequent operations and the target storage space corresponding to the output data are determined. When the target processing unit for performing the subsequent operations is the second processing unit, the local storage space associated with the second processing unit is determined as the target storage space, and the first encoding layout corresponding to the output data is converted into a second encoding layout that matches the storage layout of the target storage space. Based on the target storage space and the second encoding layout corresponding to the output data, determine the storage mapping information corresponding to the output data; The data operation compilation result corresponding to the computing kernel is generated based on the storage mapping information. The data operation compilation result includes data writing information for writing the output data to the target storage space and data reading information for reading the output data from the target storage space.

2. The method according to claim 1, characterized in that, The step of determining the target processing unit for executing the subsequent operations and the target storage space corresponding to the output data based on the data dependencies between operations in the computing kernel and the hardware configuration information includes: The subsequent operations for consuming the output data are determined based on the data dependencies. Based on the operation type of the subsequent operation and the hardware configuration information, a target processing unit for performing the subsequent operation is determined from the at least one second processing unit; The local storage space associated with the determined second processing unit is determined as the target storage space corresponding to the output data.

3. The method according to claim 1, characterized in that, The hardware configuration information also includes at least one third processing unit, at least one candidate storage space other than the local storage space, and the data access relationship between the third processing unit and the candidate storage space. No direct data access path is set between the third processing unit and the local storage space. The step of determining the target processing unit for executing the subsequent operations and the target storage space corresponding to the output data based on the data dependencies between operations in the computing kernel and the hardware configuration information further includes: When the target processing unit for performing the subsequent operation is the third processing unit, according to the data access relationship between the target processing unit for performing the subsequent operation and each of the candidate storage spaces, a candidate storage space with a direct data access path between it and the third processing unit is selected from the at least one candidate storage space, and the selected candidate storage space is determined as the target storage space corresponding to the output data.

4. The method according to claim 2, characterized in that, The first processing unit is a tensor processing unit; The second processing unit is an execution unit that performs at least one of vector operations and scalar operations; The local storage space is the data local storage space bound to the execution unit; A data reading path is set between the execution unit and the corresponding local data storage space.

5. The method according to claim 4, characterized in that, The matrix multiplication operation is either a matrix multiplication operation or a matrix multiplication accumulation operation; The execution unit includes a vector register, and the data read path includes a data transfer path between the local data storage space and the vector register; The subsequent operations include at least one of vector addition, vector multiplication, and scalar operations.

6. The method according to claim 1, characterized in that, The output data is tensor data; the first encoding layout is a block encoding layout corresponding to the block calculation method of the matrix multiplication operation, and the first encoding layout represents the block arrangement relationship of each data element in the output data; The target storage space adopts a row-major order storage layout; The second encoding layout represents the mapping relationship between each data element in the output data and the storage location in the target storage space.

7. The method according to claim 6, characterized in that, The second encoding layout is used to establish a mapping relationship between each data element and the corresponding storage location in the target storage space according to the row index and column index of each data element in the output data, so that data elements with the same row index are mapped to consecutive storage locations in column index order.

8. The method according to claim 6, characterized in that, The target processing unit includes multiple second processing units. The step of determining the storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data includes: Based on the size of the output data in the row dimension and the number of target processing units, the output data is divided along the row dimension into multiple data regions corresponding to the multiple target processing units respectively; The local storage space associated with each of the multiple target processing units is determined as multiple target storage spaces; Establish a mapping relationship between each of the data regions and each of the target storage spaces; Based on the second encoding layout and the mapping relationship, the target storage space and storage location information corresponding to the data elements in each data area are determined.

9. The method according to claim 8, characterized in that, The step of dividing the output data into multiple data regions corresponding to multiple target processing units along the row dimension based on the size of the output data in the row dimension and the number of target processing units includes: Based on the size of the output data in the row dimension and the number of target processing units, determine the row index range corresponding to each target processing unit; The output data is divided according to the index range of each row to obtain multiple data regions.

10. The method according to claim 8 or 9, characterized in that, The step of determining the target storage space and storage location information corresponding to the data elements in each of the data regions based on the second encoding layout and the mapping relationship includes: Based on the mapping relationship between each data region and each target storage space, the target storage space identifier corresponding to each data region is determined; Based on the second encoding layout, determine the relative storage offset of the data elements in each data area in the corresponding target storage space; Based on the target storage space identifier and the storage relative offset, the first processing unit generates data writing information for writing each of the data regions to the corresponding target storage space.

11. The method according to claim 1, characterized in that, The computational kernel includes at least two matrix multiplication operations, and the method further includes: For each of the matrix multiplication operations, the survival interval of the corresponding output data is determined, and the survival interval is determined at least based on the generation position of the corresponding output data and the position of the last consumption of the output data; Based on the survival interval corresponding to each output data, a storage area in the corresponding target storage space is allocated to each output data.

12. The method according to claim 11, characterized in that, The step of allocating a storage area in the target storage space for each output data according to the survival interval corresponding to each output data includes: When the live intervals corresponding to any two output data do not overlap, the two output data reuse at least a portion of the storage area in the target storage space; When the live intervals corresponding to any two output data overlap, non-overlapping storage areas are allocated to the two output data.

13. The method according to claim 11 or 12, characterized in that, The step of allocating a corresponding storage area in the target storage space for each of the output data includes: The corresponding storage space requirements are determined based on the data size of the output data. At least one candidate free storage region is determined based on the address range of the allocated storage regions in the target storage space; From the at least one candidate free storage area, determine a free storage area whose size meets the storage space requirement, and allocate the storage area corresponding to the output data in the determined free storage area; If no free storage area with a size that meets the storage space requirement exists in at least one candidate free storage area, the storage area corresponding to the output data shall be allocated outside the allocated storage area in the target storage space.

14. The method according to claim 6, characterized in that, The storage mapping information includes the target storage space identifier and the relative storage offset; The step of determining the storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data includes: Determine the initial relative offset of the output data in the target storage space; Based on the second encoding layout and the index position of each data element in the output data, determine the element local offset of each data element relative to the initial relative offset; Based on the initial relative offset and the element local offset, the storage relative offset of each data element in the target storage space is determined.

15. The method according to claim 14, characterized in that, The method further includes: When compiling the subsequent operations, the target storage space where the input data corresponding to the subsequent operations is located is determined; The target storage space identifier and the relative storage offset are passed to the data read operation corresponding to the subsequent operation; Based on the target storage space identifier and the relative storage offset, a data read instruction corresponding to the subsequent operation is generated.

16. The method according to claim 15, characterized in that, The method further includes: When the compiler intermediate representation does not include an explicit data write operation corresponding to the data read operation, the data read operation is generated as a memory read instruction with volatile memory access semantics. During the compilation optimization process, the memory read instruction is not deleted because it lacks a corresponding explicit data write operation.

17. A compilation processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire the computing kernel and hardware configuration information in the program to be compiled. The computing kernel includes matrix multiplication operation and subsequent operations that consume the output data of the matrix multiplication operation. The hardware configuration information includes a first processing unit that performs the matrix multiplication operation, at least one second processing unit, and local storage space associated with each second processing unit. The storage space determination module is used to determine the target processing unit for executing the subsequent operation and the target storage space corresponding to the output data based on the data dependency relationship between each operation in the computing kernel and the hardware configuration information. The layout conversion module is used to determine the local storage space associated with the second processing unit as the target storage space when the target processing unit for performing the subsequent operation is the second processing unit, and to convert the first encoding layout corresponding to the output data into a second encoding layout that matches the storage layout of the target storage space. The mapping information determination module is used to determine the storage mapping information corresponding to the output data based on the target storage space and the second encoding layout corresponding to the output data; The compilation result generation module is used to generate the data operation compilation result corresponding to the computing kernel based on the storage mapping information. The data operation compilation result includes data writing information for writing the output data to the target storage space and data reading information for reading the output data from the target storage space.

18. An electronic device, characterized in that, The electronic device includes a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for establishing communication between the processor and the memory. The program is executed by the processor to implement the compilation method as described in any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the compilation processing method as described in any one of claims 1 to 16.

Images