Fusion operator execution method, electronic device, storage medium and program product

Abstract

The invention relates to the technical field of artificial intelligence, and provides a fusion operator execution method, electronic equipment, a storage medium and a program product, and the method comprises the steps: determining a plurality of to-be-fused target operators; according to the tensors of the multiple target operators, a pseudo address mapping table is generated, and the pseudo address mapping table is used for recording the mapping relation between the actual memory addresses of the tensors and pseudo addresses; operator fusion is carried out on the multiple target operators, a fusion operator is obtained, and the memory address of the tensor of the fusion operator is a continuous pseudo address; and executing the fusion operator, converting the pseudo address into an actual memory address according to the pseudo address mapping table in the execution process, and accessing data of the tensor according to the actual memory address. According to the method, a large amount of extra memory copy overhead caused by physical movement and rearrangement of the original tensor in a traditional operator fusion scheme is avoided, and the computing resource utilization rate and the execution efficiency are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a fusion operator execution method, electronic device, storage medium, and program product. Background Technology

[0002] Accelerating artificial intelligence (AI) computing is one of the key technologies for improving overall computing efficiency, especially when dealing with specific computing tasks.

[0003] Currently, to accelerate AI computation on multiple small tensors that are continuous and independent of data, the approach is often to merge multiple operations into a single operation with multiple inputs. However, this method usually requires rewriting operator code for different numbers of inputs, and also requires using complex logic within the operator to determine the number of inputs, leading to difficulties in development and maintenance, complex operator logic, and poor versatility. Summary of the Invention

[0004] This invention provides a fusion operator execution method, electronic device, storage medium, and program product to address the deficiencies in related technologies.

[0005] This invention provides a method for executing a fusion operator, comprising the following steps: Identify multiple target operators to be fused; Based on tensors of multiple target operators, a pseudo-address mapping table is generated. The pseudo-address mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the tensor. The multiple target operators are fused to obtain a fused operator, and the memory address of the tensor of the fused operator is a contiguous pseudo address. The fusion operator is executed, and during the execution process, the pseudo address is converted into the actual memory address according to the pseudo address mapping table, and the tensor data is accessed according to the actual memory address.

[0006] According to a fusion operator execution method provided by the present invention, the step of generating a pseudo-address mapping table based on tensors of multiple target operators includes: Collect the actual memory addresses of the input and output tensors of the multiple target operators; Assign contiguous pseudo-addresses to the input and output tensors; Establish a mapping relationship between the pseudo address and the actual memory address, and record it in the pseudo address mapping table.

[0007] According to a fusion operator execution method provided by the present invention, the step of allocating contiguous pseudo-addresses for the input tensor and the output tensor includes: Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the input tensor, and the size of each pseudo-address is consistent with the size of the corresponding input tensor. Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the output tensor, and the size of each pseudo-address is consistent with the size of the corresponding output tensor.

[0008] According to a fusion operator execution method provided by the present invention, the step of determining multiple target operators to be fused includes: The graph compiler scans the computation graph to identify multiple target operators that satisfy a preset pattern. The preset pattern includes that the multiple target operators are operators of the same type and / or that the input tensors and output tensors of the multiple target operators have the same shape.

[0009] According to a fusion operator execution method provided by the present invention, the step of scanning the computation graph by a graph compiler to identify a plurality of target operators that satisfy a preset pattern includes: The computation graph is scanned by the graph compiler, and operators that have no data dependencies between operators and whose tensors do not overlap in memory are selected as candidate operators. The candidate operator that satisfies the preset mode is taken as the target operator.

[0010] According to a fusion operator execution method provided by the present invention, the step of converting the pseudo-address to the actual memory address according to the pseudo-address mapping table, and accessing the tensor data according to the actual memory address, includes: The lookup index corresponding to the pseudo-address is obtained by calculating the thread index or tensor index. Based on the lookup index, the corresponding actual memory address is searched and obtained from the pseudo-address mapping table; Based on the actual memory address, access the data of the corresponding tensor to perform read or write operations.

[0011] According to a fusion operator execution method provided by the present invention, the step of fusing the plurality of target operators to obtain a fusion operator includes: The pseudo-address mapping table is received as an input parameter, and the calculation logic of the multiple target operators is integrated into a single calculation process to obtain the fusion operator; In the calculation process, the access addresses of the input tensors and output tensors of the multiple target operators are pseudo-addressed, so that the fusion operator logically accesses consecutive pseudo-addresses.

[0012] According to a fusion operator execution method provided by the present invention, the pseudo-address mapping table includes an input mapping table and an output mapping table. The input mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the input tensor, and the output mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the output tensor.

[0013] The present invention also provides a fusion operator execution device, comprising the following units: The determining unit is used to determine multiple target operators to be fused. The generation unit is used to generate a pseudo address mapping table based on tensors of multiple target operators. The pseudo address mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the tensor. A fusion unit is used to perform operator fusion on the multiple target operators to obtain a fusion operator, wherein the memory address of the tensor of the fusion operator is a contiguous pseudo address; An execution unit is used to execute the fusion operator, and during the execution process, converts the pseudo address into the actual memory address according to the pseudo address mapping table, and accesses the tensor data according to the actual memory address.

[0014] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the fusion operator execution method as described above.

[0015] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the fusion operator execution method as described above.

[0016] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the fusion operator execution method as described above.

[0017] The fusion operator execution method, electronic device, storage medium, and program product provided by this invention achieve the goal of fusing multiple small-scale tensor operations into a single large-scale tensor operation by determining multiple target operators to be fused, generating a pseudo-address mapping table, performing operator fusion to obtain a fused operator, and performing address translation based on the pseudo-address mapping table during the execution of the fused operator. Due to the introduction of the pseudo-address mechanism, the fused operator can logically access data using contiguous addresses, while physically converting it to dispersed actual memory addresses through the mapping table. This avoids the significant additional memory copy overhead caused by the physical relocation and rearrangement of the original tensors in traditional operator fusion schemes. Simultaneously, the fused operator can launch more parallel computing units at once, fully utilizing hardware resources and significantly improving computing resource utilization and execution efficiency, avoiding the function call overhead and low hardware utilization problems caused by multiple small-scale operator calls. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating the fusion operator execution method provided by the present invention.

[0020] Figure 2 This is a schematic diagram of the operator computation graph provided by the present invention.

[0021] Figure 3 This is a schematic diagram of operator fusion provided by the present invention.

[0022] Figure 4 This is a schematic diagram of the structure of the fusion operator execution device provided by the present invention.

[0023] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0025] With the rapid development of artificial intelligence technology, accelerating and optimizing AI computing has become crucial for improving overall computational efficiency. In practical applications, it is often necessary to accelerate multiple continuous AI computations with no data dependencies and small input tensors to meet the demands for real-time performance and high performance.

[0026] Currently, for this type of computational task, the relevant technologies mainly adopt two technical solutions: one is to integrate multiple operations into a single operation with multiple inputs; the other is to use generalized matrix multiplication (group GEMM) for the operation.

[0027] However, the first approach requires rewriting the operator code for different numbers of inputs, or using complex logic within the operator to determine the number of inputs and operators. This not only increases the difficulty of development and maintenance but also reduces the operator's versatility and scalability. The second approach, group GEMM, has relatively complex code writing and limited application scope, only applicable to GEMM operations and unable to effectively support other types of AI computational operations.

[0028] In summary, when dealing with continuous, data-independent AI computations involving multiple small tensors, the relevant technologies generally suffer from problems such as high code complexity, poor versatility, and limited applicability, making it difficult to meet the demands for flexibility and efficiency in practical applications.

[0029] In response, this invention provides a fusion operator execution method, which aims to achieve efficient acceleration and flexible optimization of small tensor AI computation by distributing multiple consecutive AI computation tasks into multiple pipelines for parallel execution, without rewriting operator code or introducing complex logical judgments.

[0030] in, Figure 1 This is a flowchart illustrating the fusion operator execution method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0031] Step 110: Determine multiple target operators to be fused.

[0032] During neural network inference or training, multiple consecutive operators of the same type with no data dependencies and small input tensors are frequently encountered. These operators can be considered as the target operators in this embodiment. If these target operators are executed individually, the small tensor size will lead to low utilization of computational resources.

[0033] For example, in a neural network layer with a multi-head attention mechanism, eight independent linear transformations need to be performed on the input features, each corresponding to an attention head. These eight linear transformation operators have the same input tensor shape, the same operational logic, and no data dependencies between them. If each linear transformation is executed individually, only a small tensor can be processed at a time, which cannot fully utilize the parallel computing capabilities of GPUs or other accelerators.

[0034] As an optional implementation, the computation graph of the neural network can be scanned first using a graph compiler, which analyzes the data flow relationships and dependencies between the operators in the computation graph. If multiple operators are identified as belonging to the same type (e.g., all are convolution operators, fully connected operators, activation function operators, etc.), or if the input and output tensors of multiple operators have the same or similar shapes, or if there are no data dependencies between multiple operators, or if the tensors of multiple operators do not overlap in memory, or if multiple operators share a common input operator, these operators can be used as target operators for fusion.

[0035] in, Figure 2 This is a schematic diagram of the operator computation graph provided by the present invention, such as... Figure 2 As shown, the output tensors of the mem_op operator include tensor0, tensor1, tensor2, ..., tensorn. These tensors serve as inputs to multiple operators of the same type (op0, op1, op2, ..., opn). Each operator executes independently and produces a corresponding output tensor (output0, output1, output2, ..., outputn). Finally, these output tensors are aggregated into another mem_op operator. In this case, op0, op1, op2, ..., opn can be considered as the target operators to be fused.

[0036] Step 120: Generate a pseudo-address mapping table based on the tensors of multiple target operators. The pseudo-address mapping table is used to record the mapping relationship between the actual memory address and pseudo address of the tensor.

[0037] In traditional operator execution, each tensor has its own actual virtual address in physical memory, and these addresses are typically scattered and non-contiguous. For example, the input tensor of the first operator might be stored at address 0x10000, while the input tensor of the second operator might be stored at address 0x50000, with a large address jump between them. This address discontinuity makes operator fusion difficult because the fused operator needs to treat multiple small tensors as a single logically large tensor, and accessing large tensors usually assumes that the data is stored contiguously in memory.

[0038] To address this issue, this embodiment introduces pseudo-addresses, which are logically contiguous addresses used to uniformly represent multiple small tensors within the fusion operator. Specifically, even if the addresses of multiple small tensors are scattered in actual memory, the pseudo-address mechanism can map them to a single contiguous logical address. This allows the fusion operator to be written as if accessing a contiguous large tensor, while during actual execution, the pseudo-address is dynamically converted to the actual memory address using a pseudo-address mapping table.

[0039] The process of generating a pseudo-address mapping table first requires collecting the actual memory addresses of the input and output tensors of multiple target operators. For example... Figure 2 As shown, for n target operators op0, op1, op2, ..., opn, it is necessary to obtain the actual memory addresses VA_input0, VA_input1, VA_input2, ..., VA_inputn for their input tensors tensor0, tensor1, tensor2, ..., tensorn respectively. Similarly, it is also necessary to obtain the actual memory addresses of the output tensors output0, output1, output2, ..., outputn of these operators, denoted as VA_output0, VA_output1, VA_output2, ..., VA_outputn.

[0040] After collecting the actual memory addresses of all tensors, the next step is to allocate contiguous pseudo-addresses for the input and output tensors. The allocation of pseudo-addresses follows the principle of continuity, that is, the first tensor is allocated a starting pseudo-address, the second tensor is allocated a pseudo-address immediately following the first tensor, and so on.

[0041] Specifically, a base pseudo-address can be set as the starting point of the pseudo-address space, for example, setting the base pseudo-address to 0. Then, according to the number of target operators and the size of each tensor, pseudo-address spaces are allocated sequentially for each input tensor. Assuming the size of each input tensor is input_tensor_size bytes, the pseudo-address range corresponding to the first input tensor tensor0 is [0, input_tensor_size), the pseudo-address range corresponding to the second input tensor tensor1 is [input_tensor_size, 2input_tensor_size), and the pseudo-address range corresponding to the i-th input tensor tensorori is [iinput_tensor_size, (i+1)input_tensor_size). Similarly, it is also necessary to allocate continuous pseudo-addresses for the output tensors, using the same method as allocating continuous pseudo-addresses for the input tensors, which will not be repeated here.

[0042] In some implementations, if the input tensor and the output tensor are the same size, they can be assigned the same pseudo-address range partitioning method. However, in order to clearly distinguish between input and output, separate pseudo-address mapping tables are usually maintained for them.

[0043] A mapping relationship is established between pseudo-addresses and actual memory addresses. This mapping relationship can be recorded in a data structure for fast lookup during subsequent fusion operations. To reduce the overhead of table lookups, the pseudo-address mapping table is usually defined as a simple one-dimensional array.

[0044] For example, we can define two one-dimensional arrays, `input_map` and `output_map`, where `input_map` records the mapping relationships of input tensors, and `output_map` records the mapping relationships of output tensors. Specifically, `input_map[i]` stores the actual memory address `VA_inputi` of the i-th input tensor `tensori`, and `output_map[i]` stores the actual memory address `VA_outputi` of the i-th output tensor `outputi`. In this way, given a pseudo-address, we can first calculate which tensor corresponds to that pseudo-address, and then quickly obtain the actual starting memory address of that tensor using either `input_map[i]` or `output_map[i]`.

[0045] In different implementations, the data structure of the pseudo-address mapping table can vary. If the sizes of the tensors are not exactly the same, the pseudo-address mapping table needs to record not only the actual memory address but also the starting pseudo-address and size of each tensor. In this case, the mapping table may be an array of structures, with each element containing three fields: actual address, starting pseudo-address position, and size. When the sizes of the tensors are the same, the actual memory address can be recorded.

[0046] Step 130: Perform operator fusion on multiple target operators to obtain a fused operator. The memory address of the tensor of the fused operator is a contiguous pseudo address.

[0047] Specifically, after generating the pseudo-address mapping table, operator fusion is performed on multiple target operators. Operator fusion integrates multiple small-scale target operators into a large-scale unified operator for computation, thereby improving the utilization of computing resources and execution efficiency. In traditional neural network execution, each operator is called as an independent kernel or function. Even if multiple operators execute the same computational logic, they need to be started multiple times. This approach incurs overhead from multiple function calls, multiple memory accesses, and low utilization of hardware computing units due to the small data scale of each operation. For example, when performing N small-scale matrix multiplications on a GPU, only a small portion of the GPU's computing cores are activated each time, while most cores remain idle, resulting in wasted resources.

[0048] Operator fusion combines N small-scale operations into one large-scale operation, allowing a sufficient number of threads or computing units to be launched at once to make full use of hardware resources.

[0049] As an optional implementation, the graph compiler scans the computation graph, identifies the multiple target operators to be fused as determined in the preceding steps, deletes the operator nodes corresponding to the target operators from the computation graph, and then inserts a new fusion operator node at the positions of these operator nodes. This fusion operator node logically represents the overall computational task of the original multiple target operators; its input is all the input tensors of the original multiple target operators, and its output is all the output tensors of the original multiple target operators. However, unlike the original, the fusion operator integrates the computational logic of multiple target operators into a single execution function and uses contiguous pseudo-addresses to access input and output data.

[0050] In the internal implementation of the fusion operator, the computational logic needs to be adjusted to adapt to the pseudo-address access mode. While individual operators typically access specific input and output tensors directly, the fusion operator integrates the computation processes of multiple target operators. This can be achieved by logically concatenating all input tensors into a single large input tensor and all output tensors into a single large output tensor, followed by writing a computational loop to process the large tensor. Throughout this process, all address accesses are based on pseudo-addresses, assuming that the input and output tensors are stored in contiguous memory addresses.

[0051] It's important to note that traditional operator fusion schemes may require complex address jumps and conditional checks within the fusion operator itself. For example, determining which original tensor the currently processed element belongs to and then jumping to different memory addresses based on the specific circumstances. This complex control flow can lead to performance issues such as branch prediction failures and instruction pipeline stalls. However, the fusion operator in this embodiment uses contiguous pseudo-addresses. From the operator's perspective, all data is stored contiguously, eliminating the need for any branch checks and allowing for a simple sequential access pattern. This sequential access pattern is highly compatible with modern processor cache systems and prefetching mechanisms, resulting in significantly improved memory access performance.

[0052] Step 140: Execute the fusion operator, and during the execution process, convert the pseudo address to the actual memory address according to the pseudo address mapping table, and access the tensor data according to the actual memory address.

[0053] Specifically, during the internal execution of the fusion operator, there is a critical address translation step, which requires converting the continuous pseudo-addresses used by the fusion operator into the actual memory addresses of tensors.

[0054] As an optional embodiment, the original tensor corresponding to the currently accessed pseudo-address can be determined first, and the actual memory address of the original tensor can be found according to the pseudo-address mapping table established in step 120. Since the actual memory address records the starting position of the original tensor in memory, while the pseudo-address may point to the location of a specific element within the tensor, to obtain the data of the original tensor, the offset of the pseudo-address within the tensor needs to be added to the base address to locate the specific data element. Therefore, the offset of the actual access address can be calculated, and the base address can be added to the offset to obtain the final actual physical address, thereby allowing the data of the corresponding original tensor to be read according to the actual physical address. Here, the actual physical address refers to the virtual address of the actual storage location of the tensor data element in physical memory. This address can be directly used by the processor for memory read and write operations to access the specific value of the element.

[0055] The fusion operator execution method provided in this embodiment achieves the goal of merging multiple small-scale tensor operations into a single large-scale tensor operation by determining multiple target operators to be merged, generating a pseudo-address mapping table, performing operator fusion to obtain a fusion operator, and performing address translation based on the pseudo-address mapping table when executing the fusion operator. Due to the introduction of the pseudo-address mechanism, the fusion operator can logically access data using contiguous addresses, while physically converting it to dispersed actual memory addresses through the mapping table. This avoids the significant additional memory copy overhead caused by the physical relocation and rearrangement of the original tensors in traditional operator fusion schemes. Simultaneously, the fused operator can launch more parallel computing units at once, fully utilizing hardware resources and significantly improving computing resource utilization and execution efficiency, avoiding the function call overhead and low hardware utilization problems caused by multiple small-scale operator calls.

[0056] Furthermore, since the overhead of converting pseudo-addresses to actual addresses is relatively small and can be highly optimized, this embodiment achieves a significant performance improvement while ensuring correctness, making it particularly suitable for application scenarios such as neural network inference and training that require processing a large number of small tensor operations.

[0057] It should be noted that the execution subject of this embodiment can be any electronic device capable of operator fusion, such as a smartphone, smartwatch, desktop computer, laptop, server, etc. In this embodiment, the target operator can be regarded as a basic computational unit or component in a neural network model. The neural network model can be applied to fields such as text processing, image processing, speech processing, and video processing. The neural network model can be a convolutional neural network (CNN), a recurrent neural network (RNN), a Transformer model, a generative adversarial network (GAN), etc.

[0058] Taking the application of neural network models to text processing as an example, the aforementioned target operators can be any computational unit used, input, or generated in tasks such as text classification, sentiment analysis, text generation, and machine translation. Examples include fully connected layer operators, activation function operators, and attention mechanism operators. The tensors processed by these operators can be word embedding vectors, hidden state features, attention weights, etc. Based on the method described in the above embodiments, multiple small-scale operators of the same type are fused into a single large-scale operator to efficiently process the features of the input text, obtaining computational results. The model then uses these results to perform inference, processing the input text to obtain the corresponding output text or classification results.

[0059] Taking the application of neural network models to image processing as an example, the aforementioned target operators can be any computational unit used, input, or generated in tasks such as image classification, object detection, image segmentation, and image generation, such as convolution operators, pooling operators, and normalization operators. The tensors processed by these operators can be image feature maps, convolution kernels, activation values, etc. Based on the method of the above embodiments, the convolution operations of multiple small-sized feature maps are fused into a single large-batch convolution operation to efficiently transform the input image features and obtain the calculation results. The model performs inference based on the calculation results, processes the input image, and obtains the corresponding output results, such as classification labels, bounding boxes, segmentation masks, and generated images.

[0060] Taking the application of neural network models to speech processing as an example, the aforementioned target operators can be any computational unit used, input, or generated in tasks such as speech recognition, speech synthesis, speech enhancement, and speaker recognition, such as one-dimensional convolution operators, recurrent unit operators, and attention operators. The tensors processed by these operators can be spectral features, temporal features, or encoding vectors of speech. Based on the method of the above embodiments, the small tensor operations corresponding to multiple short-term speech segments are fused into large tensor operations of a long sequence, which efficiently processes the input speech signal to obtain the corresponding output text, synthesized speech, or enhanced speech signal.

[0061] Furthermore, the execution entity in this embodiment can also be an AI acceleration chip or dedicated computing hardware such as a GPU, GPGPU, TPU, NPU, or FPGA. These chips are typically hardware-optimized for specific types of parallel computation and have a large number of parallel computing units. This embodiment combines multiple small-scale operations into a single large-scale operation through operator fusion, which can fully utilize the large number of parallel computing units on these chips and significantly improve hardware utilization. For example, on a GPU, the fused operator can launch more CUDA threads, allowing more stream processors to work simultaneously; on a TPU, the fused operator can better utilize the computational power of the matrix multiplication unit; on an NPU, the fused operator can improve the parallelism of the vector operation unit and the tensor operation unit. In this way, this embodiment can achieve performance optimization on various hardware platforms, accelerating the inference and training process of neural networks.

[0062] Based on the above embodiments, a pseudo-address mapping table is generated according to the tensors of multiple target operators, including: Collect the actual memory addresses of the input and output tensors of multiple target operators; Assign contiguous pseudo-addresses to the input and output tensors; Establish a mapping relationship between pseudo addresses and actual memory addresses, and record it in the pseudo address mapping table.

[0063] During the execution of a neural network, each tensor is allocated a memory space to store its data, and the starting address of this memory space is the actual memory address of the tensor. For multiple target operators to be fused, it is necessary to collect the address information of all input and output tensors involved in them.

[0064] Specifically, the collection process can be accomplished using a graph compiler. During the compilation of a neural network, the graph compiler constructs a computation graph where each operator node contains references to its input and output tensors. These references typically point to tensor descriptors, which record various attributes of the tensor, such as shape, data type, storage format, and memory address. By traversing the target operator nodes to be fused, their input and output tensor descriptors can be obtained, and the actual memory addresses can then be extracted from these descriptors.

[0065] Furthermore, considering that the pseudo-address space needs to be divided according to the tensor size when allocating consecutive pseudo-addresses, the size information of each tensor can also be collected synchronously when collecting actual memory addresses.

[0066] During compilation, if the memory allocation of a tensor is statically determined, address collection and mapping table generation can be completed during the compilation phase, and the generated mapping table can be embedded as constant data into the compiled executable code. At runtime, if the memory allocation of a tensor is dynamic, address information needs to be collected and a mapping table generated in real time before each execution of the fusion operator.

[0067] After collecting the actual memory addresses and size information of all tensors, the next step is to allocate contiguous pseudo-addresses to these tensors. The allocation of pseudo-addresses follows the principles of continuity and order, with the aim of constructing a logically contiguous address space so that the fusion operator can access multiple small tensors as if they were a single large tensor.

[0068] For input tensors, pseudo-address allocation can start from the base pseudo-address. If the base pseudo-address is set to 0, representing the starting point of the pseudo-address space, the pseudo-address range allocated to the first input tensor is [0, input_tensor_size), meaning it starts at pseudo-address 0 and ends at input_tensor_size. This range is exactly equal to the actual size of the first input tensor. The pseudo-address range allocated to the second input tensor is [input_tensor_size, 2×input_tensor_size), meaning it immediately follows the pseudo-address of the first input tensor. Similarly, the pseudo-address range allocated to the third input tensor is [2×input_tensor_size, 3×input_tensor_size). And so on, the pseudo-address range allocated to the i-th input tensor, tensor_in_i, is [i×input_tensor_size, (i+1)×input_tensor_size).

[0069] For output tensors, the pseudo-address allocation is similar to that of input tensors, but a separate pseudo-address space can be used. The pseudo-address of output tensors can also be allocated starting from 0. The pseudo-address range for the first output tensor is [0, output_tensor_size), the range for the second output tensor is [output_tensor_size, 2×output_tensor_size), and the range for the i-th output tensor is [i×output_tensor_size, (i+1)×output_tensor_size).

[0070] In some embodiments, if the input tensors of the target operators to be fused are not of the same size, the allocation of pseudo-addresses needs to be based on the actual size of each tensor. For example, assuming there are three input tensors with sizes of 1000 bytes, 2000 bytes, and 1500 bytes, the pseudo-addresses are allocated as follows: the pseudo-address range of the first input tensor is [0, 1000), the pseudo-address range of the second input tensor is [1000, 3000), and the pseudo-address range of the third input tensor is [3000, 4500].

[0071] In one implementation, in addition to recording the actual memory address, the pseudo address mapping table also records the start and end positions of the pseudo address for each tensor, and the tensor index corresponding to the pseudo address is determined by range lookup.

[0072] After allocating pseudo-addresses, a mapping relationship between pseudo-addresses and actual memory addresses is established, and this mapping relationship is recorded in a pseudo-address mapping table. To enable fast address lookup and conversion during the execution of the fusion operator, the mapping table can be a one-dimensional array structure. For input tensors, an array `input_map` is defined, where `input_map[i]` stores the actual starting address of the i-th input tensor in memory. For output tensors, an array `output_map` is defined, where `output_map[i]` stores the actual starting address of the i-th output tensor in memory.

[0073] Based on any of the above embodiments, assigning contiguous pseudo-addresses to the input tensor and the output tensor includes: Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the input tensor, and the size of each pseudo-address is consistent with the size of the corresponding input tensor. Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the output tensor, and the size of each pseudo-address is consistent with the size of the corresponding output tensor.

[0074] In some embodiments, the process of allocating contiguous pseudo-addresses to the input and output tensors can be further refined into allocating independent pseudo-addresses for each tensor. This separate allocation strategy provides clear boundaries between input and output data in the pseudo-address space, facilitating address management and access control within the fusion operator, while also offering flexibility for subsequent optimization. For example, on some hardware platforms, a more aggressive prefetching strategy can be used for read-only input data, while write-back cache consistency needs to be considered for writable output data. By separating the pseudo-address spaces of input and output, different optimization methods can be used for different access characteristics during the implementation of the fusion operator without mutual interference.

[0075] Specifically, when allocating contiguous pseudo-addresses to input tensors, the first step is to determine the number of target operators, N. Since each target operator has one or more input tensors, pseudo-addresses need to be allocated to these input tensors. The allocation of pseudo-addresses follows the principles of continuity and size matching. Continuity means that the pseudo-addresses of each input tensor are logically adjacent without gaps, thus forming a continuous range of pseudo-addresses. Size matching means that the size of the pseudo-address space allocated to each input tensor should be consistent with the actual data size of that input tensor. This ensures that the pseudo-address space can completely cover all input data, avoiding both address waste and address shortage.

[0076] It is important to emphasize that the pseudo-address spaces of the input and output tensors are independent of each other. That is, since the input and output tensors correspond to different mapping tables, the pseudo-addresses of the input and output tensors can use the same numerical range without conflict. For example, the pseudo-address 1000 of the input tensor and the pseudo-address 1000 of the output tensor are two completely different addresses. The former is converted to the actual memory address of a certain input tensor through the pseudo-address mapping table of the input tensor, while the latter is converted to the actual address of a certain output tensor through the pseudo-address mapping table of the output tensor.

[0077] By employing this refined pseudo-address allocation strategy, this embodiment can accurately match the actual size of each tensor while maintaining address continuity, avoiding wasted address space and data access errors caused by insufficient address range. Furthermore, the input and output tensors utilize independent pseudo-address spaces, providing a clear interface and flexible optimization space for the implementation of the fusion operator.

[0078] Based on any of the above embodiments, multiple target operators to be fused are determined, including: The graph compiler scans the computation graph to identify multiple target operators that satisfy a preset pattern. The preset pattern includes multiple target operators being operators of the same type and / or multiple target operators having input and output tensors with the same shape.

[0079] Specifically, a graph compiler is used to parse, analyze, optimize, and generate code for the computation graph of a neural network. When scanning a computation graph, a graph compiler can identify metadata information such as the type information of each operator, the shape of the input and output tensors, the data dependencies between operators, the parameter configuration of operators, and the topological position of operators in the computation graph.

[0080] Considering that not all operators are suitable for fusion, inappropriate operator fusion may lead to excessive complexity in the implementation of the fusion operator, insignificant fusion benefits, or even performance degradation. Therefore, the target operators in this embodiment need to meet a preset mode, which includes multiple target operators being operators of the same type and / or multiple target operators having the same shape in their input and output tensors.

[0081] In other words, the target operators to be fused can satisfy only the condition that they are operators of the same kind, or only the condition that the input tensor and the output tensor have the same shape, or both conditions can be satisfied at the same time.

[0082] Among them, having multiple target operators of the same type ensures that the fused operators adopt unified computational logic and unified code implementation, avoiding the need to execute different operation branches according to different operator types within the fused operator. This simplifies the implementation of the fused operator and improves execution efficiency, avoiding problems such as control flow complexity, branch prediction failure, and code bloat caused by different operator types.

[0083] Having the same shape for the input and output tensors of multiple target operators means that the input tensors and output tensors of each target operator have completely identical sizes across all dimensions. For example, all input tensors have the shape [batch_size, channels, height, width], and all output tensors also have the shape [batch_size, channels, height, width]. This allows the fusion operator to use a uniform address stride and index calculation method when allocating pseudo-addresses, simplifying the conversion logic from pseudo-addresses to actual addresses. It avoids problems such as complex pseudo-address mapping table structures, increased address translation overhead, irregular memory access patterns, and decreased cache utilization caused by different tensor shapes.

[0084] Based on any of the above embodiments, a graph compiler scans the computation graph to identify multiple target operators that satisfy a preset pattern, including: The graph compiler scans the computation graph and identifies operators that have no data dependencies and whose tensors do not overlap in memory as candidate operators. Candidate operators that satisfy the preset pattern are used as target operators.

[0085] Specifically, considering that the premise of operator fusion is that multiple operators can be executed in parallel or their execution order can be reorganized without affecting the correctness of the calculation results, if there are data dependencies between the operators to be fused, such as the output of one operator being the input of another operator, or the tensors of multiple operators having overlapping regions in memory, causing the write operation of one operator to affect the read operation of another operator, then data races, read-write conflicts, or calculation errors may occur during fusion execution, which may lead to the fusion operator failing to execute correctly.

[0086] Therefore, in this embodiment, during the scanning calculation, operators that do not have data dependencies and whose tensors do not overlap in memory are identified, and these operators are selected as candidate operators. Candidate operators can be considered as a set of operators that have passed basic security checks and are feasible for fusion; they satisfy the necessary conditions for fusion but not necessarily the sufficient conditions.

[0087] Operators that do not have data dependencies are those that are independent of each other in the topology of the computation graph, where the output tensor of any one operator is not the input tensor of any other operator, meaning there is no direct or indirect data flow connection between these operators.

[0088] Operators whose tensors do not overlap in memory are those whose input and output tensors occupy memory address spaces that do not intersect, meaning that there is no situation where two tensors share the same memory region.

[0089] Because candidate operators possess both the characteristics of no data dependency and non-overlapping memory addresses, they can be safely executed in parallel or in fusion without one operator modifying the data being used by another, or causing errors in the calculation results due to changes in the execution order.

[0090] Furthermore, while candidate operators may meet the safety requirements for fusion, this does not mean that fusing all candidate operators will bring performance benefits. If the candidate operators differ significantly in type, input / output shape, or computational complexity, using them as target operators may lead to an overly complex implementation of the fusion operator. This would require writing a large amount of conditional branch code within the fusion operator for different operator types and shapes, increasing the difficulty of code maintenance and resulting in the fusion operator's execution efficiency being lower than that of executing each independent operator separately.

[0091] To address this, this embodiment further selects operators that meet a preset pattern from the candidate operators as target operators. This ensures that the final selected target operators not only meet the security requirements of fusion but also the efficiency requirements of fusion. That is, these operators have similar computational characteristics and consistent data scale, so that the fused operators can adopt a unified and efficient implementation method, maximizing the performance benefits brought by fusion while minimizing the implementation complexity and maintenance costs brought by fusion.

[0092] Based on any of the above embodiments, converting pseudo-addresses to actual memory addresses according to a pseudo-address mapping table, and accessing tensor data according to the actual memory addresses, includes: The lookup index corresponding to the pseudo-address is obtained by calculating the thread index or tensor index. Based on the lookup index, the corresponding actual memory address is searched and obtained from the pseudo-address mapping table; Based on the actual memory address, access the data of the corresponding tensor to perform read or write operations.

[0093] Specifically, a thread index is a unique identifier for each execution thread or computation unit in a parallel computing environment, used to distinguish different parallel execution entities. It can be calculated using built-in variables or functions provided by the hardware. A tensor index is the position index of a data element in the logical tensor processed by the fusion operator. It is used to locate the relative position of the element in the entire fusion tensor. It can be obtained directly by mapping from the thread index or calculated through multidimensional index calculation.

[0094] Furthermore, the lookup index refers to the index value used to locate a specific entry in the pseudo-address mapping table. It indicates the original tensor corresponding to the currently accessed pseudo-address. Since the thread index directly corresponds to a certain position in the pseudo-address space in the fusion operator, by performing division or shift operations on the thread index, we can calculate which tensor range that position belongs to, and thus obtain the corresponding lookup index based on the thread index.

[0095] Since a tensor index also represents the position of a data element in a logically fused tensor, the original tensor corresponding to the tensor index can be determined through a similar division operation, and the corresponding lookup index can be obtained based on the tensor index.

[0096] After determining the lookup index, the corresponding actual memory address can be obtained from the pseudo-address mapping table based on the lookup index. After obtaining the actual memory address, the offset within the tensor can be calculated, and the base address can be added to the offset to obtain the final physical access address. Then, the data in the corresponding tensor can be accessed for read or write operations.

[0097] Based on any of the above embodiments, multiple target operators are fused to obtain a fusion operator, including: The pseudo-address mapping table is received as input parameters, and the computational logic of multiple target operators is integrated into a single computational process to obtain the fusion operator; In the calculation process, the access addresses of the input and output tensors of multiple target operators are pseudo-addressed, so that the fusion operator can logically access consecutive pseudo-addresses.

[0098] Specifically, considering that the input and output tensors of multiple target operators are stored separately in actual memory and each has a different actual memory address, and the fusion operator needs to process these scattered small tensors into a logically continuous large tensor, a pseudo-address mapping table needs to be used as an input parameter when performing operator fusion. This allows the pseudo-address mapping table to be dynamically queried at runtime to obtain the actual memory address of each tensor when integrating the computational logic of multiple target operators into a single computational process. This achieves the correct conversion from continuous pseudo-addresses to scattered actual memory addresses, ensuring that the fusion operator can correctly access the data of each tensor while maintaining the simplicity of the code logic and execution efficiency.

[0099] In this process, pseudo-addressing refers to converting the operation of directly accessing tensor data using actual memory addresses in the original target operator into an operation of accessing it using continuous pseudo addresses. This creates a unified and continuous address access pattern within the fusion operator, eliminating the need to consider the actual distribution of each small tensor in physical memory. As a result, the fusion operator can be computed at the logical level as if it were processing a single large-scale continuous tensor. All data access is based on pseudo-address indices that are continuously incremented from 0, simplifying index calculation and loop control logic.

[0100] Based on any of the above embodiments, the pseudo-address mapping table includes an input mapping table and an output mapping table. The input mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the input tensor, and the output mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the output tensor.

[0101] Specifically, if input and output tensors are managed using the same mapping table, the number, size, and access patterns of input and output tensors are usually different. Mixing them in the same table increases lookup complexity, requires additional flags to distinguish whether the current access is for input or output, and is prone to address calculation conflicts in concurrent access scenarios. This causes the fusion operator to frequently determine the tensor type corresponding to the current pseudo-address during execution, increasing the probability of branch prediction failure and reducing instruction pipeline efficiency. At the same time, it also complicates the index calculation of the mapping table, making it impossible to fully utilize simple division or bitwise operations for fast address translation, thus affecting overall execution performance.

[0102] Based on this, this embodiment stores the mapping relationship of the input tensor and the mapping relationship of the output tensor separately in independent mapping tables to avoid the above-mentioned problems of address lookup conflict and increased computational complexity. This allows the fusion operator to only query the input mapping table when accessing input data and only query the output mapping table when accessing output data, with the two not interfering with each other, and the lookup logic being clear and concise.

[0103] That is, the mapping relationship between the actual memory address and pseudo address of the input tensor is stored in the input mapping table, and the mapping relationship between the actual memory address and pseudo address of the output tensor is stored in the output tensor mapping table. Thus, when the fusion operator is executed, the corresponding mapping table can be directly selected for querying according to the current operation type without the need for additional type judgment overhead.

[0104] In addition, the data layout and caching strategies of the input and output mapping tables can be optimized respectively. For example, the frequently accessed input mapping table can be placed in a faster storage area, and the output mapping table can be placed in a storage area suitable for write operations, thereby further improving memory access efficiency.

[0105] in, Figure 3 This is a schematic diagram of operator fusion provided by the present invention, as shown below. Figure 3 As shown, first, the input tensor is fed into the splitting operator (mem_op) to split it into multiple smaller sub-input tensors (in_tensor 0, in_tensor 1, ..., in_tensor n).

[0106] Then, these sub-input tensors, along with metadata describing their organization, are passed as input to the fused op. The fused op uses the metadata to process all the sub-input tensors in parallel or sequentially within a single kernel execution, generating multiple corresponding sub-output tensors (out_tensor 0, out_tensor 1, ..., out_tensor_orn). The input map tells the fused op how to locate and access each sub-input tensor in the original, potentially contiguous, memory blocks. The metadata includes the input map, the input tensor size (input_tensor_size), and the number of input tensors (input_tensor_num).

[0107] Next, all the sub-output tensors are passed to the merge operator (mem_op) for memory merging. This operation reassembles and splices these scattered sub-output tensors into a single, complete output tensor based on metadata such as the output map, output tensor size, and number of output tensors (output_tensor_num), thus completing the entire computation process.

[0108] The fusion operator execution device provided by the present invention is described below. The fusion operator execution device described below and the fusion operator execution method described above can be referred to in correspondence.

[0109] Based on any of the above embodiments Figure 4 This is a schematic diagram of the structure of the fusion operator execution device provided by the present invention, as shown below. Figure 4 As shown, the device includes: Determining unit 410 is used to determine multiple target operators to be fused; The generation unit 420 is used to generate a pseudo-address mapping table based on the tensors of multiple target operators. The pseudo-address mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the tensor. The fusion unit 430 is used to fuse multiple target operators to obtain a fusion operator. The memory address of the tensor of the fusion operator is a contiguous pseudo address. The execution unit 440 is used to execute the fusion operator. During the execution process, it converts the pseudo address into the actual memory address according to the pseudo address mapping table and accesses the tensor data according to the actual memory address.

[0110] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute the fusion operator execution method.

[0111] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0112] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the fusion operator execution method provided by the above methods.

[0113] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the fusion operator execution method provided by the above methods.

[0114] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0116] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for executing a fusion operator, characterized in that, include: Identify multiple target operators to be fused; Based on tensors of multiple target operators, a pseudo-address mapping table is generated. The pseudo-address mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the tensor. The multiple target operators are fused to obtain a fused operator, and the memory address of the tensor of the fused operator is a contiguous pseudo address. The fusion operator is executed, and during the execution process, the pseudo address is converted into the actual memory address according to the pseudo address mapping table, and the tensor data is accessed according to the actual memory address.

2. The fusion operator execution method according to claim 1, characterized in that, The step of generating a pseudo-address mapping table based on tensors of multiple target operators includes: Collect the actual memory addresses of the input and output tensors of the multiple target operators; Assign contiguous pseudo-addresses to the input and output tensors; Establish a mapping relationship between the pseudo address and the actual memory address, and record it in the pseudo address mapping table.

3. The fusion operator execution method according to claim 2, characterized in that, Assigning contiguous pseudo-addresses to the input and output tensors includes: Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the input tensor, and the size of each pseudo-address is consistent with the size of the corresponding input tensor. Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the output tensor, and the size of each pseudo-address is consistent with the size of the corresponding output tensor.

4. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The determination of multiple target operators to be fused includes: The graph compiler scans the computation graph to identify multiple target operators that satisfy a preset pattern. The preset pattern includes that the multiple target operators are operators of the same type and / or that the input tensors and output tensors of the multiple target operators have the same shape.

5. The fusion operator execution method according to claim 4, characterized in that, The step of scanning the computation graph using a graph compiler to identify the plurality of target operators that satisfy a preset pattern includes: The computation graph is scanned by the graph compiler, and operators that have no data dependencies between operators and whose tensors do not overlap in memory are selected as candidate operators. The candidate operator that satisfies the preset mode is taken as the target operator.

6. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The step of converting the pseudo-address to the actual memory address according to the pseudo-address mapping table, and accessing the tensor data according to the actual memory address, includes: The lookup index corresponding to the pseudo-address is obtained by calculating the thread index or tensor index. Based on the lookup index, the corresponding actual memory address is searched and obtained from the pseudo-address mapping table; Based on the actual memory address, access the data of the corresponding tensor to perform read or write operations.

7. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The step of fusing the multiple target operators to obtain a fusion operator includes: The pseudo-address mapping table is received as an input parameter, and the calculation logic of the multiple target operators is integrated into a single calculation process to obtain the fusion operator; In the calculation process, the access addresses of the input tensors and output tensors of the multiple target operators are pseudo-addressed, so that the fusion operator logically accesses consecutive pseudo-addresses.

8. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The pseudo-address mapping table includes an input mapping table and an output mapping table. The input mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the input tensor, and the output mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the output tensor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the fusion operator execution method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the fusion operator execution method as described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the fusion operator execution method as described in any one of claims 1 to 8.

Images