Fusion operator execution method, electronic device, storage medium, and program product

Abstract

The application relates to the technical field of artificial intelligence, and provides a fusion operator execution method, an electronic device, a storage medium and a program product.The method comprises the following steps: determining a plurality of target operators to be fused; generating a pseudo address mapping table according to tensors of the plurality of target operators, the pseudo address mapping table being used for recording a mapping relationship between actual memory addresses of the tensors and pseudo addresses; performing operator fusion on the plurality of target operators to obtain a fusion operator, and the memory address of the tensor of the fusion operator being a continuous pseudo address; executing the fusion operator, and in the execution process, converting the pseudo address into an actual memory address according to the pseudo address mapping table, and accessing data of the tensor according to the actual memory address. The application avoids a large amount of additional memory copying overhead caused by physical moving and rearranging of original tensors in a traditional operator fusion scheme, and significantly improves the utilization rate of computing resources and the execution efficiency.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a fusion operator execution method, an electronic device, a storage medium and a program product. BACKGROUND

[0002] Acceleration of artificial intelligence (AI) computing is one of the key technologies to improve overall computing efficiency, especially when processing specific computing tasks.

[0003] Currently, to accelerate continuous, data-independent multiple small tensor AI computing, multiple operations are usually fused into a single operation with multiple inputs. However, this method usually requires rewriting operator code for different numbers of inputs, and in addition, complex logic is also needed to determine the number of inputs inside the operator, resulting in difficult development and maintenance, complex operator logic, and poor generality. SUMMARY

[0004] The present application provides a fusion operator execution method, an electronic device, a storage medium and a program product to solve the defects in the related art.

[0005] The present application provides a fusion operator execution method, comprising the following steps:

[0006] Determine a plurality of target operators to be fused;

[0007] Generate a pseudo address mapping table according to the tensors of the plurality of target operators, the pseudo address mapping table being used to record the mapping relationship between the actual memory addresses of the tensors and the pseudo addresses;

[0008] Operator fusion is performed on the plurality of target operators to obtain a fusion operator, and the memory address of the tensor of the fusion operator is a continuous pseudo address;

[0009] The fusion operator is executed, and in the execution process, the pseudo address is converted into the actual memory address according to the pseudo address mapping table, and the data of the tensor is accessed according to the actual memory address.

[0010] According to the fusion operator execution method provided by the present application, the pseudo address mapping table is generated according to the tensors of the plurality of target operators, comprising:

[0011] Collect the actual memory addresses of the input tensors and output tensors of the plurality of target operators;

[0012] Assign continuous pseudo addresses to the input tensors and output tensors;

[0013] Establish the mapping relationship between the pseudo addresses and the actual memory addresses, and record it to the pseudo address mapping table.

[0014] The fusion operator execution method provided by the application assigns continuous pseudo addresses to the input tensor and the output tensor, and comprises the following steps:

[0015] According to the number of target operators, a corresponding number of continuous pseudo addresses are assigned to the input tensor, and the space size of each pseudo address is consistent with the size of the corresponding input tensor;

[0016] According to the number of target operators, a corresponding number of continuous pseudo addresses are assigned to the output tensor, and the space size of each pseudo address is consistent with the size of the corresponding output tensor.

[0017] The fusion operator execution method provided by the application determines a plurality of target operators to be fused, and comprises the following steps:

[0018] The plurality of target operators satisfying a preset mode are identified by scanning the computation graph through a graph compiler, and the preset mode comprises that the plurality of target operators are operators of the same type and / or the input tensor and the output tensor of the plurality of target operators have the same shape.

[0019] The fusion operator execution method provided by the application comprises the following steps:

[0020] The operators between which there is no data dependency relationship and the tensors of which do not overlap in the memory are taken as candidate operators by scanning the computation graph through a graph compiler;

[0021] The candidate operators satisfying the preset mode are taken as the target operators.

[0022] The fusion operator execution method provided by the application comprises the following steps:

[0023] The lookup index corresponding to the pseudo address is obtained through a computation thread index or a tensor index;

[0024] The corresponding actual memory address is found and obtained in the pseudo address mapping table according to the lookup index;

[0025] According to the actual memory address, the data of the corresponding tensor is accessed for reading or writing operation.

[0026] The fusion operator execution method provided by the application comprises the following steps:

[0027] receive the pseudo address mapping table as an input parameter, and integrate the calculation logics of the plurality of target operators into a single calculation flow to obtain the fusion operator;

[0028] In the calculation flow, the access addresses of the input tensor and the output tensor of the plurality of target operators are subjected to pseudo address processing, so that the fusion operator logically accesses continuous pseudo addresses.

[0029] According to the fusion operator execution method provided by the application, the pseudo address mapping table includes an input mapping table and an output mapping table, the input mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the input tensor, and the output mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the output tensor.

[0030] The application further provides a fusion operator execution device, comprising the following units:

[0031] A determination unit is configured to determine a plurality of target operators to be fused.

[0032] A generation unit is configured to generate a pseudo address mapping table according to the tensors of the plurality of target operators, the pseudo address mapping table being used to record the mapping relationship between the actual memory address and the pseudo address of the tensors.

[0033] A fusion unit is configured to fuse the plurality of target operators to obtain a fusion operator, the memory address of the tensor of the fusion operator being a continuous pseudo address.

[0034] An execution unit is configured to execute the fusion operator, and in the execution process, the pseudo address is converted into the actual memory address according to the pseudo address mapping table, and the data of the tensor is accessed according to the actual memory address.

[0035] The application further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the fusion operator execution method according to any one of the above when executing the program.

[0036] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the fusion operator execution method according to any one of the above.

[0037] The application further provides a computer program product, comprising a computer program, and the computer program is executable on a processor to implement the fusion operator execution method according to any one of the above.

[0038] The fusion operator execution method, the electronic device, the storage medium and the program product provided by the application, by determining a plurality of target operators to be fused, generating a pseudo address mapping table, performing operator fusion to obtain a fusion operator, and performing address conversion according to the pseudo address mapping table when executing the fusion operator, the target of fusing a plurality of small-scale tensor operations into a single large-scale tensor operation is achieved. Due to the introduction of the pseudo address mechanism, the fusion operator can logically access data in a continuous address manner, and physically convert into scattered actual memory addresses through the mapping table, thereby avoiding the large additional memory copy overhead caused by the physical movement and rearrangement of the original tensor in the traditional operator fusion scheme. At the same time, the fused operator can start more parallel computing units at a time, fully utilize hardware resources, significantly improve the utilization rate of computing resources and execution efficiency, and avoid the problems of function call overhead and low hardware utilization caused by multiple small-scale operator calls. BRIEF DESCRIPTION OF DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are some embodiments of the application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0040] Figure 1 is a flowchart of the fusion operator execution method provided by the application.

[0041] Figure 2 is a schematic diagram of an operator computation graph provided by the application.

[0042] Figure 3 is a schematic diagram of operator fusion provided by the application.

[0043] Figure 4 is a structural schematic diagram of the fusion operator execution device provided by the application.

[0044] Figure 5 is a structural schematic diagram of the electronic device provided by the application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical scheme and advantages of the application more clear, the technical scheme in the application will be described clearly and completely in the following combined with the drawings in the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the protection scope of the application.

[0046] With the rapid development of artificial intelligence technology, the acceleration and optimization of AI computing have become the key to improving overall computing efficiency. In actual application scenarios, it is often necessary to accelerate the processing of multiple AI computations that are continuous, data-independent, and input small tensors, in order to meet the real-time and high-performance requirements.

[0047] Currently, for such computing tasks, the related technology mainly adopts two technical solutions: one is to fuse multiple operations into a single operation with multiple inputs; the other is to use group GEMM for operation.

[0048] However, the first solution requires rewriting operator code for different numbers of inputs, or using complex logic inside the operator to determine the number of inputs and the number of operators, which not only increases the difficulty of development and maintenance, but also reduces the universality and scalability of the operator. The code writing of group GEMM in the second solution is relatively complex, and its application range is limited, only suitable for GEMM operation, and cannot effectively support other types of AI computing operations.

[0049] In summary, the related technology has the problems of high code complexity, poor universality, limited application range, etc. when processing continuous, data-independent, multiple small tensor AI computations, which is difficult to meet the requirements of flexibility and high efficiency in actual applications.

[0050] To this end, the present application provides a fusion operator execution method, which aims to distribute multiple continuous AI computing tasks into multiple pipelines for parallel execution, without rewriting operator code or introducing complex logic judgment, to achieve efficient acceleration and flexible optimization of small tensor AI computing.

[0051] Wherein, Figure 1 is a flowchart of the fusion operator execution method provided by the present application, as Figure 1 shown, the method comprises steps 110, 120, 130 and 140.

[0052] Step 110, determine a plurality of target operators to be fused.

[0053] In the process of neural network inference or training, multiple continuous, data-independent, and input small tensor operators of the same type are often encountered, which can be regarded as target operators in the present embodiment. If these target operators are executed separately, the utilization rate of computing resources will be low due to the small tensor scale.

[0054] For example, in a multi-head attention mechanism neural network layer, 8 independent linear transformations are required to be performed on input features respectively, each transformation corresponding to an attention head, the input tensor of the 8 linear transformation operators has the same shape, the same operation logic, and no data dependency between each other. If each linear transformation is executed separately, only one small tensor can be processed at a time, and the parallel computing capability of the GPU or other accelerators cannot be fully utilized.

[0055] As an optional embodiment, the computation graph of the neural network can be first scanned by a graph compiler, and the data flow relationship and dependency relationship between various operators in the computation graph are analyzed by the graph compiler. When it is identified that multiple operators are operators of the same type (such as convolution operators, fully connected operators, activation function operators, etc.), or the input tensor and the output tensor of the multiple operators have the same or similar shapes, or there is no data dependency between the multiple operators, or the tensors of the multiple operators do not overlap in the memory, or the multiple operators have a shared input operator, these operators can be taken as target operators to be fused.

[0056] wherein, Figure 2 is a schematic diagram of the operator computation graph provided by the present application, as shown in Figure 2 The output tensors of the mem_op operator include tensor0, tensor1, tensor2,..., tensorn, which are respectively taken as the inputs of multiple operators of the same type (op0, op1, op2,..., opn), the corresponding output tensors (output0, output1, output2,..., outputn) are generated after the independent execution of each operator, and finally these output tensors are converged into another mem_op operator. In this case, op0, op1, op2,..., opn can be regarded as target operators to be fused.

[0057] Step 120, generating a pseudo address mapping table according to the tensors of the multiple target operators, the pseudo address mapping table being used to record the mapping relationship between the actual memory address of the tensor and the pseudo address.

[0058] In the traditional operator execution, each tensor has its actual virtual address in the physical memory, and these addresses are usually scattered and discontinuous. For example, the input tensor of the first operator can be stored at address 0x10000, and the input tensor of the second operator can be stored at address 0x50000, and there is a large address jump between them. Such address discontinuity will bring difficulties to operator fusion, because the fused operator needs to treat multiple small tensors as a logically large tensor for processing, and the access of the large tensor usually assumes that the data is stored continuously in the memory.

[0059] To solve this problem, the embodiment introduces a pseudo address, which is a logically continuous address used to uniformly represent multiple small tensors inside the fusion operator. Specifically, even if the addresses of multiple small tensors in the actual memory are scattered, they can be mapped to a continuous logical address through the pseudo address mechanism. In this way, the fusion operator can be written in the manner of accessing a continuous large tensor when writing code, and the pseudo address is dynamically converted to the actual memory address through the pseudo address mapping table when actually executing.

[0060] The process of generating the pseudo address mapping table first needs to collect the actual memory addresses of the input and output tensors of multiple target operators. As shown in Figure 2 VA_input0, VA_input1, VA_input2,..., VA_inputn, respectively. Similarly, the actual memory addresses of the output tensors output0, output1, output2,..., outputn of these operators also need to be obtained, denoted as VA_output0, VA_output1, VA_output2,..., VA_outputn.

[0061] After collecting the actual memory addresses of all tensors, the next step is to allocate continuous pseudo addresses for the input and output tensors. The allocation of pseudo addresses follows the continuity principle, that is, the first tensor is allocated a starting pseudo address, the second tensor is allocated a pseudo address immediately after the first tensor, and so on.

[0062] Specifically, a base pseudo address can be set as the starting point of the pseudo address space, for example, the base pseudo address is set to 0. Then, according to the number of target operators and the size of each tensor, the pseudo address space is allocated to each input tensor in turn. Assuming that the size of each input tensor is input_tensor_size bytes, the pseudo address range corresponding to the first input tensor tensor0 is [0, input_tensor_size), the pseudo address range corresponding to the second input tensor tensor1 is [input_tensor_size, 2input_tensor_size), and the pseudo address range corresponding to the ith input tensor tensori is [iinput_tensor_size, (i+1)input_tensor_size). Similarly, continuous pseudo addresses also need to be allocated for output tensors, and the method is the same as that for allocating continuous pseudo addresses for input tensors, which will not be repeated here.

[0063] In some embodiments, if the input tensors and the output tensors have the same size, the same pseudo address range division method can also be assigned to them, but generally, in order to clearly distinguish the input and the output, independent pseudo address mapping tables are maintained for them.

[0064] The mapping relationship between the pseudo address and the actual memory address is established, which can be recorded in a data structure for subsequent quick query when the fusion operator is executed. In order to reduce the additional overhead of table lookup, the pseudo address mapping table is usually defined as a simple one-dimensional array.

[0065] For example, two one-dimensional arrays input_map and output_map can be defined, wherein input_map is used to record the mapping relationship of the input tensors, and output_map is used to record the mapping relationship of the output tensors. Specifically, input_map[i] stores the actual memory address VA_inputi of the ith input tensor tensori, and output_map[i] stores the actual memory address VA_outputi of the ith output tensor outputi. In this way, given a pseudo address, the number of tensors corresponding to the pseudo address can be calculated first, and then the actual memory start address of the tensor can be quickly obtained through input_map[i] or output_map[i].

[0066] In different embodiments, the data structure of the pseudo address mapping table can vary. If the sizes of the tensors are not completely the same, the pseudo address mapping table needs to record the starting pseudo address and the size of each tensor in addition to the actual memory address, so that the mapping table can be an array of structures, each element containing three fields of actual address, pseudo address starting position and size. When the sizes of the tensors are the same, only the actual memory address can be recorded.

[0067] Step 130, performing operator fusion on the plurality of target operators to obtain a fusion operator, and the memory address of the tensor of the fusion operator is a continuous pseudo address.

[0068] Specifically, after generating the pseudo address mapping table, operator fusion is performed on the plurality of target operators. Operator fusion is to integrate a plurality of small-scale target operators into a large-scale unified operator for operation, thereby improving the utilization of computing resources and execution efficiency. In traditional neural network execution, each operator is called as an independent kernel or function, and even if a plurality of operators perform the same operation logic, multiple times of starting are required. This way will bring the overhead of multiple function calls, the overhead of multiple memory accesses, and the problem of low utilization of hardware computing units due to small data scale of each operation. For example, performing N small-scale matrix multiplications on a GPU can only activate a small part of the computing cores of the GPU, and most of the cores are in an idle state, causing resource waste.

[0069] Operator fusion can fully utilize hardware resources by combining N small-scale operations into one large-scale operation.

[0070] As an optional embodiment, the graph compiler scans the computation graph, identifies the plurality of target operators to be fused determined in the foregoing steps, deletes the operator nodes corresponding to the target operators from the computation graph, and then inserts a new fusion operator node at the positions of the operator nodes. The fusion operator node logically represents the overall computing task of the original plurality of target operators, and its input is all input tensors of the original plurality of target operators, and its output is all output tensors of the original plurality of target operators. However, unlike the original, the fusion operator integrates the computing logic of the plurality of target operators into a single execution function, and uses continuous pseudo addresses to access input and output data.

[0071] In the internal implementation of the fusion operator, the computing logic needs to be adjusted accordingly to adapt to the pseudo address access mode. The original single operator usually directly accesses specific input tensors and output tensors, while in the fusion operator, the computing processes of the plurality of target operators need to be integrated together. Among them, all input tensors can be logically spliced into a large input tensor, and all output tensors can be logically spliced into a large output tensor, and then a computing loop for processing the large tensor is written. In this process, all address accesses are based on pseudo addresses, that is, it is assumed that the input tensors and the output tensors are stored in continuous addresses.

[0072] It should be noted that the traditional operator fusion scheme may need to handle complex address jumps and conditional judgments inside the fusion operator, for example, judging which original tensor the currently processed element belongs to, and then jumping to different memory addresses according to different situations. Such complex control flow will cause performance problems such as branch prediction failure, instruction pipeline stall, etc. However, the fusion operator in the embodiment uses continuous pseudo addresses, and from the perspective of the operator inside, all data is stored continuously, without any branch judgment, and a simple sequential access mode can be used. Such a sequential access mode is very friendly to the cache system and prefetch mechanism of modern processors, and can obtain better memory access performance.

[0073] Step 140, execute the fusion operator, and in the execution process, convert the pseudo address to the actual memory address according to the pseudo address mapping table, and access the data of the tensor according to the actual memory address.

[0074] Specifically, in the internal execution process of the fusion operator, there is a key address conversion link, that is, the continuous pseudo address used by the fusion operator needs to be converted to the actual memory address of the tensor.

[0075] As an optional embodiment, the original tensor corresponding to the currently accessed pseudo address can be determined first, and the actual memory address of the original tensor can be found according to the pseudo address mapping table established in step 120. Since the actual memory address records the starting position of the original tensor in the memory, and the pseudo address may point to a specific element position in the tensor, if the data of the original tensor is to be obtained, the offset of the pseudo address in the tensor needs to be added to the base address to locate the specific data element. Therefore, the offset of the actual access address can be calculated, and the base address and the offset are added to obtain the final actual physical address, so that the data of the corresponding original tensor can be read according to the actual physical address. The actual physical address refers to the virtual address of the actual storage position of the tensor data element in the physical memory, which can be directly used by the processor for memory read and write operations to access the specific numerical value of the element.

[0076] The fusion operator execution method provided in the embodiment achieves the goal of fusing multiple small-scale tensor operations into a single large-scale tensor operation by determining multiple target operators to be fused, generating a pseudo address mapping table, performing operator fusion to obtain a fusion operator, and performing address conversion according to the pseudo address mapping table when executing the fusion operator. Due to the introduction of the pseudo address mechanism, the fusion operator can logically access data in a continuous address manner, and physically convert into scattered actual memory addresses through the mapping table dynamically, thereby avoiding the large additional memory copy overhead caused by the physical movement and rearrangement of the original tensor in the traditional operator fusion scheme. At the same time, the fused operator can start more parallel computing units at a time, fully utilize hardware resources, significantly improve the utilization rate of computing resources and execution efficiency, and avoid the problems of function call overhead and low hardware utilization caused by multiple small-scale operator calls.

[0077] In addition, since the conversion overhead from the pseudo address to the actual address is relatively small and can be highly optimized, the embodiment achieves significant performance improvement while ensuring correctness, and is particularly suitable for application scenarios such as neural network inference and training that need to process a large number of small tensor operations.

[0078] It should be noted that the execution subject of the embodiment can be any electronic device capable of implementing operator fusion, such as a smartphone, a smart watch, a desktop computer, a laptop computer, a server, or any other type. In the embodiment, the target operator can be regarded as a basic computing unit or component in a neural network model. The neural network model can be applied to the fields of text processing, image processing, speech processing, video processing, etc. The neural network model can be a convolutional neural network model (CNN), a recurrent neural network model (RNN), a Transformer model, a generative adversarial network model (GAN), etc.

[0079] Taking the application of the neural network model to text processing as an example, the above target operator can be any computing unit used, input, or generated in tasks such as text classification, sentiment analysis, text generation, and machine translation, such as a fully connected layer operator, an activation function operator, and an attention mechanism operator. The tensors processed by these operators can be word embedding vectors, hidden state features, and attention weights of text. Based on the method of the above embodiment, multiple small-scale operators of the same type are fused into a single large-scale operator, the features of the input text are efficiently processed, the calculation result is obtained, the model performs inference based on the calculation result, and the input text is processed to obtain the corresponding output text or classification result.

[0080] Taking the application of a neural network model to image processing as an example, the target operator can be any calculation unit used, input, or generated in image classification, object detection, image segmentation, image generation, and the like, such as a convolution operator, a pooling operator, a normalization operator, and the like, and the tensors processed by these operators can be feature maps, convolution kernels, activation values, and the like of an image. Based on the method of the above embodiment, the convolution operations of multiple small-size feature maps are fused into a single batch convolution operation, the input image features are efficiently transformed, and a calculation result is obtained. The model performs reasoning based on the calculation result, processes the input image, and obtains a corresponding output result, such as a classification label, a bounding box, a segmentation mask, or a generated image.

[0081] Taking the application of a neural network model to speech processing as an example, the target operator can be any calculation unit used, input, or generated in speech recognition, speech synthesis, speech enhancement, speaker recognition, and the like, such as a one-dimensional convolution operator, a recurrent unit operator, an attention operator, and the like, and the tensors processed by these operators can be spectral features, time sequence features, encoded vectors, and the like of speech. Based on the method of the above embodiment, the small tensor operations corresponding to multiple short-time speech segments are fused into long-sequence large tensor operations, the input speech signal is efficiently processed, and a corresponding output text, synthesized speech, or enhanced speech signal is obtained.

[0082] In addition, the execution subject of the present embodiment can also be a GPU, a GPGPU, a TPU, an NPU, an FPGA, or the like, an artificial intelligence acceleration chip or special-purpose computing hardware. Such chips are usually optimized at the hardware level for a specific type of parallel operation and have a large number of parallel computing units. The present embodiment combines multiple small-scale operations into a single large-scale operation through operator fusion, which can make full use of the large number of parallel computing units on such chips and significantly improve hardware utilization. For example, on a GPU, the fused operator can start more CUDA threads, allowing more stream processors to work simultaneously; on a TPU, the fused operator can better utilize the computing power of the matrix multiplication unit; and on an NPU, the fused operator can improve the parallelism of the vector operation unit and the tensor operation unit. In this way, the present embodiment can optimize performance on various hardware platforms and accelerate the inference and training processes of neural networks.

[0083] Based on the above embodiment, a pseudo address mapping table is generated according to the tensors of the multiple target operators, including:

[0084] Collecting actual memory addresses of input tensors and output tensors of the multiple target operators;

[0085] Allocating continuous pseudo addresses to the input tensors and the output tensors;

[0086] Establishing a mapping relationship between the pseudo addresses and the actual memory addresses and recording the mapping relationship to the pseudo address mapping table.

[0087] During the execution of a neural network, each tensor is allocated a block of memory space to store its data, and the starting address of this block of memory space is the actual memory address of the tensor. For a plurality of target operators to be fused that have been determined, the address information of all input tensors and output tensors involved by them needs to be collected.

[0088] Specifically, the collection process can be completed by a graph compiler. During the compilation of a neural network, the graph compiler constructs a computation graph, and each operator node contains the input tensor reference and output tensor reference of the operator, which usually points to a tensor descriptor that records various attributes of the tensor, such as shape, data type, storage format, and memory address. By traversing the target operator nodes to be fused, the input tensor descriptors and output tensor descriptors of the target operator nodes can be obtained, and the actual memory addresses can be extracted from the descriptors.

[0089] In addition, considering that the subsequent allocation of continuous pseudo addresses needs to be divided according to the size of the tensor, the size information of each tensor can also be collected synchronously when the actual memory addresses are collected.

[0090] Among them, if the memory allocation of the tensor is statically determined at compile time, the address collection and mapping table generation can be completed in the compilation phase, and the generated mapping table can be embedded as constant data in the compiled executable code. If the memory allocation of the tensor is dynamic, the address information needs to be collected and the mapping table needs to be generated in real time before each execution of the fused operator.

[0091] After collecting the actual memory addresses and size information of all tensors, the next step is to allocate continuous pseudo addresses to these tensors. The allocation of pseudo addresses follows the principles of continuity and sequentiality, and the purpose is to construct a logically continuous address space, so that the fused operator can access multiple small tensors as if it were accessing a single large tensor.

[0092] For input tensors, the allocation of pseudo addresses can start from a base pseudo address, such as setting the base pseudo address to 0, indicating the start of the pseudo address space, and the pseudo address range allocated to the first input tensor is [0, input_tensor_size), that is, starting from the pseudo address 0 and ending at input_tensor_size, and the size of this range is exactly equal to the actual size of the first input tensor. The pseudo address range allocated to the second input tensor is [input_tensor_size, 2*input_tensor_size), that is, immediately after the pseudo address of the first input tensor, and similarly, the pseudo address range allocated to the third input tensor is [2*input_tensor_size, 3*input_tensor_size). In this way, the pseudo address range allocated to the i-th input tensor tensor_in_i is [i*input_tensor_size, (i+1)*input_tensor_size).

[0093] For output tensors, the allocation of pseudo addresses is similar to that of input tensors, but a separate pseudo address space can be used. The pseudo addresses of the output tensors can also be allocated starting from 0, and the pseudo address range of the first output tensor is [0, output_tensor_size), the pseudo address range of the second output tensor is [output_tensor_size, 2*output_tensor_size), and the pseudo address range of the i-th output tensor is [i*output_tensor_size, (i+1)*output_tensor_size).

[0094] In some embodiments, if the input tensor sizes of the target operators to be fused are not exactly the same, the allocation of pseudo addresses needs to be performed according to the actual sizes of the respective tensors. For example, assuming that there are three input tensors with sizes of 1000 bytes, 2000 bytes and 1500 bytes, the pseudo address allocation is as follows: the pseudo address range of the first input tensor is [0, 1000), the pseudo address range of the second input tensor is [1000, 3000), and the pseudo address range of the third input tensor is [3000, 4500).

[0095] In an implementation, in addition to recording the actual memory addresses in the pseudo address mapping table, the start and end positions of the pseudo addresses of each tensor are also recorded, and the tensor index corresponding to the pseudo address is determined by interval searching.

[0096] After the allocation of the pseudo addresses is completed, a mapping relationship between the pseudo addresses and the actual memory addresses is established, and the mapping relationship is recorded in a pseudo address mapping table. In order to quickly find and convert the addresses when the fusion operator is executed, the mapping table can be a one-dimensional array structure. For the input tensors, an array input_map is defined, where input_map[i] stores the actual memory starting address of the i-th input tensor. For the output tensors, an array output_map is defined, where output_map[i] stores the actual memory starting address of the i-th output tensor.

[0097] Based on any of the above embodiments, the continuous pseudo addresses are allocated for the input tensors and the output tensors, including:

[0098] According to the number of target operators, a corresponding number of continuous pseudo addresses are allocated for the input tensors, and the space size of each pseudo address is consistent with the size of the corresponding input tensor.

[0099] According to the number of target operators, a corresponding number of continuous pseudo addresses are allocated for the output tensors, and the space size of each pseudo address is consistent with the size of the corresponding output tensor.

[0100] In some embodiments, the process of allocating continuous pseudo addresses for the input tensors and the output tensors can be further refined as independent pseudo address allocation for the input tensors and the output tensors respectively. This separate allocation strategy makes the input data and the output data have clear boundaries in the pseudo address space, facilitating address management and access control within the fusion operator, and also providing flexibility for subsequent optimization. For example, on some hardware platforms, read-only input data can use a more aggressive prefetching strategy, while writable output data needs to consider the write-back cache consistency problem; by separating the pseudo address spaces of the input and the output, different optimization methods can be used for different access characteristics in the implementation of the fusion operator, without interfering with each other.

[0101] Specifically, when allocating continuous pseudo addresses for the input tensors, the number N of target operators needs to be determined first. Since each target operator has one or more input tensors, pseudo addresses need to be allocated for these input tensors. The allocation of pseudo addresses follows the principles of continuity and size matching. Continuity means that the pseudo addresses of the input tensors are logically adjacent and there is no gap, so that a continuous pseudo address range is formed as a whole. Size matching means that the pseudo address space allocated for each input tensor should be consistent with the actual data size of the input tensor, so that the pseudo address space can completely cover all input data, neither wasting addresses nor causing address shortage.

[0102] It is emphasized that the pseudo address space of the input tensor and the pseudo address space of the output tensor are independent of each other. That is, since the input tensor and the output tensor correspond to different mapping tables respectively, the pseudo address of the input tensor and the pseudo address of the output tensor can use the same numerical range without conflict. For example, the pseudo address 1000 of the input tensor and the pseudo address 1000 of the output tensor are two completely different addresses, the former is converted into an actual memory address of a certain input tensor through the pseudo address mapping table of the input tensor, and the latter is converted into an actual address of a certain output tensor through the pseudo address mapping table of the output tensor.

[0103] Through this refined pseudo address allocation strategy, the embodiment can accurately match the actual size of each tensor while maintaining address continuity, avoiding waste of address space and data access errors caused by insufficient address range. At the same time, the input tensor and the output tensor use independent pseudo address spaces, providing a clear interface and flexible optimization space for the implementation of the fusion operator.

[0104] Based on any of the above embodiments, a plurality of target operators to be fused are determined, comprising:

[0105] The computation graph is scanned by a graph compiler to identify a plurality of target operators satisfying a preset mode, and the preset mode includes that the plurality of target operators are operators of the same type and / or the input tensors and the output tensors of the plurality of target operators have the same shape.

[0106] Specifically, the graph compiler refers to a tool for parsing, analyzing, optimizing and code generation of the computation graph of the neural network. When scanning the computation graph, the graph compiler can identify the type information of each operator in the computation graph, the shape of the input tensor and the output tensor, the data dependency relationship between the operators, the parameter configuration of the operator and the topology position of the operator in the computation graph, and other metadata information.

[0107] It is considered that not all operators are suitable for fusion, and inappropriate operator fusion may lead to high implementation complexity of the fusion operator, insignificant fusion benefits or even performance degradation. Therefore, the target operators in the embodiment need to satisfy a preset mode, and the preset mode includes that the plurality of target operators are operators of the same type and / or the input tensors and the output tensors of the plurality of target operators have the same shape.

[0108] That is, the target operators to be fused can only satisfy the condition that they are operators of the same type, or only satisfy the condition that the input tensors and the output tensors have the same shape, or satisfy both conditions.

[0109] The plurality of target operators are of the same type, which ensures that the fused operator adopts unified calculation logic and unified code implementation, avoids the need to perform different operation branches in the fused operator according to different operator types, thereby simplifying the implementation of the fused operator and improving execution efficiency, and avoiding problems such as control flow complexity, branch prediction failure, and code bloating caused by different operator types.

[0110] The input tensors and the output tensors of the plurality of target operators have the same shape, that is, the sizes of the input tensors of each target operator in each dimension are completely consistent, and the sizes of the output tensors in each dimension are also completely consistent. For example, the shape of all input tensors is [batch_size, channels, height, width], and the shape of all output tensors is also [batch_size, channels, height, width], which can enable the fused operator to adopt a unified address step and a unified index calculation method when assigning the pseudo address, simplify the conversion logic of the pseudo address to the actual address, and avoid problems such as complex pseudo address mapping table structure, increased address conversion overhead, irregular memory access mode, and decreased cache utilization caused by different tensor shapes.

[0111] Based on any of the above embodiments, the computation graph is scanned by a graph compiler to identify a plurality of target operators satisfying a preset pattern, including:

[0112] The operators between which there is no data dependency relationship and the tensors of which do not overlap in memory are taken as candidate operators by scanning the computation graph through the graph compiler.

[0113] The candidate operators satisfying the preset pattern are taken as target operators.

[0114] Specifically, considering that the premise of operator fusion is that a plurality of operators can be executed in parallel or their execution order can be reorganized without affecting the correctness of the calculation result, if there is a data dependency relationship between the operators to be fused, such as the output of one operator being the input of another operator, or the tensors of a plurality of operators overlapping in memory, the write operation of one operator may affect the read operation of another operator, which may cause data competition, read-write conflict, or calculation result error during fusion execution, thereby causing the fused operator to fail to execute correctly.

[0115] Therefore, in the scanning of the computation, the operators between which there is no data dependency relationship and the tensors of which do not overlap in memory are identified, and these operators are taken as candidate operators. The candidate operators can be regarded as a set of operators that pass the basic safety check and have the feasibility of fusion, which satisfy the necessary condition of fusion but not necessarily the sufficient condition of fusion.

[0116] Operators without data dependency relationship between operators refer to multiple operators being independent of each other in the topological structure of the computation graph, and the output tensor of any operator is not the input tensor of other operators, that is, there is no direct or indirect data flow connection between these operators.

[0117] Operators with non-overlapping memory addresses of tensors of operators refer to the memory address spaces occupied by the input tensors and the output tensors of each operator being mutually exclusive, and there is no case of two tensors sharing the same memory region.

[0118] Since the candidate operators simultaneously have the two characteristics of no data dependency and non-overlapping memory addresses, the candidate operators can be safely executed in parallel or fused, and there is no case of one operator modifying the data being used by another operator, nor is there a case of calculation result error caused by change of execution order.

[0119] Further, considering that the candidate operators meet the safety requirement of fusion, it does not mean that the fusion of all candidate operators can bring performance benefits. If the types of the candidate operators are quite different, the input and output shapes are different, or the calculation complexities are quite different, taking the candidate operators as target operators may lead to too complex implementation of the fused operator, a large amount of conditional branch code needs to be written in the fused operator for different operator types and shapes, increasing the code maintenance difficulty, and the execution efficiency of the fused operator is even worse than that of executing each independent operator separately.

[0120] To this end, the embodiment further selects operators meeting a preset mode from the candidate operators as target operators, so as to ensure that the finally selected target operators not only meet the safety requirement of fusion, but also meet the efficiency requirement of fusion, that is, these operators have similar calculation characteristics and consistent data size, so that the fused operator can adopt a unified and efficient implementation manner, maximize the performance benefits brought by fusion, and minimize the implementation complexity and maintenance cost brought by fusion.

[0121] Based on any of the above embodiments, the pseudo address is converted into an actual memory address according to the pseudo address mapping table, and the data of the tensor is accessed according to the actual memory address, including:

[0122] The lookup index corresponding to the pseudo address is obtained by calculating the thread index or the tensor index;

[0123] According to the lookup index, the corresponding actual memory address is found and obtained in the pseudo address mapping table;

[0124] According to the actual memory address, the data of the corresponding tensor is accessed for reading or writing operation.

[0125] Specifically, the thread index refers to a unique identifier of each execution thread or computing unit in a parallel computing environment, used to distinguish different parallel execution entities, which can be calculated by a built-in variable or function provided by hardware. The tensor index refers to the position index of a certain data element in a logical large tensor processed by the fusion operator, used to locate the relative position of the element in the entire fusion tensor, which can be directly mapped by the thread index or calculated and converted by multi-dimensional index calculation.

[0126] In addition, the lookup index refers to an index value used to locate a specific mapping table entry in the pseudo address mapping table, which indicates the original tensor corresponding to the currently accessed pseudo address. Since the thread index directly corresponds to a certain position in the pseudo address space in the fusion operator, the range of the tensor to which the position belongs can be calculated by performing division or shift operation on the thread index, so that the corresponding lookup index can be obtained according to the thread index.

[0127] Since the tensor index also represents the position of the data element in the logical fusion tensor, the original tensor corresponding to the tensor index can be determined by similar division operation, so that the corresponding lookup index can be obtained according to the tensor index.

[0128] After determining the lookup index, the corresponding actual memory address can be obtained in the pseudo address mapping table according to the lookup index. After obtaining the actual memory address, the offset in the tensor can be calculated, and the base address and the offset are added to obtain the final physical access address, so that the data of the corresponding tensor can be accessed for reading or writing operation.

[0129] Based on any of the above embodiments, the operator fusion is performed on a plurality of target operators to obtain a fusion operator, including:

[0130] The pseudo address mapping table is received as an input parameter, and the calculation logic of the plurality of target operators is integrated into a single calculation flow to obtain the fusion operator.

[0131] In the calculation flow, the access addresses of the input tensors and the output tensors of the plurality of target operators are subjected to pseudo address processing, so that the fusion operator logically accesses continuous pseudo addresses.

[0132] Specifically, considering that the input tensors and the output tensors of the plurality of target operators are stored in a scattered manner in actual memory and each has a different actual memory address, and the fusion operator needs to process these scattered small tensors as a logically continuous large tensor, the pseudo address mapping table needs to be provided as an input parameter when the operator fusion is performed, so that the actual memory address of each tensor can be obtained by querying the pseudo address mapping table dynamically at runtime when the calculation logic of the plurality of target operators is integrated into a single calculation flow, the correct conversion from the continuous pseudo address to the scattered actual memory address is implemented, and the data of each tensor can be correctly accessed by the fusion operator while the code logic is kept simple and the execution efficiency is maintained.

[0133] The pseudo address mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of each tensor.

[0134] Based on any of the above embodiments, the pseudo address mapping table includes an input mapping table and an output mapping table, the input mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the input tensor, and the output mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the output tensor.

[0135] Specifically, if the input tensors and the output tensors are managed by the same mapping table, the number, size, and access mode of the input tensors and the output tensors are usually different, and the mixed storage in the same table increases the lookup complexity, an additional identification bit is needed to distinguish whether the current access is the input or the output, and in the concurrent access scenario, address calculation conflicts are likely to occur, which causes the fusion operator to frequently judge the tensor type corresponding to the current pseudo address during execution, increases the probability of branch prediction failure, reduces the instruction pipeline efficiency, and also makes the index calculation of the mapping table complex, cannot fully utilize simple division or bit operation for fast address conversion, and affects the overall execution performance.

[0136] Therefore, in the embodiment, the mapping relationship of the input tensors and the mapping relationship of the output tensors are stored in independent mapping tables to avoid the above address lookup conflict and the increase in calculation complexity, so that the fusion operator only needs to query the input mapping table when accessing the input data and only needs to query the output mapping table when accessing the output data, and the two do not interfere with each other, the lookup logic is clear and simple.

[0137] That is, the mapping relationship between the actual memory address of the input tensor and the pseudo address is stored in the input mapping table, and the mapping relationship between the actual memory address of the output tensor and the pseudo address is stored in the output tensor mapping table, so that when the fused operator is executed, the corresponding mapping table can be directly selected for query according to the current operation type, without additional type judgment overhead.

[0138] In addition, the input mapping table and the output mapping table can optimize their data layout and cache strategy, respectively, for example, placing frequently accessed input mapping tables in faster storage areas, and placing output mapping tables in storage areas suitable for write operations, thereby further improving memory access efficiency.

[0139] wherein, Figure 3 is the operator fusion schematic diagram provided by the present application, as Figure 3 shown, first, the input tensor (input_tensor) is sent into the splitting operator (mem_op) for splitting to obtain a plurality of smaller sub-input tensors (in_tensor 0, in_tensor 1,..., in_tensor n).

[0140] Then, these sub-input tensors together with the metadata describing their organization are passed as input to the fused operator (fused op). The fused operator uses the metadata to process all sub-input tensors in parallel or serially in a single kernel execution, and generates a corresponding plurality of sub-output tensors (out_tensor 0, out_tensor 1,..., out_tensor n). Among them, the input mapping table (input_map) is used to inform the fused operator how to locate and access each sub-input tensor in the original, possibly continuous memory block. Among them, the metadata includes the input mapping table (input_map), the input tensor size (input_tensor_size) and the input tensor number (input_tensor_num).

[0141] Next, all sub-output tensors are passed to the merging operator (mem_op) for memory merging operation. According to the metadata such as the output mapping table (output_map), the output tensor size (output_tensor_size) and the output tensor number (output_tensor_num), these scattered sub-output tensors are recombined and spliced into a single, complete output tensor (output_tensor), thereby completing the entire computing process.

[0142] The fused operator execution device provided by the present application is described below. The fused operator execution device described below can be correspondingly referred to the fused operator execution method described above.

[0143] based on any of the above embodiments, Figure 4 is a structural schematic diagram of a fusion operator execution device provided by the present application, as Figure 4 shown, the device comprises:

[0144] The determination unit 410 is configured to determine a plurality of target operators to be fused.

[0145] The generation unit 420 is configured to generate a pseudo address mapping table according to the tensors of the plurality of target operators, the pseudo address mapping table being used to record the mapping relationship between the actual memory address of the tensor and the pseudo address.

[0146] The fusion unit 430 is configured to perform operator fusion on the plurality of target operators to obtain a fusion operator, and the memory address of the tensor of the fusion operator is a continuous pseudo address.

[0147] The execution unit 440 is configured to execute the fusion operator, and in the execution process, the pseudo address is converted into an actual memory address according to the pseudo address mapping table, and the data of the tensor is accessed according to the actual memory address.

[0148] Figure 5 is a structural schematic diagram of an electronic device provided by the present application, as Figure 5 shown, the electronic device can include a processor 510, a communications interface 520, a memory 530 and a communications bus 540, wherein the processor 510, the communications interface 520 and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke the logical instructions in the memory 530 to execute the fusion operator execution method.

[0149] In addition, the logical instructions in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the related art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0150] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program being stored in a non-transitory computer readable storage medium, and the computer program, when executed by a processor, enables a computer to perform the fusion operator execution method provided by the above-mentioned methods.

[0151] In yet another aspect, the present application also provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program, when executed by a processor, enables a computer to perform the fusion operator execution method provided by the above-mentioned methods.

[0152] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one place, or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the present embodiment according to actual needs. Those skilled in the art can understand and implement it without creative labor.

[0153] From the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus necessary general hardware platforms, and of course, can also be realized by hardware. Based on such understanding, the above technical solutions, essentially or in terms of related art, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes a number of instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0154] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for executing a fusion operator, characterized in that, include: Identify multiple target operators to be fused; Based on tensors of multiple target operators, a pseudo-address mapping table is generated. The pseudo-address mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the tensor. The multiple target operators are fused to obtain a fused operator, and the memory address of the tensor of the fused operator is a contiguous pseudo address. The fusion operator is executed, and during the execution process, the pseudo address is converted into the actual memory address according to the pseudo address mapping table, and the tensor data is accessed according to the actual memory address.

2. The fusion operator execution method according to claim 1, characterized in that, The step of generating a pseudo-address mapping table based on tensors of multiple target operators includes: Collect the actual memory addresses of the input and output tensors of the multiple target operators; Assign contiguous pseudo-addresses to the input and output tensors; Establish a mapping relationship between the pseudo address and the actual memory address, and record it in the pseudo address mapping table.

3. The fusion operator execution method according to claim 2, characterized in that, Assigning contiguous pseudo-addresses to the input and output tensors includes: Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the input tensor, and the size of each pseudo-address is consistent with the size of the corresponding input tensor. Based on the number of target operators, a corresponding number of consecutive pseudo-addresses are allocated to the output tensor, and the size of each pseudo-address is consistent with the size of the corresponding output tensor.

4. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The determination of multiple target operators to be fused includes: The graph compiler scans the computation graph to identify multiple target operators that satisfy a preset pattern. The preset pattern includes that the multiple target operators are operators of the same type and / or that the input tensors and output tensors of the multiple target operators have the same shape.

5. The fusion operator execution method according to claim 4, characterized in that, The step of scanning the computation graph using a graph compiler to identify the plurality of target operators that satisfy a preset pattern includes: The computation graph is scanned by the graph compiler, and operators that have no data dependencies between operators and whose tensors do not overlap in memory are selected as candidate operators. The candidate operator that satisfies the preset mode is taken as the target operator.

6. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The step of converting the pseudo-address to the actual memory address according to the pseudo-address mapping table, and accessing the tensor data according to the actual memory address, includes: The lookup index corresponding to the pseudo-address is obtained by calculating the thread index or tensor index; Based on the lookup index, the corresponding actual memory address is searched and obtained from the pseudo-address mapping table; Based on the actual memory address, access the data of the corresponding tensor to perform read or write operations.

7. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The step of fusing the multiple target operators to obtain a fusion operator includes: The pseudo-address mapping table is received as an input parameter, and the calculation logic of the multiple target operators is integrated into a single calculation process to obtain the fusion operator; In the calculation process, the access addresses of the input tensors and output tensors of the multiple target operators are pseudo-addressed, so that the fusion operator logically accesses consecutive pseudo-addresses.

8. The fusion operator execution method according to any one of claims 1 to 3, characterized in that, The pseudo-address mapping table includes an input mapping table and an output mapping table. The input mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the input tensor, and the output mapping table is used to record the mapping relationship between the actual memory address and the pseudo address of the output tensor.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the fusion operator execution method as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the fusion operator execution method as described in any one of claims 1 to 8.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the fusion operator execution method as described in any one of claims 1 to 8.

Images