Operator fusion method and device, electronic equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence chips, and provides an operator fusion method and device, electronic equipment and a storage medium, and the method comprises the steps: determining an input tensor, an output tensor and a coverage range of a to-be-fused operator, and the to-be-fused operator comprises a plurality of view operators; on the basis of the metadata of the input tensor, the metadata of the output tensor and the logic relation between the view operators in the coverage range, determining the mapping relation between the dimensions of the input tensor and the output tensor; and performing data filling on the output tensor based on the mapping relationship and each data in the input tensor, and taking the filled output tensor as a fusion result of the operator to be fused. According to the method, multiple continuous view operators are efficiently and quickly fused, so that the problem of performance reduction caused by frequent data handling in the model is effectively solved.

Description

Technical Field

[0001] The present invention relates to the technical field of artificial intelligence chips, and particularly to an operator fusion method, device, electronic device, and storage medium. Background Art

[0002] View class operators play a crucial role in deep learning frameworks and large model applications. They can assist in computing optimization and ensure the correct flow and conversion of data between different network layers. However, in actual large model applications, some View class operators (such as transpose, permute, etc.) can cause tensors to be discontinuous in memory. Once such discontinuity occurs, deep learning frameworks (such as PyTorch) will automatically call the contiguous operation to perform data movement to restore the continuity of tensors in memory.

[0003] However, when these View class operators are frequently used in a model, the contiguous operation will be frequently triggered, resulting in a large amount of data movement. And the data movement of tensors is an extremely time-consuming operation. Especially in large-scale computing tasks and when the tensor dimensions are high, it is very likely to become a performance bottleneck in the model inference or training process. Summary of the Invention

[0004] The present invention provides an operator fusion method, device, electronic device, and storage medium to solve the problem of performance degradation caused by frequent data movement when using View class operators in a model.

[0005] The present invention provides an operator fusion method, including: Determine the input tensor, output tensor, and coverage range of the operator to be fused, where the operator to be fused includes multiple view class operators; Based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationship between the view class operators within the coverage range, determine the mapping relationship between the dimensions of the input tensor and the output tensor; Based on the mapping relationship and the data in the input tensor, fill the output tensor with data, and use the filled output tensor as the fusion result of the operator to be fused.

[0006] According to an operator fusion method provided by the present invention, the coverage range is obtained based on user input or automatic detection by the framework. The step of determining the logical relationship between the view class operators within the coverage range includes: Based on the connection relationship between the view class operators within the coverage range, the operator attributes of the view class operators, and the tensor attributes of the tensors connected between the view class operators, determine the logical relationship between the view class operators.

[0007] An operator fusion method provided by the present invention, the data filling of the output tensor based on the mapping relationship and each data in the input tensor includes: Traverse the target indices on each dimension of the output tensor; Based on the mapping relationship, determine the source index corresponding to the currently traversed target index in the input tensor; Based on the source index, read the corresponding data from the input tensor, and fill the data into the output tensor according to the currently traversed target index.

[0008] An operator fusion method provided by the present invention, the reading of the corresponding data from the input tensor based on the source index and the filling of the data into the output tensor according to the currently traversed target index include: Based on the source index, read the corresponding data from the input tensor into a temporary storage area, and the temporary storage area is any one of a register, a shared memory, and a cache; Based on the currently traversed target index, write the data in the temporary storage area into the output tensor.

[0009] An operator fusion method provided by the present invention, the data filling of the output tensor based on the mapping relationship and each data in the input tensor includes: Based on the mapping relationship, determine the corresponding dimension of any dimension of the output tensor in the input tensor; When the target indices of any dimension are continuous and in the same order as the source indices of the corresponding dimension, use a data block reading instruction to read each data on the corresponding dimension of the input tensor, and fill each data into the any dimension of the output tensor.

[0010] An operator fusion method provided by the present invention, the input tensor includes one or more, and the output tensor includes one or more.

[0011] An operator fusion method provided by the present invention, the metadata includes the shape, dimension, and stride of the tensor.

[0012] The present invention also provides an operator fusion device, including: A range determination unit for determining the input tensor, output tensor, and coverage range of the operator to be fused, and the operator to be fused includes multiple view class operators; A relationship determination unit, configured to determine a mapping relationship between each dimension of the input tensor and the output tensor based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationship between view class operators within the coverage range; An operator fusion unit, configured to perform data filling on the output tensor based on the mapping relationship and each data in the input tensor, and use the filled output tensor as the fusion result of the operator to be fused.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and running on the processor. When the processor executes the computer program, the operator fusion method described in any one of the above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the operator fusion method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, the operator fusion method described in any one of the above is implemented.

[0016] The operator fusion method, device, electronic device, and storage medium provided by the present invention can accurately deduce the mapping relationship between each dimension of the input tensor and the output tensor of the operator to be fused by based on the metadata of the input tensor and the output tensor, as well as the logical relationship between each view class operator. Therefore, according to this mapping relationship, each data in the input tensor can be accurately filled into the corresponding position of the output tensor, efficiently realizing operator fusion, that is, integrating and optimizing multiple view class operators that originally need to be continuously executed into one fused operator. Through this fusion method, during the operation of the model, the number of executions of view transformation operations is significantly reduced, and the number of data transfers is also reduced accordingly, thereby effectively reducing the additional overhead caused by data transfer and solving the problem of performance degradation caused by frequent data transfer in the model. Description of the Drawings

[0017] In order to more clearly illustrate the technical solutions in the present invention or related technologies, the following will briefly introduce the drawings required for use in the description of the embodiments or related technologies. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0018] Figure 1 is a structural schematic diagram of a graphics processor provided by the present invention; Figure 2 is a flowchart of the operator fusion method provided by the present invention; Figure 3 It is a schematic connection diagram of each view class operator provided by the present invention; Figure 4 It is a schematic diagram of the specific structure and coverage of the fusion operator provided by the present invention; Figure 5 It is a schematic diagram of the mapping relationship between the input tensor and the output tensor provided by the present invention; Figure 6 It is a schematic structural diagram of the operator fusion device provided by the present invention; Figure 7 It is a schematic structural diagram of the electronic device provided by the present invention. Detailed implementation manners

[0019] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present invention without making creative efforts shall fall within the protection scope of the present invention.

[0020] During the large model training and inference processes, parallel strategies such as data parallelism, tensor parallelism, and sequence parallelism are often adopted to improve efficiency, and these parallel strategies all require splitting the tensors. To ensure that the tensors after splitting can adapt to the interface requirements of subsequent different operators, tensor transformation operations are frequently introduced in the model. Such tensor transformations are also called View class transformations, which are mainly implemented through View class operators such as reshape (reshaping), transpose (transposing), view (deforming), permute (permuting), repeat (repeating), split (splitting), etc. The core function of these View class operators is to change the shape, arrangement order, or memory layout of the tensors to adapt to different computing requirements.

[0021] In large models, View class operations mainly involve the transformation and adjustment of tensor shapes, aiming to optimize the computing process, strengthen memory management, and ensure smooth adaptation of data between different network layers. The following are several common View class operators and their application scenarios: The reshape operator: It is used to change the shape of a tensor without changing the memory layout of the data. For example, reshaping a tensor from shape (2, 6) to (3, 4). In an AI (Artificial Intelligence) model, there are often differences in the input and output shapes between different layers. Therefore, the reshape operator is needed to adjust the data shape so that it can be passed to the next layer smoothly. For example, between a fully connected layer and a convolutional layer, the tensor output by the convolutional layer usually needs to be converted to a one-dimensional tensor through the reshape operator to meet the input requirements of the fully connected layer.

[0022] The transpose operator: It is used to swap the dimensions of a tensor. For example, transposing a tensor with shape (2, 3) to (3, 2). This operation is widely used in scenarios such as matrix multiplication and convolutional operations. Especially when dealing with input and output of different dimensions, the transpose operation can ensure that the data is processed in the correct dimension order, thus avoiding calculation errors caused by dimension mismatches.

[0023] The view operator: It creates a new view by changing the shape of a tensor, usually without involving memory copying. For example, converting a tensor with shape (2, 6) to (12). In a deep learning model, the shape of the tensor needs to be adjusted dynamically according to the task requirements, but users often hope to avoid additional memory overhead. The view operation flexibly adjusts the tensor shape by sharing the same piece of memory without copying the data, thus improving the calculation efficiency and reducing memory consumption.

[0024] The permute operator: It is used to rearrange the dimensions of a tensor. Some operations have strict requirements on the dimension order of the tensor. For example, in image processing, it is often necessary to adjust the position of the channel dimension. For example, adjusting a tensor with shape (C, H, W) to (H, W, C). Here, C represents the channel dimension of the tensor, H represents the height dimension, and W represents the width dimension. This operation helps to optimize the performance of matrix operations because different hardware platforms (such as GPUs) have different requirements for data layout, and the permute operation can ensure that the data layout meets the calculation requirements.

[0025] The repeat operator: It is used to repeat and expand a tensor along a specified dimension to generate a new tensor with a larger dimension. For example, in an image processing task, the repeat operator can be used to repeat the image feature map along the channel dimension. For example, repeating a feature map with shape (C, H, W) k times along the channel dimension to generate a new feature map with shape (C k, H, W) to increase the diversity of training samples.

[0026] The split operator: It is used to split a tensor into multiple sub-tensors along a specified dimension, which can be achieved by specifying the number of splits or the size of each sub-tensor. For example, in a convolutional neural network, the feature map can be split into multiple sub-feature maps along the channel dimension by the split operator and then fed into different branch networks for processing. For instance, a feature map with the shape (C, H, W) is split into k sub-feature maps along the channel dimension, and the shape of each sub-feature map is (C / k, H, W), which is used to achieve multi-branch feature extraction.

[0027] By reasonably using the above View class operators, large models can process data more efficiently and improve the overall computing performance.

[0028] As a bridge connecting the theory of large models and practical applications, deep learning frameworks provide developers with a series of convenient APIs (Application Programming Interfaces), modules, and tools to easily implement model construction, training, optimization, and deployment. Currently, common deep learning frameworks include TensorFlow, PyTorch, Keras, Caffe, etc. Various data processed by large models often exist in the form of multi-dimensional arrays, and in deep learning frameworks, multi-dimensional array data is usually modeled as Tensor (tensor).

[0029] In deep learning frameworks, some operations (such as the view operation in PyTorch) require the tensor to be contiguous in memory, that is, the data elements in the tensor are stored sequentially in memory, otherwise an error will be reported. If the tensor is not contiguous, PyTorch will automatically call the contiguous operation to ensure the continuity of the tensor in memory. Specifically, PyTorch will apply for a new memory space and copy the data in the tensor to the new memory space according to the logical data layout.

[0030] However, in actual applications, some View class operators (such as transpose, permute, etc.) will cause the tensor to be non-contiguous in memory. When such View class operators are frequently used in the model, the contiguous operation will be frequently called, resulting in a large amount of data movement. And the data movement of tensors is a very time-consuming operation, especially in the case of large-scale computing and high tensor dimensions. Moreover, frequently calling View class operators will increase the complexity of the computational graph, especially in each backpropagation step during model training. In addition, each time a View class operator is called to perform a view transformation on a tensor, it may be necessary to recalculate the size and memory layout of the tensor in the background, thus affecting the performance of model training or inference.

[0031] In response to this, the present invention proposes an operator fusion method for View - type operators in large models. By efficiently and quickly fusing multiple consecutive View - type operators, the problem of performance degradation caused by frequent data transfer in the model is solved, thereby overcoming the above - mentioned defects. The technical solutions provided by the present invention will be introduced in detail below.

[0032] It should be noted that the execution subject of the operator fusion method provided by the present invention can be an artificial intelligence chip, such as a GPU (Graphics Processing Unit), a GPGPU (General - purpose computing on Graphics Processing Units), a TPU (Tensor Processing Unit), etc. Taking the GPU as an example, the structure of the execution subject of the present invention will be briefly introduced below.

[0033] Figure 1 is a schematic structural diagram of the graphics processing unit provided by the present invention, as Figure 1 shown, the graphics processing unit 100 at least includes multiple Streaming Processor Clusters (SPC) 101 and a global memory 102. Among them, each Streaming Processor Cluster 101 includes multiple computing units 103, and each computing unit 103 includes multiple cores. In addition to the above - mentioned structure, the graphics processing unit 100 in the present invention may also include other structures, and the present invention does not make specific limitations thereto. It should be understood that the GPU has a large number of computing cores (i.e., the cores within the computing units), is good at parallel computing, and is extremely efficient in processing large - scale data computing tasks. In computing - intensive fields such as deep learning, the GPU has become the mainstream computing hardware. Many deep - learning frameworks are optimized for the GPU, enabling operations such as operator fusion to be efficiently executed on the GPU.

[0034] It can be understood that in the related art, when the model frequently uses View - type operators, the contiguous operation will be frequently called. In this operation, the GPU core needs to read non - contiguous data element - by - element from the original memory and write it into the new memory. This process involves the reading and writing of the global memory, generating a large amount of data transfer, thereby affecting the performance of model training and inference. In response to this, the present invention can effectively reduce data transfer by efficiently and quickly fusing multiple consecutive View - type operators, thereby reducing the number of accesses to the global memory and improving the overall performance.

[0035] Figure 2 is a schematic flow diagram of the operator fusion method provided by the present invention, as Figure 2 shown, the method includes: Step 210, determine the input tensor, output tensor, and coverage range of the operator to be fused, where the operator to be fused includes multiple view-based operators.

[0036] Specifically, the operator to be fused refers to a continuous series of View-based operators that need to be fused together. The input tensor of the operator to be fused refers to the data source on which the operator to be fused operates. It is a multi-dimensional array that contains the data required for operator processing. The output tensor of the operator to be fused refers to the result obtained after the operator to be fused processes the input tensor, and its shape may change according to the operations of the operator. The coverage range of the operator to be fused refers to the operation range involved by multiple View-based operators participating in the fusion. By determining the coverage range, it can be clear which operators need to be fused, as well as the operation sequence and logical relationship between them.

[0037] It can be understood that the operator fusion method provided by the present invention can be applied to the fields of image processing, text processing, speech processing, etc. According to different application scenarios, the input tensor and output tensor have different physical meanings.

[0038] Exemplarily, in the field of image processing, the input tensor can be a tensor representing an image, with a shape of [B, C, H, W], where B is the batch dimension, which represents the batch size, C is the channel dimension, which represents the number of channels, and H and W are the height dimension and width dimension respectively, representing the height and width of the image. For example, an RGB image can be represented as an input tensor with a shape of [1, 3, 224, 224], where the batch size is 1, 3 represents the three RGB channels, and 224 and 224 represent the height and width of the image respectively. The shape of the output tensor depends on the processing operation on the image. For example, if the reshape operator and permute operator are used to rearrange the number of channels and spatial dimensions of the image, a tensor with a shape of [1, 224 224, 3] is obtained, and this tensor is the output tensor.

[0039] For another example, in the field of text processing, the input tensor can be a tensor representing a text sequence. For the sentence "I love natural language processing", it can be encoded as an input tensor with a shape of [1, 5, 768], where the batch size is 1, 5 represents the length of the sentence (assuming the sentence is segmented into 5 words), and 768 represents the vector dimension of each word. The shape of the output tensor depends on the text processing task. For example, if the permute operator is used to transform the dimensions of the text sequence from [1, 5, 768] to [5, 1, 768], this tensor can be used as the output tensor.

[0040] In one embodiment, the input tensor, output tensor, and coverage range of the operator to be fused can be automatically detected through a deep learning framework. Specifically, the internal graph structure of the deep learning framework can be used to parse and traverse the computational graph to identify and detect the View class operators. When it is detected that all the operators from a certain node to another node in the computational graph are View class operators, it can be determined that the combination of these View class operators corresponding to these nodes constitutes an operator to be fused, and its coverage range is all the View class operators between the starting node and the ending node. Further, starting from the detected starting node, trace back along the computational graph in the reverse direction to find the first non-View class operator (such as convolution, fully connected, etc. operators), and the output of this operator is the input tensor of the operator to be fused. If the starting node itself is the input node of the model, the tensor of this node is the input tensor. Similarly, starting from the ending node of the detected View class operator combination, move forward along the flow of the computational graph to find the first non-View class operator or the output node of the model, and the input of this operator is the output tensor of the operator to be fused. If the ending node itself is the output node of the model, the tensor of this node is the output tensor.

[0041] In another embodiment, the input tensor, output tensor, and coverage range of the operator to be fused can also be obtained by user input, that is, when the user calls the operator in the upper-level framework, the user can directly specify which View class operators the operator to be fused includes, the connection relationship between the operators, and the corresponding input tensor, output tensor, etc.

[0042] Figure 3 is a connection schematic diagram of each view class operator provided by the present invention. As Figure 3 shown, tensor 1 is processed by the permutation operator to obtain tensor 2. Reshaping operations are respectively performed on tensor 2 and tensor 3 to obtain the corresponding tensors 4 and 5. Tensors 4 and 5 are concatenated through the concatenation operator to obtain tensor 6. In Figure 3In the connection schematic diagram shown, since the permutation operator, reshape operator, and concatenation operator are multiple consecutive View class operators, they constitute the operator to be fused. The input tensors of this operator to be fused include Tensor 1 and Tensor 3, and the output tensor is Tensor 6. The connection path between the input tensor and the output tensor is the coverage range of the operator to be fused.

[0043] Step 220: Based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationship between each view class operator within the coverage range, determine the mapping relationship between each dimension of the input tensor and the output tensor.

[0044] It should be noted that the metadata of a tensor refers to the information describing the attributes or structure of the tensor itself, rather than the actual data stored in the tensor. The metadata of a tensor includes, but is not limited to, the shape, dimension, stride, etc. of the tensor. Among them, the shape is used to describe the dimension structure and dimension size of the tensor, and the dimension is used to describe the number of dimensions of the tensor. For example, the shape of a tensor can be [3, 4], which means the tensor is a matrix with 3 rows and 4 columns, and the dimension of this tensor is 2, indicating that it is a two-dimensional tensor. The stride refers to the number of bytes to skip when accessing tensor elements in memory when moving one unit along each dimension. For example, (4, 1) means that the adjacent row elements in memory are separated by 4 bytes, and the adjacent column elements are separated by 1 byte. It should be understood that in a deep learning framework, the metadata of a tensor can be obtained by calling corresponding interface functions (such as shape, stride).

[0045] The logical relationship between each view class operator within the coverage range refers to the dependency relationship and operation order between multiple View class operators. For example, as Figure 3 shown, the logical relationship between the permutation operator and the reshape operator is that the output of the permutation operator is the input of the reshape operator, and the permutation operator must be executed before the reshape operator. The logical relationships between other View class operators are similar and will not be elaborated here.

[0046] Specifically, based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationships among the View class operators within the coverage range, the mapping relationships between the dimensions of the input tensor and the output tensor can be determined. Here, the mapping relationship refers to the correspondence between each dimension of the input tensor and the dimensions of the output tensor. For example, assume the shape of the input tensor is [B, C, H, W], where dimension 0 is the batch dimension, dimension 1 is the channel dimension, dimension 2 is the height dimension, and dimension 3 is the width dimension; the shape of the output tensor is [B, H, W, C], where dimension 0 is the batch dimension, dimension 1 becomes the height dimension, dimension 2 becomes the width dimension, and dimension 3 becomes the channel dimension. Then the mapping relationships between their respective dimensions are: input dimension 0 corresponds to output dimension 0 (the batch dimension remains unchanged), input dimension 1 corresponds to output dimension 3 (the channel dimension becomes the last dimension), input dimension 2 corresponds to output dimension 1 (the height dimension becomes the second dimension), and input dimension 3 corresponds to output dimension 2 (the width dimension becomes the third dimension).

[0047] It can be understood that to determine the mapping relationships between the dimensions of the input tensor and the output tensor, it can be specifically achieved through the following steps: Based on the shape, stride, and other metadata of the obtained input tensor and output tensor, as well as the logical relationships among the View class operators within the coverage range, conduct derivation and analysis. During the analysis process, gradually derive according to the operation functions and operation sequences of each View class operator, and finally obtain the dimension mapping relationship from the input tensor to the output tensor, and a dimension index mapping table can be used to describe the correspondence between the input and output dimensions.

[0048] As Figure 3 shown, assume the shape of tensor 1 is [n, h, w], and the shape of tensor 3 is [n, 2 w, h], where n represents the batch dimension, h represents the height dimension, and w represents the width dimension. Represent tensors 1 to 6 as tensor1 to tensor⑥ respectively, and their transformation relationships can be described as follows: tensor2 = tensor1.permute(0, 2, 1); tensor4 = tensor2.reshape(n w, h); tensor5 = tensor3.reshape(2 n w, h); tensor6 = torch.cat((tensor4, tensor5), dim = 0); Since the shape of the input tensor tensor1 is [n, h, w], and tensor2 is obtained by rearranging the dimensions of tensor1 using the permute operator. The above permute(0, 2, 1) means swapping the second dimension (dim = 1) and the third dimension (dim = 2) of the tensor. Therefore, based on the shape of tensor1 and the operation function of the permute operator, it can be determined that the shape of tensor2 is [n, w, h]. Tensor4 is obtained by reshaping the shape of tensor2 using the reshape operator. The above reshape(n w, h) means flattening the tensor into two dimensions, where the first dimension is n w and the second dimension is h. Similarly, based on the shape of tensor2 and the operation function of the reshape operator, it can be determined that the shape of tensor4 is [n w, h].

[0049] Since the shape of the input tensor tensor3 is [n, 2 w, h], and tensor5 is obtained by reshaping the shape of tensor3 using the reshape operator. The above reshape(2 n w, h) means flattening the tensor into two dimensions, where the first dimension is 2 n w and the second dimension is h. Based on the shape of tensor3 and the operation function of the reshape operator, it can be determined that the shape of tensor5 is [2 n w, h]. Finally, by using the cat operator to concatenate tensor4 and tensor5 along the first dimension (dim = 0), based on the shapes of tensor4 and tensor5 and the operation function of the cat operator, it can be determined that the shape of tensor6 is [n w + 2 n w, h] = [3 n w, h].

[0050] According to the above step-by-step analysis and derivation process, the mapping relationship between the dimensions of the input tensor and the output tensor can be determined. That is, the first dimension of the output tensor is divided into two parts. The first part n w is determined by the batch dimension (n) and the width dimension (w) of the input tensor tensor1, and the second part 2 n w is determined by the batch dimension (n) and the width dimension (2 w) of the input tensor tensor3, while the second dimension of the output tensor is directly determined by the height dimensions of tensor1 and tensor3.

[0051] In addition, after determining the mapping relationship between the dimensions of the input tensor and the output tensor, the mapping relationship can be verified according to the metadata of the input tensor and the metadata of the output tensor to ensure that the mapping relationship conforms to the logical definition of the operator.

[0052] Step 230, based on the mapping relationship and the data in the input tensor, perform data filling on the output tensor, and use the filled output tensor as the fusion result of the operator to be fused.

[0053] Specifically, after determining the mapping relationship between the dimensions of the input tensor and the output tensor, each data in the input tensor can be mapped and filled into the corresponding position of the output tensor according to this mapping relationship. Here, each data in the input tensor refers to the specific numerical values stored in the input tensor. A tensor is a multi-dimensional array, and each of its elements is a scalar value (such as float32, int64, etc.).

[0054] Specifically, the mapping relationship defines the corresponding relationship between the dimensions of the input tensor and the output tensor. According to the mapping relationship, the following steps can be used to fill each data in the input tensor into the output tensor: First, traverse the indices of each data point on each dimension in the output tensor. Through the mapping relationship, determine the corresponding index of the currently traversed index in the input tensor, and then read the corresponding data from the input tensor according to this corresponding index, and store this data in the corresponding position of the output tensor. After the traversal is completed, the filled output tensor can be obtained. Here, the filled output tensor refers to the tensor obtained by rearranging the data in the input tensor according to the mapping relationship. Its shape may be different from that of the input tensor, the data content is the same as that of the input tensor, but the arrangement order may be different. It should be understood that performing data filling on the output tensor based on the mapping relationship and the data in the input tensor can be executed by the computing core of the GPU. Specifically, the computing core of the GPU will read each data from the global memory address corresponding to the input tensor; then, according to the previously determined mapping relationship, the computing core writes the read data of the input tensor into the global memory corresponding to the output tensor.

[0055] It can be understood that after obtaining the filled output tensor, this output tensor can be used as the final fusion result of the operator to be fused. Here, the fusion result is the output tensor directly obtained after combining the operations of multiple View-like operators into one operator. Logically, it is equivalent to the result of the original computational graph, but the computational efficiency is higher. For example,Figure 3 As shown, according to the mapping relationship, the data in the input tensors (i.e., Tensor 1 and Tensor 3) is directly filled into the output tensor (i.e., Tensor 6). The fused operator directly outputs the result without separately executing operators such as permutation operators, reshaping operators, and splicing operators, etc. The data content of the filled output tensor is consistent with the result of the original computational graph, but the computational process is more efficient.

[0056] The method provided by the embodiments of the present invention can accurately deduce the mapping relationship between the dimensions of the input tensor and the output tensor of the operator to be fused based on the metadata of the input tensor and the output tensor in time, and the logical relationship between various view class operators. Thus, according to this mapping relationship, each data in the input tensor can be accurately filled into the corresponding position of the output tensor, efficiently realizing operator fusion, that is, integrating and optimizing multiple view class operators that originally need to be continuously executed into one fused operator. Through this fusion method, during the operation of the model, the number of executions of view transformation operations is significantly reduced, and the number of data transfers also decreases accordingly, effectively reducing the additional overhead caused by data transfer and solving the problem of performance degradation caused by frequent data transfer in the model.

[0057] Based on the above embodiments, the coverage range is automatically detected based on user input or the framework.

[0058] Specifically, the coverage range of the operator to be fused can be specified through user input or automatically detected by the deep learning framework. The embodiments of the present invention do not make specific limitations on this. It should be noted that the specific implementation method for determining the coverage range through user input or framework automatic detection can refer to the above embodiments and will not be elaborated here.

[0059] Based on any of the above embodiments, the steps for determining the logical relationship between various view class operators within the coverage range include: Based on the connection relationship between various view class operators within the coverage range, the operator attributes of each view class operator, and the tensor attributes of the connections between each view class operator, determine the logical relationship between each view class operator.

[0060] It should be noted that the connection relationship between various view class operators within the coverage range refers to the associations in aspects such as the sequence, dependency relationship, and data flow in the computational process among multiple View class operators within the same coverage range in the operator fusion scenario.

[0061] The operator attributes of each view class operator refer to various parameters and characteristics that describe the characteristics and behaviors of the view class operator itself. These attributes reflect the functions, limitations, and characteristics of the operator during the calculation process. Operator attributes can include operation types, input / output requirements, parameter configurations, etc. Among them, the operation type refers to the specific operation that the operator performs on the tensor, such as permutation, reshaping, splicing, etc.; the input / output requirements refer to the requirements for the shapes, data types, etc. of the input tensor and the output tensor; the parameter configuration refers to the additional information required when the operator is executed. For example, the parameter of the permute operator is a tuple of dimension orders. For example, the parameter (0, 2, 1) means that the 0th dimension of the input tensor remains unchanged, and the 1st and 2nd dimensions are swapped.

[0062] The tensor attributes of the connection between each view class operator refer to various characteristics and parameters of the tensor that connects two adjacent view class operators. These attributes describe information such as the shape, data type, and storage layout of the tensor.

[0063] Specifically, according to the connection relationships of each view class operator, the operator attributes of each view class operator, and the tensor attributes of the connection between each view class operator within the above coverage range, the logical relationships between each view class operator can be determined through the following steps: First, based on the connection relationships, the sequence and dependency relationships between the operators can be clarified. For example, if the output tensor of operator A is the input tensor of operator B, then it can be determined that A must be executed before B, and B depends on the output result of A. At the same time, according to the connection relationships, it can be known which operator the data flows from and which operator it flows to, as well as the data transfer method between the operators. Then, for each view class operator within the coverage range, according to its operation type, parameter configuration, and the tensor attributes of the connection with adjacent view class operators, the logical operation method between the operators can be determined. For example, if A is a permute operator and B is a reshape operator, according to the parameter configuration of A and the attributes of the input tensor, it can be determined which dimensions of the input tensor are rearranged, and according to the parameter configuration of B and the attributes of the output tensor of A, it can be further determined how B specifically reshapes the output tensor of A. Repeat this analysis process until the logical relationships between each view class operator within the coverage range are obtained, thus providing an accurate basis for subsequent operator fusion.

[0064] Based on any of the above embodiments, in step 230, the data filling of the output tensor based on the mapping relationship and each data in the input tensor includes: Step 231, traverse the target indices on each dimension of the output tensor.

[0065] Specifically, the target indices on each dimension of the output tensor refer to the combination of indices used to uniquely identify the position of each element in the multi-dimensional array structure of the output tensor. For example, for a three-dimensional output tensor with a shape of [B, H, W], the position of an element can be represented by a triple (i, j, k), where i is the index of the first dimension, j is the index of the second dimension, and k is the index of the third dimension. These indices (i, j, k) are the target indices, which specify the target position of the data in the output tensor.

[0066] It can be understood that traversing the target indices on each dimension of the output tensor can be achieved through nested loops, and the specific implementation method depends on the number of dimensions of the output tensor. For example, for a three-dimensional output tensor, the target indices on each dimension can be traversed through the following nested loop: for i in range(output_tensor.shape[0]): for j in range(output_tensor.shape[1]): for k in range(output_tensor.shape[2]): In the above example, output_tensor represents the output tensor, shape[0] represents the first dimension of the tensor, shape[1] represents the second dimension of the tensor, and shape[2] represents the third dimension of the tensor. For tensors with higher dimensions, it can be inferred by analogy, and more nested loops can be used to implement it, which will not be elaborated here.

[0067] Step 232: Based on the mapping relationship, determine the source index in the input tensor corresponding to the currently traversed target index.

[0068] It should be noted that the currently traversed target index refers to the combination of indices that are currently being processed when traversing each dimension of the output tensor in Step 231. For example, when traversing a three-dimensional output tensor, the currently traversed target index may be (2, 3, 1), indicating that the element in the output tensor with the first dimension index of 2, the second dimension index of 3, and the third dimension index of 1 is currently being processed.

[0069] The source index in the input tensor corresponding to the currently traversed target index refers to the combination of indices obtained by mapping the currently traversed target index to the input tensor according to the mapping relationship. This source index is used to read data from the input tensor to fill the currently traversed target position in the output tensor.

[0070] Specifically, the mapping relationship characterizes the corresponding relationship between the dimensions of the input tensor and the output tensor, which can be a function, a lookup table, or a mathematical formula, and the specific form depends on the shapes and transformation methods of the input and output tensors.

[0071] For example, if the mapping relationship is represented by a mathematical formula, the source index can be directly calculated through a function. Assume that the target index (i, j, k) of the output tensor corresponds to the source index (i', j', k') of the input tensor, and the mapping relationship is i' = f1(i, j, k), j' = f2(i, j, k), k' = f3(i, j, k). Then, the source index can be calculated by calling these functions.

[0072] For another example, if the mapping relationship is relatively complex, the mapping relationship between all target indices and source indices can be pre-calculated and stored in a lookup table. When traversing the output tensor, directly look up the corresponding source index from the lookup table.

[0073] Step 233: Based on the source index, read the corresponding data from the input tensor, and fill the data into the output tensor according to the currently traversed target index.

[0074] Specifically, after determining the source index corresponding to the target index, the data can be read from the input tensor using the source index. For example, if the source index is (i', j', k'), the data at the corresponding position in the input tensor can be read through input_tensor[i', j', k']. Subsequently, using the currently traversed target index, the read data is filled into the output tensor. For example, if the currently traversed target index is (i, j, k), the data can be filled into the corresponding position in the output tensor through output_tensor[i, j, k] = data.

[0075] Based on any of the above embodiments, step 233 specifically includes: Step 2331: Based on the source index, read the corresponding data from the input tensor into a temporary storage area, where the temporary storage area can be any one of a register, shared memory, and cache; Step 2332: Based on the currently traversed target index, write the data in the temporary storage area to the output tensor.

[0076] It should be noted that considering that in some hardware devices (such as GPUs), the access latency of global memory is relatively high and the bandwidth is limited. If data is directly read from global memory and written back to global memory, it will cause a memory bandwidth bottleneck and reduce the computing efficiency. In view of this, based on the consideration of hardware architecture and performance optimization, the embodiments of the present invention propose that during the process of data mapping and filling, the data in the input tensor can be first read into the scratchpad area and then written from the scratchpad area to the output tensor.

[0077] Specifically, the scratchpad area is a memory area for temporarily storing data, which can accelerate data access and processing. Here, the scratchpad area can be registers, shared memory, caches, etc. Their access latency is much lower than that of global memory and is suitable as a data transfer station. Specifically, data can be read from the global memory address corresponding to the source index of the input tensor through a memory access instruction and stored in the scratchpad area; subsequently, through a memory access instruction, the data in the scratchpad area is written back to the global memory address corresponding to the target index of the output tensor.

[0078] Based on any of the above embodiments, in step 230, the data filling of the output tensor based on the mapping relationship and each data in the input tensor includes: Based on the mapping relationship, determine the corresponding dimension of any dimension of the output tensor in the input tensor; When the target index of any dimension and the source index of the corresponding dimension are both continuous and in the same order, use a data block read instruction to read each data on the corresponding dimension of the input tensor and fill each data on the any dimension of the output tensor.

[0079] It should be noted that considering that GPUs usually have dedicated optimized access methods for tensors of specific layouts (layouts). For example, for tensor types of Matrix3D (three-dimensional matrix), GPU kernels can read and write multiple data at once through coalesced memory access during memory access. By using such dedicated memory access instructions, the memory access performance of the fusion operator can be further improved.

[0080] Specifically, when filling data into the output tensor based on the mapping relationship and each piece of data in the input tensor, the corresponding relationship between each dimension of the output tensor and each dimension of the input tensor can be determined according to the mapping relationship first. Then, the relationship between the target index on each output dimension and the source index on the corresponding input dimension is judged. If the target index on a certain output dimension and the source index on the corresponding input dimension are both continuous and in the same order, it indicates that the data on this output dimension is also continuously stored on the corresponding input dimension, and the order of the target index on this output dimension is the same as the order of the source index on the corresponding input dimension. In this case, since the data is continuous and in the same order, the data block read instruction can be used to read multiple pieces of data at one time, merge the access operations, thereby improving the memory access efficiency and reducing the occupation of the memory bandwidth. Here, the data block read instruction refers to an instruction that can read multiple consecutive pieces of data at one time.

[0081] If the target index on a certain output dimension and the source index on the corresponding input dimension are not continuous or not in the same order, it indicates that the data on this output dimension is not continuously stored on the corresponding dimension of the input tensor. In this case, since the data is not continuous or the order is not always the same, the data block read instruction cannot be used, and each element needs to be read one by one, and the corresponding data element is filled into the output tensor.

[0082] In the embodiments of the present invention, through the intelligent analysis and optimization of the memory layout of the tensor, it can be ensured that consecutive view operations are completed in one memory access, avoiding unnecessary memory copies and layout conversions, and improving the overall performance.

[0083] Based on any of the above embodiments, the input tensor includes one or more, and the output tensor includes one or more.

[0084] Specifically, in practical applications, the operator may need to process different numbers and types of input data. For example, in an image processing task, an operator may need to process multiple input images simultaneously (such as multi-view images or images of different scales), or in a deep learning model, a layer may need to receive the outputs from multiple previous layers. The operator fusion method provided by the embodiments of the present invention can support scenarios with one or more input tensors and one or more output tensors, enabling the fused operator to flexibly adapt to various complex scenarios.

[0085] Based on any of the above embodiments, the metadata includes the shape, dimension, and stride of the tensor.

[0086] Specifically, the metadata of a tensor refers to the information that describes the properties or structure of the tensor itself, rather than the actual data stored in the tensor. The metadata of a tensor includes, but is not limited to, the shape, dimension, stride, etc. of the tensor. Among them, the shape is used to describe the dimensional structure and dimensional size of the tensor, and the dimension is used to describe the number of dimensions of the tensor. For example, the shape of a tensor can be [3, 4], which means the tensor is a matrix with 3 rows and 4 columns, and the dimension of this tensor is 2, indicating that it is a two-dimensional tensor.

[0087] The stride refers to the number of bytes to skip when accessing tensor elements in memory and moving one unit along each dimension. For example, (4, 1) means that the adjacent row elements in memory are separated by 4 bytes, and the adjacent column elements are separated by 1 byte. It should be understood that in a deep learning framework, the metadata of a tensor can be obtained by calling corresponding interface functions (such as shape, stride).

[0088] Based on any of the above embodiments, the embodiments of the present invention provide an operator fusion method for View operators in a large model. By efficiently and quickly fusing multiple consecutive View operators, the problem of performance degradation caused by frequent data movement in the model is solved. The core of the operator fusion optimization strategy of the present invention lies in the fast implementation of operator fusion. According to the shape transformation relationship of tensors, the position relationship of data points in the input tensor and output tensor of the fused operator is deduced, so as to optimize multiple consecutive View operators into one fused operator. In this way, when the model is executed, not only the number of view transformations is reduced, but also the overhead of data movement is effectively reduced.

[0089] The View operator fusion optimization strategy proposed in the embodiments of the present invention is applicable to application scenarios where multiple consecutive View operators appear in the model structure. The specific implementation steps of this fusion optimization strategy include: Step S1, determine the coverage range, input tensor, and output tensor of the fused operator. Here, there may be more than one input tensor or output tensor.

[0090] It should be noted that the coverage range of the above-mentioned fused operator can be specified by the user, that is, when the user calls the operator in the upper-level framework, directly specify which View operators are included in the fused operator and the connection relationship between each operator; it can also be determined by the automatic detection of the combination of View operators inside the framework. When it is detected that all operators from a certain node to a certain node are View operators, these View operator combinations can be automatically replaced with the corresponding fused operator.

[0091] Step S2, obtain the relevant information of the input tensor, including the shape, dimension, stride, etc. of the input tensor, as a reference.

[0092] Step S3, obtain the relevant information of the output tensor, including the shape, dimension, stride, etc. of the output tensor, as a reference.

[0093] Step S4, determine the corresponding relationship between each dimension of the input tensor and the output tensor according to the logical relationship between the View class operators used within the coverage of the fusion operator, as well as the relevant information of the input tensor and the output tensor; Step S5, according to the above corresponding relationship, traverse the data points of each dimension of the output tensor, directly read the corresponding original data from the input tensor, and store it into the data points of the output tensor.

[0094] Next, take the fusion operator in a specific model application (such as the DeepSeek model) as an example to introduce the specific implementation of the above optimization strategy.

[0095] The DeepSeek model structure uses the MLA (Multi-Latent Attention) mechanism, which is a new method for processing the attention mechanism in deep learning, especially when dealing with complex inputs with multiple latent spaces. The traditional attention mechanism (such as the self-attention mechanism) uses the same query (Q), key (K), and value (V) matrices during calculation, while MLA adopts a different approach and introduces the concept of multiple latent spaces in its attention calculation, thus allowing the model to perform more fine-grained learning in different latent subspaces.

[0096] In the traditional self-attention mechanism, the shapes of the Q, K, and V tensors are the same, usually (batch_size, sequence_length, embedding_dim). Here, the first dimension batch_size is the batch size, indicating the number of samples processed simultaneously; the second dimension sequence_length is the sequence length, indicating the number of elements in the sequence for each sample; the third dimension embedding_dim is the embedding dimension, indicating the dimension of the embedding vector for each sequence element. However, in MLA, due to the use of mappings in different latent spaces, the shapes of the K and V tensors are different. Before entering MLA, the K and V tensors need to undergo a series of transformations, including multiple consecutive View class operators. By fusing these consecutive View class operators into a single fusion operator, the data movement in memory can be significantly reduced.

[0097] Figure 4 is a schematic diagram of the specific structure and coverage of the fusion operator provided by the present invention, as Figure 4As shown, the input tensor includes KV_B and K_PE, where the shape of KV_B is [batch_size, seqlen, headdim_b], and the shape of K_PE is [batch_size, seqlen, headdim_pe]; the output tensors are K and V, where the shape of K is [batch_size repeat, seqlen, headdim_k], and the shape of V is [batch_size repeat, seqlen, headdim_v]. Here, headdim_k = headdim_pe + headdim_v.

[0098] In the process of transforming the input tensor to obtain the output tensor, the operations of the View class operator involved include: 1) KV_B is transformed into [batch_size repeat, seqlen, 2 headdim_v] through the reshape (reshaping) operator and the permute (permutation) operator, where headdim_b = repeat (2 headdim_v), denoted as KV ( Figure 4 only reshape is shown here, actually it is reshape + permute); 2) KV is split into two tensors, namely K_NOPE and V, by the split operator, and the split dimension is the innermost dimension (i.e., Dim = 2), and the V tensor is directly output as the output tensor; 3) K_PE is repeated repeat times in the Dim = 0 dimension through the repeat operator to obtain the tensor K_PE_repeat; 4) K_PE_repeat and K_NOPE are concatenated through the concat operator in the innermost dimension (i.e., Dim = 2) and merged into K, which is directly output as the output tensor.

[0099] Assume the shape of the input tensor K_PE is [batch_size, seqlen, headdim_pe] = [B, 4096, 64], then after repeating K_PE 128 times in the Dim = 0 dimension through the repeat operator, the shape of the obtained tensor K_PE_repeat is [batch_size repeat, seqlen, headdim_pe] = [B 128, 4096, 64]。

[0100] Assume the shape of the input tensor KV_B is [batch_size, seqlen, headdim_b]=[B, 4096, 32768], then after reshape and permute, the obtained tensor KV is [batch_size repeat, seqlen, 2 headdim_v]= [B 128, 4096, 256]; subsequently, use the split operator to split the second dimension of KV, and the shapes of the two obtained tensors K_NOPE and V are both [batch_size repeat, seqlen, headdim_v] = [B 128, 4096, 128], and the tensor V is directly output as the output tensor. Use the concat operator to concatenate the second dimensions of K_PE_repeat and K_NOPE, and the output tensor K can be obtained as [batch_size repeat, seqlen, headdim_pe + headdim_v]= [B 128, 4096, 192].

[0101] As described above, since non - continuous memory processing operators such as permute, split, repeat, and concat appear within the scope of this fusion operator (as shown by the dashed box in Figure 4 ), it cannot be processed through view transformation at the framework level, and actual memory relocation must be performed. The operators within this scope require a total of 4 tensor reads and 4 tensor stores from global memory. For the DeepSeek model, due to the large shape of the tensors, the involved memory relocation will have a significant negative impact on performance.

[0102] After using the operator fusion optimization strategy provided by the embodiments of the present invention, the fusion operator as shown in Figure 5 can be obtained. Figure 5 is a schematic diagram of the mapping relationship between the input tensor and the output tensor provided by the present invention. As shown in Figure 5 , the green tensor in the figure is the input tensor, and the blue and orange tensors are the K and V output tensors respectively. The gray tensor is a virtual tensor, which no longer applies for an actual global memory space, but performs data relocation through registers inside the GPU kernel. After determining this fusion operator, the specific processing flow is as follows: 1) Traverse the target indices on each dimension of the output tensors K and V; 2) According to the corresponding relationship, determine the source indices in the input tensors KV_B and K_PE corresponding to the target index being traversed currently, and read the corresponding data from the input tensors KV_B and K_PE into the register according to these source indices; 3) Write the data in the register to K and V according to the target index being traversed currently.

[0103] Compared with the multiple memory accesses at the beginning, the above process significantly reduces the overhead of data transfer. In addition, GPUs usually have dedicated optimized memory access methods for tensors with specific layouts. For example, for the tensor type of Matrix3D, the GPU Kernel can read and write multiple data at once through coalesced memory access during memory access. By using such dedicated memory access instructions, the memory access performance of the fused operator can be further improved.

[0104] The operator fusion optimization strategy proposed in the embodiments of the present invention can perform intelligent analysis and optimization on the memory layout of tensors, ensuring that consecutive view operations can be completed in one memory access, avoiding unnecessary memory copies and layout conversions. Moreover, the present invention also takes into account the requirements of the hardware architecture for memory access, such as the memory access mode and cache optimization mechanism of GPUs. During the implementation of the strategy, it is possible to perform special scheduling and optimization on view fusion according to the characteristics of different hardware platforms, so as to maximize the memory access efficiency of data transfer. For the frequent and large number of View class operations in large models, the present invention can effectively reduce the number of memory accesses and the overhead of data transfer, thereby optimizing the performance degradation problem caused by frequent data transformations.

[0105] The operator fusion device provided by the present invention will be described below. The operator fusion device described below can be correspondingly referred to the operator fusion method described above.

[0106] Based on any of the above embodiments, Figure 6 is a schematic structural diagram of the operator fusion device provided by the present invention, as Figure 6 shown. The device includes: A range determination unit 610, configured to determine the input tensors, output tensors, and coverage range of the operators to be fused, where the operators to be fused include multiple view class operators; A relationship determination unit 620, configured to determine the mapping relationship between each dimension of the input tensor and the output tensor based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationship between each view class operator within the coverage range; The operator fusion unit 630 is configured to perform data filling on the output tensor based on the mapping relationship and each piece of data in the input tensor, and use the filled output tensor as the fusion result of the operator to be fused.

[0107] The device provided by the embodiment of the present invention can accurately deduce the mapping relationship between the dimensions of the input tensor and the output tensor of the operator to be fused based on the metadata of the input tensor and the output tensor and the logical relationship between each view class operator. Therefore, according to this mapping relationship, each piece of data in the input tensor can be accurately filled into the corresponding position of the output tensor, efficiently realizing operator fusion, that is, integrating and optimizing multiple view class operators that originally need to be continuously executed into one fused operator. Through this fusion method, during the operation of the model, the number of executions of view transformation operations is significantly reduced, and the number of data transfers is also reduced accordingly, thus effectively reducing the additional overhead caused by data transfer and solving the problem of performance degradation caused by frequent data transfer in the model.

[0108] Based on any of the above embodiments, the coverage range is obtained based on user input or automatic detection by the framework. The device further includes a logical relationship determination unit, and the logical relationship determination unit is configured to: Determine the logical relationship between each view class operator based on the connection relationship between each view class operator within the coverage range, the operator attributes of each view class operator, and the tensor attributes of the tensors connected between each view class operator.

[0109] Based on any of the above embodiments, the operator fusion unit 630 includes: A traversal subunit, configured to traverse the target indexes on each dimension of the output tensor; An index determination subunit, configured to determine the source index corresponding to the currently traversed target index in the input tensor based on the mapping relationship; A data filling subunit, configured to read the corresponding data from the input tensor based on the source index and fill the data into the output tensor according to the currently traversed target index.

[0110] Based on any of the above embodiments, the data filling subunit is specifically configured to: Read the corresponding data from the input tensor to a temporary storage area based on the source index, where the temporary storage area is any one of a register, a shared memory, and a cache; Write the data in the temporary storage area to the output tensor based on the currently traversed target index.

[0111] Based on any of the above embodiments, the operator fusion unit 630 is specifically configured to: Determine the corresponding dimension of any dimension of the output tensor in the input tensor based on the mapping relationship; When the target index in any of the dimensions is continuous and in the same order as the source index in the corresponding dimension, a data block read instruction is used to read each piece of data in the corresponding dimension of the input tensor and fill each piece of data into any dimension of the output tensor.

[0112] Based on any of the above embodiments, the input tensor includes one or more, and the output tensor includes one or more.

[0113] Based on any of the above embodiments, the metadata includes the shape, dimension, and stride of the tensor.

[0114] Figure 7 An entity structure diagram of an electronic device is exemplified, as Figure 7 shown. The electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communication interface 720, and the memory 730 complete communication with each other through the communication bus 740. The processor 710 may call the logical instructions in the memory 730 to execute the operator fusion method, and the method includes: determining the input tensor, output tensor, and coverage range of the operator to be fused, where the operator to be fused includes multiple view class operators; determining the mapping relationship between the dimensions of the input tensor and the output tensor based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationship between the view class operators within the coverage range; filling the output tensor with data based on the mapping relationship and the data in the input tensor, and using the filled output tensor as the fusion result of the operator to be fused.

[0115] In addition, when the logical instructions in the above-mentioned memory 730 are implemented in the form of a software functional unit and sold or used as an independent product, they may be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the related technology, or a part of this technical solution, may be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0116] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the operator fusion method provided by each of the above methods. The method includes: determining the input tensor, output tensor, and coverage range of the operator to be fused, where the operator to be fused includes multiple view class operators; based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationship between the view class operators within the coverage range, determining the mapping relationship between the dimensions of the input tensor and the output tensor; based on the mapping relationship and the data in the input tensor, filling the output tensor with data, and using the filled output tensor as the fusion result of the operator to be fused.

[0117] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it is used to execute the operator fusion method provided by each of the above methods. The method includes: determining the input tensor, output tensor, and coverage range of the operator to be fused, where the operator to be fused includes multiple view class operators; based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationship between the view class operators within the coverage range, determining the mapping relationship between the dimensions of the input tensor and the output tensor; based on the mapping relationship and the data in the input tensor, filling the output tensor with data, and using the filled output tensor as the fusion result of the operator to be fused.

[0118] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. A person of ordinary skill in the art can understand and implement it without creative labor.

[0119] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that contributes to the related technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disc, etc., and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. An operator fusion method, characterized in that Including: Determine the input tensor, output tensor, and coverage range of the operator to be fused, where the operator to be fused includes multiple view-based operators; Based on the metadata of the input tensor, the metadata of the output tensor, and the logical relationships among the view-based operators within the coverage range, determine the mapping relationships between the dimensions of the input tensor and the output tensor; Based on the mapping relationships and the data in the input tensor, perform data filling on the output tensor, and use the filled output tensor as the fusion result of the operator to be fused.

2. The operator fusion method according to claim 1, wherein The coverage range is obtained based on user input or automatic detection by the framework. The steps for determining the logical relationships among the view-based operators within the coverage range include: Based on the connection relationships of the view-based operators within the coverage range, the operator attributes of each view-based operator, and the tensor attributes of the connections between the view-based operators, determine the logical relationships among the view-based operators.

3. The operator fusion method according to claim 1, wherein The performing data filling on the output tensor based on the mapping relationships and the data in the input tensor includes: Traverse the target indices on each dimension of the output tensor; Based on the mapping relationships, determine the source indices in the input tensor corresponding to the currently traversed target indices; Based on the source indices, read the corresponding data from the input tensor, and fill the data into the output tensor according to the currently traversed target indices.

4. The operator fusion method according to claim 3, wherein The based on the source indices, reading the corresponding data from the input tensor, and filling the data into the output tensor according to the currently traversed target indices includes: Based on the source indices, read the corresponding data from the input tensor into a temporary storage area, where the temporary storage area is any one of a register, shared memory, and cache; Based on the currently traversed target indices, write the data in the temporary storage area into the output tensor.

5. The operator fusion method according to claim 1, wherein The performing data filling on the output tensor based on the mapping relationships and the data in the input tensor includes: Based on the mapping relationships, determine the corresponding dimension in the input tensor for any dimension of the output tensor; When the target indices of any dimension are continuous and in the same order as the source indices of the corresponding dimension, use a data block read instruction to read the data on the corresponding dimension of the input tensor, and fill the data into the any dimension of the output tensor.

6. The operator fusion method according to any one of claims 1 to 5, characterized in that The input tensor includes one or more, and the output tensor includes one or more.

7. The operator fusion method according to any one of claims 1 to 5, characterized in that The metadata includes the shape, dimension, and stride of the tensor.

8. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the operator fusion method according to any one of claims 1 to 7.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the operator fusion method according to any one of claims 1 to 7.

10. A computer program product comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the operator fusion method according to any one of claims 1 to 7.

Images