Optimizing operation of tensors for processing devices

Abstract

According to implementations of the present disclosure, a solution is provided to optimize operation of tensors for processing devices. According to the scheme, the target window size is determined based on the size of the target tensor for the target operation which is to be executed on the processing equipment and is associated with the target tensor. At least one data block is determined based on the target window size and the target tensor. The set of elements included in the at least one database covers at least the target tensor. And for each data block, based on the position of each element in the database in the target tensor, executing at least one predetermined task in a plurality of predetermined tasks corresponding to the target operation and the target window size. The predetermined tasks are pre-compiled to implement a target operation for the processing device. In this manner, a universal and efficient solution for tensors of different sizes may be implemented for various types of processing devices.

Description

Background Art

[0001] Deep neural networks (DNNs) are rapidly developing and have been applied or have promising applications in a wide range of fields. This has significantly increased the demand for running DNNs on various hardware accelerators, such as central processing units (CPUs), graphics processing units (CPUs), and tensor processing units (TPUs). DNN computations can typically be abstracted as compute-intensive operators on tensors, such as matrix multiplication, and implemented as kernels running on hardware accelerators. Therefore, optimizing operations on tensors can significantly improve the efficiency of DNN training and inference. Summary of the Invention

[0002] According to the implementation of the present disclosure, a scheme for optimizing operations on tensors is proposed. In this scheme, for a target operation associated with a target tensor to be executed on a processing device, a target window size is determined based on the size of the target tensor. Based on the target window size and the target tensor, at least one data block is determined. The set of elements included in the at least one database covers at least the target tensor. For each data block, based on the position of each element in the database in the target tensor, at least one predetermined task of a plurality of predetermined tasks corresponding to the target operation and the target window size is executed. These predetermined tasks are pre-compiled for the processing device to implement the target operation. The implementation of the present disclosure supports the processing of tensors of non-fixed size on a processing device while avoiding the need for intensive code generation and compilation. In this way, for various types of processing devices, a code execution scheme in the form of a general runtime library that is efficient for tensors of different sizes can be implemented.

[0003] This section is provided to introduce a selection of objects in a simplified form, which will be further described in the detailed description below. This section is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Figure 1 A block diagram illustrating an example environment in which various implementations of the present disclosure can be implemented;

[0005] Figure 2 A schematic diagram illustrating an example architecture for performing target operations according to some implementations of the present disclosure is shown;

[0006] Figures 3A to 3D Several examples of processing tensors using primitive tasks according to some implementations of the present disclosure are shown;

[0007] Figure 4 A schematic diagram showing how primitive task allocation varies with computation size according to some implementations of the present disclosure;

[0008] Figure 5A An example of predetermining an optimized window size by sampling according to some implementations of the present disclosure is shown;

[0009] Figure 5B An example of using a predetermined mapping table according to some implementations of the present disclosure is shown;

[0010] Figure 6 A flowchart illustrating a process for processing tensors according to some implementations of the present disclosure; and

[0011] Figure 7 A schematic block diagram of an electronic device capable of implementing various implementations of the present disclosure is shown. DETAILED DESCRIPTION

[0012] The present disclosure will now be discussed with reference to several example implementations. It should be understood that these implementations are discussed only to enable those skilled in the art to better understand and implement the present disclosure, rather than to imply any limitation on the scope of the present disclosure.

[0013] As used herein, the term "including" and its variations are to be interpreted as open-ended terms meaning "including but not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The terms "an implementation" and "an implementation" are to be interpreted as "at least one implementation." The term "another implementation" is to be interpreted as "at least one other implementation." The terms "first," "second," and so on may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0014] It should be noted that the titles of any section / subsection provided herein are not limiting. Various implementations are described throughout this document, and any type of implementation may be included under any section / subsection. Furthermore, the implementation described in any section / subsection may be combined in any manner with any other implementation described in the same section / subsection and / or in different sections / subsections.

[0015] As used herein, a group of elements, element groups or similar statements may include zero, one or more such elements. The group of elements may be ordered or unordered. For example, "a group of separators" may include zero, one or more separators. As used in the text, an element sequence or similar statements may include one or more such elements, and the elements in the sequence are ordered.

[0016] As used herein, the term "model" can learn the association between the corresponding input and output from the training data, so that after the training is completed, the corresponding output can be generated for a given input. The generation of the model can be based on machine learning technology. Deep learning (DL) is a machine learning algorithm that processes input and provides corresponding output by using multiple layers of processing units. A neural network model is an example of a model based on deep learning. In this article, "model" may also be referred to as "machine learning model", "learning model", "machine learning network" or "learning network", and these terms are used interchangeably in this article.

[0017] As used herein, the term "tensor" can have any suitable number of dimensions, and each dimension can have any suitable length. For example, a tensor can be a one-dimensional tensor (i.e., a vector), a two-dimensional tensor (i.e., a matrix), a three-dimensional tensor, or a tensor of any other number of dimensions. As used herein, "tensor size," "tensor shape," or similar expressions refers to the length of a tensor in one or more of its dimensions.

[0018] Sample Environment

[0019] Figure 1 A schematic diagram of an example environment 100 is shown in which implementations of the present disclosure can be implemented. Figure 1 In the environment 100, a machine learning model 120 is run on a processing device 110 to obtain an inference result or train the machine learning model 120. The processing device 110 can be any hardware system or hardware platform with computing capabilities. The processing device 110 can be any suitable type of general-purpose processing device or a dedicated processing device. Examples of general-purpose processing devices include, but are not limited to, CPUs, GPUs, field programmable gate arrays (FPGAs), etc. Examples of dedicated processing devices include, but are not limited to, accelerated processing units (APUs), TPUs, neural network processing units (NPUs), or other existing or future developed artificial intelligence (AI chips). It is understandable that such accelerated processing resources are mainly used to increase the computing speed of AI-related applications.

[0020] like Figure 1 As shown, the machine learning model 120 can be abstracted into one or more operations 130-1, 130-2, ..., 130-N on tensors, which are also collectively or individually referred to as operations 130, where N is a positive integer. Operations 130 can include various suitable general-purpose calculations or operations, such as but not limited to matrix multiplication, convolution, accumulation, transposition, etc. Alternatively or additionally, operations 130 can include custom calculations or operations, such as padding, convolution padding, wildcard matching, etc.

[0021] The operations 130 of the machine learning model 110 are compiled into machine-executable code, also known as a kernel. When an operation 130 of various computational sizes is requested, the computational result is obtained by running the implementation of the operation 130 on the processing device 110 at runtime.

[0022] It should be understood that the structure and functionality of the environment 100 are described for exemplary purposes only and do not imply any limitation on the scope of the present disclosure. In embodiments of the present disclosure, the operations and custom calculations performed by the processing device 110 may be derived from other types of models, algorithms, or traditional data processing services, and are not limited to machine learning models.

[0023] Taking the widely used deep neural network (DNN) as an example, DNN operators need to process tensors of arbitrary sizes. In order to run DNN on a processing device, there are conventionally two main types of solutions. In one solution, the tensor compiler generates device-specific kernels on demand through tuning. This solution generates a large number of kernel configurations for a given DNN model, compiles each kernel configuration into an executable kernel, and evaluates the performance of these kernels on the target processing device to find the optimal configuration. This solution generates an optimized solution for tensors of a specific size, which is a static solution. If the size of the tensor changes, re-optimization is required. Therefore, this solution is efficient for DNNs with static and known tensor sizes, but it is not suitable for more dynamic models. In other words, although this solution is efficient, it is not universal.

[0024] Another solution aims to manually implement code that can handle common "tensor size" situations. This makes optimizing tensor operations difficult and time-consuming. In other words, while this solution is general, it is not efficient.

[0025] Despite the existence of these solutions, some issues remain. One major issue is that all of these solutions require generating and compiling a new kernel for each new tensor size. This can be a limiting factor in some cases, such as on processing devices without suitable compilers or in applications where compilation latency is unacceptable. Another issue with these solutions is that the kernel configurations that are designed to quickly estimate a good kernel configuration and generate new binary code on the fly for each new tensor size may not accurately reflect the performance of the final kernel. This is because the same kernel configuration or even the same kernel code can produce very different binary executables when compiled by different compilers, or multiple logically equivalent codes using the same compiler and the same kernel configuration can also produce very different binary executables.

[0026] To address each of the above problems, the implementation of the present disclosure provides a solution for optimizing operations on tensors for processing. According to the implementation of the present disclosure, multiple primitive tasks for performing target operations are pre-optimized and compiled for a processing device. Each primitive task processes a tensor of a specific size (also known as a window size). Each primitive task has multiple implementations, also known as multiple scheduled tasks. Each scheduled task can have a different degree of vectorization and a region-specific fast computing preference for processing different types of data blocks at runtime.

[0027] At runtime, the target window size is first determined for the target tensor to be processed, and the target tensor is divided into at least one data block of the target window size. A primitive task, pre-optimized and compiled for the target window size, is then executed for each data block. Based on the position of the elements included in the data block in the target tensor, at least one of the multiple predetermined tasks of the primitive task is executed. For each tensor position, the predetermined task with the best performance for that area is typically selected for execution.

[0028] In the implementation of the present disclosure, primitive tasks that can be shared by different tensor sizes are optimized by ahead of time (AoT) compilation, and the pre-compiled primitive tasks are used to implement the target operation at runtime. Therefore, the implementation of the present disclosure supports the processing of tensors of non-fixed sizes on the processing device while avoiding the optimization overhead of intensive code generation and compilation at runtime. In this way, a universal and efficient "pre-optimization before runtime, immediate execution at runtime" solution can be implemented for tensors of different sizes for various types of processing devices.

[0029] The following continues to describe example embodiments of the present disclosure with reference to the accompanying drawings.

[0030] Example Architecture

[0031] Figure 2 A schematic diagram of an example architecture 200 for executing target operations according to some implementations of the present disclosure is shown. Generally, the architecture 200 includes an execution-ahead portion, including a pre-optimization module 250 and a pre-sampling module 240. The architecture 200 also includes a runtime portion, including a target operation 230 associated with a target tensor, a task scheduling layer 210, and an abstraction layer 220 for a device interface to the processing device 110. The target operation 230 can be an example of the operation 130. The target tensor can be an input tensor or an output tensor of the target operation 230.

[0032] The pre-optimization module 250 is configured to pre-optimize and compile one or more primitive tasks 201-1, 201-2, ..., 201-M for the processing device 110, which are also collectively or individually referred to as primitive tasks 201, where M is a positive integer. Each primitive task 201 corresponds to an operation and a tensor size (which is also called a window size). The tensor size can be the size of the input tensor of the task, or the size of the output tensor. In other words, each primitive task 201 is pre-compiled for the processing device 110 to perform a corresponding operation for a corresponding window size. For example, a primitive task can be pre-optimized and compiled to perform matrix multiplication on at least two input matrices to output a matrix of size P×Q. In this example, the primitive task corresponds to the operation of "matrix multiplication" and the window size of P×Q.

[0033] At runtime, a window size to be used is determined for the target tensor, also referred to as the target window size. The target tensor is then divided into at least one data block according to the target window size. Each data block is executed by an instance of the primitive task corresponding to the target window size and the target operation 230.

[0034] Pre-optimization module 201 can optimize task 201 for processing device 110 using any suitable method. For example, an automatic tuning method can be used to find the optimal kernel configuration for processing device 110 and the corresponding window size. Another example is a configuration generator that can be used to recommend an efficient configuration for the corresponding window size. Since each primitive task is pre-generated once, the tuning cost is also one-time.

[0035] In some implementations, for the same operation, multiple primitive tasks corresponding to multiple window sizes (also referred to as candidate window sizes) may be pre-optimized and compiled. For example, for matrix multiplication, multiple primitive tasks corresponding to window sizes of 32×32, 64×64, and 128×128 may be pre-optimized and compiled. In such an implementation, during runtime processing of the target operation 230, a target window size for the target tensor may be selected from these candidate window sizes based on the size of the target tensor targeted by the target operation 230.

[0036] In some implementations, if the tensor size to be processed is limited, a window size appropriate for each possible tensor size can be pre-determined. Thus, at runtime, the pre-determined window size can be selected as the target window size based on the target tensor size.

[0037] In some cases, the tensor sizes that need to be processed may be diverse or even infinite. In view of this, in some implementations, the architecture 200 may also include a pre-sampling module 240, which is configured to determine the optimized window size for multiple sampling tensor sizes respectively. At runtime, a sampling tensor size that matches or is close to the target tensor size can be selected from multiple sampling tensor sizes. The optimized window size predetermined for the selected sampling tensor size can be determined as the target window size. In this way, versatility and flexibility are further improved. The recording process of the pre-sampling module 240 to determine the most efficient primitive task from the sampling tensor size is also a one-time process before running. The following will refer to Figure 5A 、 Figure 5B Such embodiments are further described.

[0038] The sampling tensor size and the corresponding optimization window size may be maintained in any suitable manner, such as using a mapping table 202. The key of the mapping table 202 may be the sampling tensor size, and the value may be the corresponding optimization window size.

[0039] The task scheduling layer 210 is configured to automatically schedule target operations 230 as primitive tasks. Furthermore, the task scheduling layer 210 can also support customized scheduling policies between different operations. For example, users can use the task scheduling layer 210 to define the scheduling order and scheduling throttling between different operations in the DNN model.

[0040] Abstraction layer 220 provides a unified interface for launching primitive tasks. Through this interface, a group of instances of the same primitive task can be launched, each instance processing a portion of a target tensor, for example, a data block with a target window size. For some advanced processing devices (such as CPUs), multiple primitive tasks designed for different target operations can be combined into a larger composite task for simultaneous execution, taking advantage of higher device utilization.

[0041] While the example architecture 200 is described above, it should be understood that the various components of the architecture 200 are merely exemplary and not intended to be limiting. Furthermore, it should be understood that the specific values ​​of tensor sizes and window sizes listed throughout this disclosure are merely exemplary and not intended to limit the scope of this disclosure. Implementations of this disclosure can be applied to any tensor size and window size.

[0042] Example primitive tasks

[0043] As mentioned above, each primitive task 201 corresponds to an operation and a window size. That is, the primitive task corresponding to a certain operation is the basic computational unit of the operation, which processes the portion of the target tensor with the corresponding window size (i.e., the data block with the corresponding window size). For example, for a DNN, a primitive task can be the basic computational unit of a DNN operator, which processes the portion of the output tensor with the corresponding window size.

[0044] Considering that the target tensor may be of any size, or the values ​​of the elements in the target tensor have a certain characteristic distribution (such as the padding operation or block sparse matrix example described below), the data blocks obtained by dividing the target tensor using the determined window size can be of different types. To this end, each primitive task 201 can have multiple scheduled tasks, each of which can be used to process a type of data block. These scheduled tasks can have different execution efficiencies. In particular, in some implementations, these scheduled tasks can have different degrees of vectorization.

[0045] Depending on the specific implementation or the differences in program execution interfaces constrained by the processing device (e.g., a single instruction multiple data (SIMD) processor, a multiple instruction multiple data (MIMD) processor, etc.), the multiple predetermined tasks included in each primitive task may be compiled together into a kernel, or each predetermined task may be compiled separately into a kernel. Implementations of the present disclosure are not limited in this respect.

[0046] Accordingly, at runtime, for each data block, at least one of the multiple predetermined tasks is executed based on the position of the elements in the data block in the target tensor. For example, the type of the data block can be determined based on the position of the elements in the data block in the target tensor, and then the predetermined task corresponding to the type from the multiple predetermined tasks is executed.

[0047] In some implementations, determining which predetermined task to perform may be based on a predetermined limit associated with a target tensor. If the elements included in a data block do not cross the predetermined limit, a high-efficiency predetermined task among multiple predetermined tasks, such as a high-vectorization predetermined task, may be performed for the data block. If the elements included in a data block cross the predetermined limit, a low-efficiency predetermined task, such as a low-vectorization or even non-vectorized predetermined task, may be performed for the data block. In the following, for illustrative purposes only, a data block whose elements do not cross the predetermined limit is referred to as an internal block, and a data block whose elements cross the predetermined limit is referred to as a boundary block.

[0048] The predetermined limits are associated with changes to elements in the target tensor. In some implementations, the predetermined limits may include one or more boundaries of the target tensor. Figure 3AAn example is described that shows a general basic partitioning example. In this example, the target tensor can be the output tensor of a general matrix multiplication (GEMM), and the predetermined boundaries are the boundaries of the target tensor, that is, the column with the largest column index and the row with the largest row index. According to the determined target window size, the target tensor is divided into 3×3 data blocks. Among these data blocks, the data block identified as AA (for example, data block 311) is a data block that does not cross the boundary, and thus the data block identified as AB (for example, data block 312) is a data block that crosses the boundary. Accordingly, the vectorized predetermined task AA is performed for the data block identified as AA; and the non-vectorized predetermined task AB is performed for the data block identified as AB.

[0049] Alternatively or additionally, in some implementations, the predetermined limit may include a limit for distinguishing the predetermined operations (also referred to as actions) applied by the target operation to the elements in the target tensor. That is, in the execution of the target operation, the specific operations applied to the elements on different sides of the limit are different. Figure 3B An example is described. In this example, the target operation is to pad the input tensor into an output tensor. Predetermined boundaries include two types of boundaries. One type of boundary is the boundary of the output tensor, namely the column with the smallest column index, the column with the largest column index, and the row with the largest row index. The other type of boundary 320 is used to distinguish the predetermined operations applied to the elements. Specifically, the operation applied to the elements within the boundary 320 is a copy of the input tensor, and the operation applied to the elements outside the boundary is padding. The data blocks identified as BA and BC are data blocks that do not cross the predetermined boundaries, so the vectorized predetermined tasks BA and BC are executed on these data blocks. The data blocks identified as BB are data blocks that cross the predetermined boundaries, so the non-quantized predetermined task BB is executed on these data blocks. Because the predetermined task BB needs to distinguish and process a variety of different computational details in a fine-grained manner, the predetermined task BB can also be used to calculate the areas identified as BC and BA, but the efficiency is often not as good as the predetermined tasks BC and BA.

[0050] In some implementations, the predetermined limits may include limits associated with the distribution of values ​​of elements in the target tensor. In other words, the target tensor may be divided into different regions by such limits, and the values ​​of elements in different regions have different distribution characteristics. As an example, the sparsity of tensors may be common in DNNs. For example, depending on whether the value of an element is zero, a tensor may be divided into sparse regions and dense regions. Figure 3CAn example is described in which predetermined boundaries 311 divide a target tensor into a sparse region and a dense region, and the predetermined boundaries also include the boundaries of the target tensor, namely, the column with the largest column index and the row with the largest row index. Data blocks identified as CA and CC do not cross predetermined boundaries 311 or the boundaries. Therefore, vectorized predetermined tasks CA and CC are executed for these data blocks. Data blocks identified as CB cross predetermined boundaries 331 and / or the boundaries, so non-vectorized predetermined tasks CB are executed for these data blocks.

[0051] As can be seen from the above examples, for data blocks that do not cross the predetermined boundaries, vectorized tasks can be used to achieve efficient processing, while for data blocks that cross the predetermined boundaries, a conservative implementation method can be used. It can be understood that for tensors, especially large tensors, the number of internal blocks is larger than the number of boundary blocks. Therefore, the proportion of relatively inefficient tasks is relatively low. In this case, computational efficiency can be improved.

[0052] In some implementations, the processing of data blocks that do not cross a predetermined boundary can be further refined. For example, the predetermined boundary can divide the target tensor into different parts. Each part can correspond to one of multiple predetermined tasks of the primitive task. For data blocks that do not cross the predetermined boundary, the predetermined task corresponding to the part in which the data block is located can be selected for execution.

[0053] As an example, refer again to Figure 3B The predetermined boundary 320 divides the target tensor into an inner region within the predetermined boundary and an outer region outside the predetermined boundary 320. For the data blocks identified as BA in the inner region, the predetermined task BA is executed, which may be a copy of the input tensor from the memory. For the data blocks identified as BC in the outer region, the predetermined task BC is executed, which may be filling the entire data block with a certain value.

[0054] As another example, refer again to Figure 3C The predetermined boundary 331 divides the target tensor into a dense region in the lower left corner and a sparse region in the upper right corner. For the data blocks identified as CA in the dense region, the predetermined task CA is performed, for example, it can be a matrix multiplication implemented in a vectorized manner. For the data blocks identified as CC in the sparse region, the predetermined task CC is performed, for example, it can be filling with a predetermined value.

[0055] In this implementation, the execution efficiency of various custom calculations can be improved by refining the processing of data blocks that do not cross predetermined boundaries and extracting the implicit and shared local calculation characteristics within the region.

[0056] In some implementations, the processing of data blocks that cross a predetermined boundary can be further refined. For example, the data block can be further divided into sub-blocks that cross the predetermined boundary and sub-blocks that do not cross the predetermined boundary. A vectorized predetermined task can be performed on the sub-blocks that do not cross the predetermined boundary, with the degree of vectorization of the predetermined task being less than the degree of vectorization of the predetermined task performed on the data blocks that do not cross the predetermined boundary. In some implementations, a non-vectorized predetermined task can be performed on the sub-blocks that cross the predetermined boundary. Alternatively, in some implementations, the processing of the sub-blocks that cross the predetermined boundary can be further refined, for example, by further dividing them into a portion that crosses the predetermined boundary and a portion that does not cross the predetermined boundary.

[0057] refer to Figure 3D An example is described. In this example, the predetermined boundaries are the boundaries of the target tensor, which include the column with the largest column index and the row with the largest row index. Data blocks identified as DA do not cross the predetermined boundaries, so the fully vectorized predetermined task DA, such as matrix multiplication, is performed on these data blocks. Data blocks 341-345 cross the predetermined boundaries. For data blocks 343-345, the number of elements within the predetermined boundaries does not exceed half of the data blocks, so further refinement is not performed on data blocks 343-345, and instead the non-vectorized predetermined task DB is performed. Refinement is performed on data blocks 341 and 342. Data blocks 341 and 342 are each divided into two sub-blocks. For sub-blocks identified as DC that do not cross the predetermined boundaries, the semi-vectorized predetermined task DC can be performed. For sub-blocks identified as DB that cross the predetermined boundaries, the non-vectorized predetermined task DB can be performed.

[0058] The sub-block and / or data block identified as DB may be further refined. For example, a predetermined task of optimizing and compiling a quarter of vectorization may be scheduled to process a portion of the sub-block and / or data block identified as DB.

[0059] Examples of primitive tasks are described above. Each primitive task can have multiple implementations, i.e., multiple predetermined tasks. Among these predetermined tasks, tasks corresponding to data blocks and / or sub-blocks that do not cross the predetermined boundaries can be optimized to improve the efficiency of the main calculation, while tasks corresponding to data blocks and / or sub-blocks that cross the predetermined boundaries are used to ensure the continuity of performance. In this way, in the implementation of the present disclosure, the continuity of computing performance can be ensured when the computing size changes. In the above example, the column with the largest column index and the row with the largest row index are set as predetermined boundaries, but it should be understood that this is merely exemplary. Other boundaries can also be set as predetermined boundaries. For example, the column with the smallest column index and the row with the smallest row index can be set as predetermined boundaries.

[0060] by Figure 4This is explained as an example. For ease of explanation, Task A represents the scheduled task executed for data blocks that do not cross the boundary, and Task B represents the scheduled task executed for data blocks that cross the boundary. When the target tensor has boundary 421, Task A is executed for data blocks 401, 402, 405, and 406, and Task B is executed for data blocks 403, 407, 409, 410, and 411. That is, four Tasks A and five Tasks B are required to perform the target operation on the target tensor. When the boundary is between boundary 421 and boundary 422, the scheduled tasks executed do not change.

[0061] If expanded to edit 422, Task A is executed for data blocks 401, 402, 403, 405, 406, and 407, and Task B is executed for data blocks 409, 410, and 411. That is, six Tasks A and three Tasks B are executed. Further, if expanded to boundary 423, Task A is executed for data blocks 401, 402, 403, 405, 406, and 407, and Task B is executed for data blocks 404, 408, 409, 410, 411, and 412. That is, six Tasks A and six Tasks B are executed.

[0062] As can be seen in the above example, when the computation size continues to increase until it exceeds the window size of the primitive task, a new instance of the primitive task is appended. At the same time, the primitive task originally used to process the boundary blocks is changed to process the internal blocks. This allows for efficient processing of target tensors that are larger than the window size.

[0063] Target window size selection

[0064] If the window size used to partition the target tensor is small, the target tensor will be divided into a larger number of data blocks. In this case, the parallelism of data block processing can be improved, but the scheduling and switching between primitive task instances of different data blocks may also cause additional overhead. If the window size used to partition the target tensor is large, the target tensor will be divided into a smaller number of data blocks. In this case, the parallelism is low, and unused computing resources may exist in the processing device 110. In view of this, the determination of the target window size requires a trade-off and optimization based on the size of the processing device 110 and the target tensor.

[0065] On the other hand, as mentioned above Figure 2 As mentioned, in some cases, the tensor sizes to be processed may be diverse or even infinite. Therefore, it is difficult to pre-record each optimized window size for all infinite possible tensor sizes.

[0066] Considering the performance continuity described above, in some implementations, an optimization window size can be pre-determined for multiple sampling tensor sizes. When processing a target tensor, a sampling tensor size that is close to or matches the target tensor size is selected, and the optimization window size corresponding to the selected sampling tensor size is determined as the target window size.

[0067] In some implementations, for each sampling tensor size, the optimized window size can be predetermined in the following manner. Specifically, multiple candidate window sizes can be pre-set, such as 32×32, 64×64, and 128×128. For each candidate window size, it has the ability to implement target operations (e.g., custom elements) for "arbitrary tensor sizes", but the throughput that can be achieved by each may be different. For each candidate window size, its performance at each sampling tensor size is measured, and the throughput that can be achieved by any window size for any sampling tensor size can be obtained. Based on the corresponding throughput, the optimized window size for the sampling tensor size is selected from these candidate window sizes. For example, the candidate window size with the largest throughput is determined as the optimized window size preferred for the specific sampling tensor size.

[0068] refer to Figure 5A Describe an example. Figure 5A Each vertical line in represents a sampled tensor size. Smaller tensor sizes are sampled densely, while larger tensor sizes are sampled sparsely. Sampling can be stopped until the throughput changes relatively slowly or even remains unchanged. Thus, one or more optimization window sizes can be pre-determined for each sampled tensor size. It should be understood that for the same sampled tensor size, the optimization window sizes used for different operations may be different.

[0069] Accordingly, one can construct Figure 5B The mapping table 500 shown can be considered as an example of the mapping table 202. In the mapping table 500, the sample index row indicates the sampling tensor size, and the window size index row indicates the corresponding optimized window size. The mapping table 500 can be used to select the target window size for the target tensor at runtime.

[0070] At runtime, based on the size of the target tensor, a sampling tensor size that is close to or matches the target tensor size is selected from these sampling tensor sizes. By looking up a mapping table such as mapping table 500, the optimized window size for the selected sampling tensor size can be determined. The optimized window size can be determined as the target window size.

[0071] In some implementations, characteristics of the processing device 110 and the target operation can be taken into consideration when selecting the sampling tensor size. To this end, the sampling tensor size can be further selected based on parameters related to the processing device 110 and the target operation. Parameters described herein as being related to or associated with the processing device 110 and the target operation mean that their parameter values ​​depend on the processing device 110 and the target operation.

[0072] Such parameters may include a first parameter α, which indicates the size of the tensor that changes in these sampling tensor sizes corresponding to the sampling interval. For example, the first parameter α may be related to Figure 5A The change from a smaller sampling interval to a larger sampling interval is related to the above. The first parameter α is used to balance the size of the mapping table and the sampling interval. Such parameters may also include a second parameter β, which indicates the maximum sampling tensor size among these sampling tensor sizes. The second parameter β can be determined by evaluating the minimum sampling tensor size at which the peak throughput of the primitive task becomes flat for various window sizes.

[0073] As an example, assuming the target tensor has length x in some dimension, the sampling tensor size can be chosen as follows:

[0074]

[0075] Where ToSample represents the selected sampling tensor size. During runtime, the corresponding first parameter α and second parameter β can be found based on the processing device 110 and the target operation. Then, the selected sampling tensor size is calculated according to equation (1). The corresponding optimized window size is then found according to the mapping table. Thus, the target window size can be determined.

[0076] In this implementation, the sampling tensor size is selected by considering parameters related to the processing device and the target operation, which helps to find the window size that is correctly optimized for the target tensor. In this way, computational efficiency can be ensured, as well as coverage of the best selection in the largest size range.

[0077] Starting a primitive task

[0078] Assume that for the target tensor D, the determined primitive task is t. Then, in order to fully cover the target tensor, the number of instances of the primitive task t that needs to be launched can be determined by the following formula:

[0079]

[0080] Where RequiredTask represents the number of instances of the primitive task t to be launched, N is the dimension size of the target tensor, and D i is the length of the i-th dimension, tw iIt is the window length of the window size corresponding to the primitive task t in the i-th dimension.

[0081] Thus, at runtime, the number of instances of a primitive task can be determined. In some implementations, these instances can be launched together. In some implementations, these instances can be grouped. Instances in each group can be launched together, and different groups can be launched sequentially. Launching instances of primitive tasks in groups helps further improve the scheduling flexibility of tensor computations.

[0082] In some implementations, there may be a pipeline for tensor processing. For example, the result of a target operation performed by processing device 110 on a target tensor may be sent to another processing device for further processing. To this end, the data blocks or instances of primitive tasks into which the target tensor is divided may be divided into multiple groups, such as at least a first group and a second group. The grouping may be performed in any suitable manner, and implementations of the present disclosure are not limited in this respect.

[0083] The execution of a predetermined task for the first set of data blocks can be initiated first, and then the execution results can be asynchronously sent to another processing device downstream in the pipeline. Simultaneously with the asynchronous sending, the execution of a predetermined task for the second set of data blocks can be initiated, and then the execution results of the second set of data blocks can be sent to another processing device. As used herein, "asynchronous sending" means that the sending end sends data to the receiving end and can perform other operations while waiting for the receiving end's response. For example, while the results of the first set of data blocks are being asynchronously sent, processing device 110 can process the second set of data blocks.

[0084] From the perspective of primitive task instances, a first group of instances can be started first, and then the execution results of the first group of instances can be asynchronously sent to another processing device. Then, a second group of instances can be started, and then the execution results of the second group of instances can be asynchronously sent to another processing device.

[0085] In this implementation, the computation tasks and the communication tasks overlap in time, thereby improving the efficiency of pipeline operations.

[0086] Example Process

[0087] Figure 6 A flowchart of a process 500 for processing tensors according to some implementations of the present disclosure is shown. The process 600 may be implemented at a computing system including the processing device 110.

[0088] At block 610 , the computing system determines a target window size based on the size of the target tensor. The target window size is used to perform a target operation associated with the target tensor on the processing device 110 .

[0089] In some implementations, the computing system may select a sampling tensor size from a plurality of sampling tensor sizes based on the size of the target tensor. An optimization window size is predetermined for each of the plurality of sampling tensor sizes. The computing system may determine the predetermined optimization window size for the selected sampling tensor size as the target window size.

[0090] In some implementations, selecting the sampling tensor size from the plurality of sampling tensor sizes is further based on parameters associated with the processing device 110 and the target operation.

[0091] In some implementations, an optimized window size for each of a plurality of sampling tensor sizes is determined by determining, for each of a plurality of candidate window sizes, a throughput by performing a target operation with the candidate window size on a tensor having the sampling tensor size; and selecting, from the plurality of candidate window sizes, an optimized window size for the sampling tensor size based on the throughputs respectively determined for the plurality of candidate window sizes.

[0092] At block 620 , the computing system determines at least one data block based on the target window size and the target tensor, wherein the set of elements included in the at least one data block at least covers the target tensor.

[0093] At block 630, for each data block in the at least one data block, the computing system executes at least one predetermined task from a plurality of predetermined tasks corresponding to a target operation and a target window size based on a position of each element in the data block in the target tensor. The plurality of predetermined tasks are pre-compiled for the processing device 110 to execute the target operation on the tensor having the target window size.

[0094] In some implementations, the computing system may determine whether the data block includes elements that cross a predetermined boundary, the predetermined boundary being associated with the target tensor; and in response to determining that the data block does not include elements that cross the predetermined boundary, perform a first predetermined task of vectorization among multiple predetermined tasks on the data block.

[0095] In some implementations, the computing system may further, in response to determining that the data block includes elements crossing a predetermined boundary, divide the data block into a first sub-block that does not cross the predetermined boundary and a second sub-block that crosses the predetermined boundary; execute a vectorized second predetermined task among multiple predetermined tasks on the first sub-block, wherein the degree of vectorization of the second predetermined task is less than the degree of vectorization of the first predetermined task; and execute a non-vectorized third predetermined task among the multiple predetermined tasks on the second sub-block.

[0096] In some implementations, the predetermined boundary divides the target tensor into different parts. The computing system may select a first predetermined task corresponding to the part where the data block is located from the plurality of predetermined tasks.

[0097] In some implementations, the predetermined bound includes at least one of: a boundary of the target tensor, a bound associated with a distribution of values ​​of elements in the target tensor, or a bound for distinguishing between predetermined operations applied by the target operation to elements in the target tensor.

[0098] In some implementations, the computing system may further divide at least one data block into at least a first group of data blocks and a second group of data blocks; initiate execution of at least one of a plurality of predetermined tasks for the first group of data blocks; initiate asynchronous transmission of corresponding execution results of the first group of data blocks to another processing device 110; simultaneously with the asynchronous transmission, initiate execution of at least one of a plurality of predetermined tasks for the second group of data blocks; and initiate asynchronous transmission of corresponding execution results of the second group of data blocks to another processing device.

[0099] Example device

[0100] Figure 7 Schematic block diagram of an electronic device capable of implementing multiple implementations of the present disclosure is shown. It should be understood that Figure 7 The illustrated electronic device 700 is merely exemplary and should not be construed as limiting the functionality and scope of the implementations described in the present disclosure.

[0101] like Figure 7 As shown, electronic device 700 comprises an electronic device in the form of a general purpose computing device 700. Components of electronic device 700 may include, but are not limited to, one or more processors or processing units 710, memory 720, storage device 730, one or more communication units 740, one or more input devices 750, and one or more output devices 760.

[0102] In some implementations, the electronic device 700 may be implemented as a computing device, a computing system, a server, a mainframe, or other device with computing capabilities.

[0103] The processing unit 710 may be a real or virtual processor and may be capable of executing various processes according to a program stored in the memory 720. In a multi-processor system, multiple processing units execute computer-executable instructions in parallel to improve the parallel processing capabilities of the electronic device 700. The processing unit 710 may include a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor, a controller, and / or a microcontroller, etc. The processing device 110 may be the same as or different from the processing unit 710.

[0104] The electronic device 700 typically includes a plurality of computer storage media. Such media can be any available media accessible to the electronic device 700, including but not limited to volatile and non-volatile media, removable and non-removable media. The memory 720 may include volatile memory (e.g., registers, cache, random access memory (RAM)), non-volatile memory (e.g., read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory) or some combination thereof. The storage device 730 may include removable or non-removable media and may include computer-readable media, such as memory, flash drive, disk or any other media that can be used to store information and / or data and can be accessed within the electronic device 700.

[0105] The electronic device 700 may further include additional removable / non-removable, volatile / non-volatile storage media. Figure 7 As shown in FIG, a magnetic disk drive for reading or writing from a removable, nonvolatile magnetic disk and an optical disk drive for reading or writing from a removable, nonvolatile optical disk can be provided. In these cases, each drive can be connected to a bus (not shown) by one or more data media interfaces.

[0106] The communication unit 740 enables communication with another computing device via a communication medium. Additionally, the functions of the components of the electronic device 700 can be implemented as a single computing cluster or multiple computing machines that can communicate via a communication connection. Thus, the electronic device 700 can operate in a networked environment using logical connections to one or more other servers, personal computers (PCs), or another general network node.

[0107] The input device 750 may be one or more various input devices, such as a mouse, a keyboard, a data import device, etc. The output device 760 may be one or more output devices, such as a display, a data export device, etc. The electronic device 700 may also communicate with one or more external devices (not shown) through the communication unit 740 as needed, such as a storage device, a display device, etc., with one or more devices that allow a user to interact with the electronic device 700, or with any device that allows the electronic device 700 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface (not shown).

[0108] In some implementations, in addition to being integrated on a single device, some or all of the various components of the electronic device 700 may also be arranged in the form of a cloud computing architecture. In a cloud computing architecture, these components can be remotely located and can work together to implement the functionality described in this disclosure. In some implementations, cloud computing provides computing, software, data access, and storage services that do not require the end user to know the physical location or configuration of the systems or hardware providing these services. In various implementations, cloud computing uses appropriate protocols to provide services over a wide area network (such as the Internet). For example, a cloud computing provider provides applications over a wide area network, and they can be accessed through a web browser or any other computing component. The software or components of the cloud computing architecture and the corresponding data can be stored on servers at a remote location. The computing resources in a cloud computing environment can be consolidated at a remote data center location or they can be dispersed. The cloud computing infrastructure can provide services through a shared data center, even if they appear to be a single access point for users. Therefore, the components and functionality described herein can be provided from a service provider at a remote location using a cloud computing architecture. Alternatively, they can be provided from a conventional server, or they can be installed directly or otherwise on the client device.

[0109] The electronic device 700 may be used to implement various implementations of the present disclosure. The memory 720 may include one or more modules having one or more program instructions that may be accessed and executed by the processing unit 610 to implement the functions of the various implementations described herein. For example, the memory 720 may include a tensor processing module 722 for processing target operations associated with target tensors. Figure 7 As shown, the electronic device 700 can obtain input required for tensor processing through the input device 750 and provide tensor processed output through the output device 760. In some implementations, the electronic device 700 can also receive input from other devices (not shown) via the communication unit 740.

[0110] Example Implementation

[0111] Some example implementations of the present disclosure are listed below.

[0112] In one aspect, the present disclosure provides a computer-implemented method, comprising: determining a target window size based on a size of a target tensor, the target window size being used to perform a target operation associated with the target tensor on a processing device; determining at least one data block based on the target window size and the target tensor, the set of elements included in the at least one data block covering at least the target tensor; and performing, for each data block in the at least one data block, at least one predetermined task from a plurality of predetermined tasks corresponding to the target operation and the target window size based on positions of respective elements in the data block in the target tensor, the plurality of predetermined tasks being pre-compiled for the processing device to perform the target operation on a tensor having the target window size.

[0113] In some example implementations, performing at least one of the plurality of predetermined tasks includes: determining whether the data block includes elements that cross a predetermined boundary, the predetermined boundary being associated with a target tensor; and in response to determining that the data block does not include elements that cross the predetermined boundary, performing a vectorized first predetermined task of the plurality of predetermined tasks for the data block.

[0114] In some example implementations, the method further includes: in response to determining that the data block includes elements that cross the predetermined boundary, dividing the data block into a first sub-block that does not cross the predetermined boundary and a second sub-block that crosses the predetermined boundary; executing a vectorized second predetermined task among the multiple predetermined tasks on the first sub-block, wherein the degree of vectorization of the second predetermined task is less than the degree of vectorization of the first predetermined task; and executing a non-vectorized third predetermined task among the multiple predetermined tasks on the second sub-block.

[0115] In some example implementations, the predetermined boundary divides the target tensor into different parts, and the method further includes: selecting the first predetermined task corresponding to the part where the data block is located from the plurality of predetermined tasks.

[0116] In some example implementations, the predetermined bound includes at least one of: a boundary of the target tensor, a bound associated with a distribution of values ​​of elements in the target tensor, or a bound for distinguishing between predetermined operations applied by the target operation to the elements in the target tensor.

[0117] In some example implementations, the method further includes: dividing the at least one data block into at least a first group of data blocks and a second group of data blocks; initiating execution of at least one of the plurality of predetermined tasks for the first group of data blocks respectively; initiating sending corresponding execution results of the first group of data blocks to another processing device; simultaneously with the asynchronous sending, initiating execution of at least one of the plurality of predetermined tasks for the second group of data blocks respectively; and initiating asynchronous sending of the corresponding execution results of the second group of data blocks to another processing device.

[0118] In some example implementations, determining the target window size includes: selecting a sampling tensor size from a plurality of sampling tensor sizes based on the size of the target tensor, wherein optimized window sizes are respectively predetermined for the plurality of sampling tensor sizes; and determining the optimized window size predetermined for the selected sampling tensor size as the target window size.

[0119] In some example implementations, selecting the sampling tensor size from the plurality of sampling tensor sizes is further based on at least one of the following parameters associated with the processing device and the target operation: a first parameter indicating a tensor size of the plurality of sampling tensor sizes that changes corresponding to a sampling interval, or a second parameter indicating a maximum tensor size of the plurality of sampling tensor sizes.

[0120] In some example implementations, the optimized window size for each of the multiple sampling tensor sizes is determined as follows: for each candidate window size from a plurality of candidate window sizes, determining a throughput by performing the target operation with the candidate window size on a tensor having the sampling tensor size; and selecting the optimized window size for the sampling tensor size from the plurality of candidate window sizes based on the throughputs respectively determined for the plurality of candidate window sizes.

[0121] In another aspect, the present disclosure provides an electronic device. The electronic device includes: a processor; and a memory coupled to the processor and containing instructions stored thereon, wherein when the instructions are executed by the processor, the device performs the following actions, the actions including: determining a target window size based on a size of a target tensor, the target window size being used to perform a target operation associated with the target tensor on a processing device; determining at least one data block based on the target window size and the target tensor, the set of elements included in the at least one data block covering at least the target tensor; and for each data block in the at least one data block, performing at least one predetermined task of a plurality of predetermined tasks corresponding to the target operation and the target window size based on positions of respective elements in the data block in the target tensor, the plurality of predetermined tasks being pre-compiled for the processing device to perform the target operation on a tensor having the target window size.

[0122] In some example implementations, executing at least one of the plurality of predetermined tasks includes: determining whether the data block includes elements that cross a predetermined boundary, the predetermined boundary being associated with the target tensor; and in response to determining that the data block does not include elements that cross the predetermined boundary, executing a vectorized first predetermined task of the plurality of predetermined tasks for the data block.

[0123] In some example implementations, the action further includes: in response to determining that the data block includes elements that cross the predetermined boundary, dividing the data block into a first sub-block that does not cross the predetermined boundary and a second sub-block that crosses the predetermined boundary; executing a vectorized second predetermined task among the multiple predetermined tasks on the first sub-block, wherein the degree of vectorization of the second predetermined task is less than the degree of vectorization of the first predetermined task; and executing a non-vectorized third predetermined task among the multiple predetermined tasks on the second sub-block.

[0124] In some example implementations, the predetermined bounds divide the target tensor into different parts, and the actions further include: selecting the first predetermined task corresponding to the part where the data block is located from the plurality of predetermined tasks.

[0125] In some example implementations, the predetermined bound includes at least one of: a boundary of the target tensor, a bound associated with a distribution of values ​​of elements in the target tensor, or a bound for distinguishing between predetermined operations applied by the target operation to the elements in the target tensor.

[0126] In some example implementations, the action further includes: dividing the at least one data block into at least a first group of data blocks and a second group of data blocks; initiating execution of at least one of the multiple predetermined tasks for the first group of data blocks respectively; initiating sending corresponding execution results of the first group of data blocks to another processing device; simultaneously with the asynchronous sending, initiating execution of at least one of the multiple predetermined tasks for the second group of data blocks respectively; and initiating asynchronous sending of the corresponding execution results of the second group of data blocks to another processing device.

[0127] In some example implementations, determining the target window size includes: selecting a sampling tensor size from a plurality of sampling tensor sizes based on the size of the target tensor, wherein optimized window sizes are respectively predetermined for the plurality of sampling tensor sizes; and determining the optimized window size predetermined for the selected sampling tensor size as the target window size.

[0128] In some example implementations, selecting the sampling tensor size from the plurality of sampling tensor sizes is further based on at least one of the following parameters associated with the processing device and the target operation: a first parameter indicating a tensor size of the plurality of sampling tensor sizes that changes corresponding to a sampling interval, or a second parameter indicating a maximum tensor size of the plurality of sampling tensor sizes.

[0129] In some example implementations, the optimized window size for each of the multiple sampling tensor sizes is determined as follows: for each candidate window size from a plurality of candidate window sizes, determining a throughput by performing the target operation with the candidate window size on a tensor having the sampling tensor size; and selecting the optimized window size for the sampling tensor size from the plurality of candidate window sizes based on the throughputs respectively determined for the plurality of candidate window sizes.

[0130] On the other hand, the present disclosure provides a computer program product, which is tangibly stored in a computer storage medium and includes computer-executable instructions. When the computer-executable instructions are executed by a device, the device performs the following actions, the actions including: determining a target window size based on the size of a target tensor, the target window size being used to perform a target operation associated with the target tensor on a processing device; determining at least one data block based on the target window size and the target tensor, the set of elements included in the at least one data block covering at least the target tensor; and for each data block in the at least one data block, based on the position of each element in the data block in the target tensor, performing at least one predetermined task of a plurality of predetermined tasks corresponding to the target operation and the target window size, the plurality of predetermined tasks being pre-compiled for the processing device to perform the target operation on a tensor having the target window size.

[0131] In some example implementations, executing at least one of the plurality of predetermined tasks includes: determining whether the data block includes elements that cross a predetermined boundary, the predetermined boundary being associated with the target tensor; and in response to determining that the data block does not include elements that cross the predetermined boundary, executing a vectorized first predetermined task of the plurality of predetermined tasks for the data block.

[0132] In some example implementations, the action further includes: in response to determining that the data block includes elements that cross the predetermined boundary, dividing the data block into a first sub-block that does not cross the predetermined boundary and a second sub-block that crosses the predetermined boundary; executing a vectorized second predetermined task among the multiple predetermined tasks on the first sub-block, wherein the degree of vectorization of the second predetermined task is less than the degree of vectorization of the first predetermined task; and executing a non-vectorized third predetermined task among the multiple predetermined tasks on the second sub-block.

[0133] In some example implementations, the predetermined bounds divide the target tensor into different parts, and the actions further include: selecting the first predetermined task corresponding to the part where the data block is located from the plurality of predetermined tasks.

[0134] In some example implementations, the predetermined bound includes at least one of: a boundary of the target tensor, a bound associated with a distribution of values ​​of elements in the target tensor, or a bound for distinguishing between predetermined operations applied by the target operation to the elements in the target tensor.

[0135] In some example implementations, the action further includes: dividing the at least one data block into at least a first group of data blocks and a second group of data blocks; initiating execution of at least one of the multiple predetermined tasks for the first group of data blocks respectively; initiating sending corresponding execution results of the first group of data blocks to another processing device; simultaneously with the asynchronous sending, initiating execution of at least one of the multiple predetermined tasks for the second group of data blocks respectively; and initiating asynchronous sending of the corresponding execution results of the second group of data blocks to another processing device.

[0136] In some example implementations, determining the target window size includes: selecting a sampling tensor size from a plurality of sampling tensor sizes based on the size of the target tensor, wherein optimized window sizes are respectively predetermined for the plurality of sampling tensor sizes; and determining the optimized window size predetermined for the selected sampling tensor size as the target window size.

[0137] In some example implementations, selecting the sampling tensor size from the plurality of sampling tensor sizes is further based on at least one of the following parameters associated with the processing device and the target operation: a first parameter indicating a tensor size of the plurality of sampling tensor sizes that changes corresponding to a sampling interval, or a second parameter indicating a maximum tensor size of the plurality of sampling tensor sizes.

[0138] In some example implementations, the optimized window size for each of the multiple sampling tensor sizes is determined as follows: for each candidate window size from a plurality of candidate window sizes, determining a throughput by performing the target operation with the candidate window size on a tensor having the sampling tensor size; and selecting the optimized window size for the sampling tensor size from the plurality of candidate window sizes based on the throughputs respectively determined for the plurality of candidate window sizes.

[0139] In yet another aspect, the present disclosure provides a computer-readable medium having computer-executable instructions stored thereon. When the computer-executable instructions are executed by a device, the device performs one or more example implementations of the method of the above aspects.

[0140] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, example types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), load programmable logic devices (CPLDs), and the like.

[0141] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0142] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0143] In addition, although each operation is described in a specific order, this should be understood as requiring such operation to be performed in the specific order shown or in a sequential order, or requiring that all illustrated operations should be performed to obtain the desired result. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate implementation can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination mode.

[0144] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A computer-implemented method, include: Determining a target window size based on a size of a target tensor, the target window size being used to perform a target operation associated with the target tensor on a processing device; Based on the target window size and the target tensor, determine at least one data block, where a set of elements included in the at least one data block at least covers the target tensor; as well as For each data block of the at least one data block, based on the position of each element in the data block in the target tensor, at least one predetermined task of a plurality of predetermined tasks corresponding to the target operation and the target window size is performed, and the plurality of predetermined tasks are pre-compiled for the processing device to perform the target operation on the tensor having the target window size.

2. The method according to claim 1, wherein performing at least one of the plurality of predetermined tasks include: determining whether the data block includes an element that crosses a predetermined limit, the predetermined limit being associated with the target tensor; as well as In response to determining that the data block does not include an element crossing the predetermined boundary, a vectorized first predetermined task of the plurality of predetermined tasks is executed for the data block.

3. According to the method of claim 2, further include: In response to determining that the data block includes an element that crosses the predetermined boundary, dividing the data block into a first sub-block that does not cross the predetermined boundary and a second sub-block that crosses the predetermined boundary; executing a vectorized second predetermined task among the plurality of predetermined tasks for the first sub-block, wherein the vectorization degree of the second predetermined task is less than the vectorization degree of the first predetermined task; as well as A non-vectorized third predetermined task among the plurality of predetermined tasks is executed for the second sub-block.

4. The method according to claim 2, wherein the predetermined boundary divides the target tensor into different parts, and the method further include: The first predetermined task corresponding to the portion where the data block is located is selected from the plurality of predetermined tasks.

5. The method according to claim 2, wherein the predetermined limit comprises at least one of the following: The bounds of the target tensor, bounds associated with the distribution of values ​​of elements in the target tensor, or Bounds that distinguish between predetermined operations that the target operation applies to the elements in the target tensor.

6. The method according to claim 1, further comprising: include: dividing the at least one data block into at least a first group of data blocks and a second group of data blocks; Initiate execution of at least one of the plurality of predetermined tasks for the first group of data blocks respectively; Initiate asynchronous transmission of corresponding execution results of the first group of data blocks to another processing device; Simultaneously with the asynchronous sending, starting to execute at least one of the plurality of predetermined tasks respectively for the second group of data blocks; as well as Initiate asynchronously sending corresponding execution results of the second group of data blocks to the another processing device.

7. The method according to claim 1, wherein determining the target window size include: Based on the size of the target tensor, selecting a sampling tensor size from a plurality of sampling tensor sizes, wherein optimization window sizes are predetermined for each of the plurality of sampling tensor sizes; as well as An optimized window size predetermined for the selected sampling tensor size is determined as the target window size.

8. The method of claim 7, wherein selecting the sampling tensor size from the plurality of sampling tensor sizes is further based on at least one of the following parameters associated with the processing device and the target operation: a first parameter indicating a tensor size of the plurality of sampling tensor sizes that changes corresponding to a sampling interval, or A second parameter indicating a maximum tensor size among the plurality of sampling tensor sizes.

9. The method of claim 7, wherein the optimized window size for each of the plurality of sampling tensor sizes is determined as follows: For each candidate window size of a plurality of candidate window sizes, determining a throughput by performing the target operation on a tensor having the sampled tensor size with the candidate window size; and Based on the throughputs respectively determined for the multiple candidate window sizes, an optimized window size for the sampling tensor size is selected from the multiple candidate window sizes.

10. An electronic device, include: Processing unit; as well as a memory coupled to the processing unit and containing instructions stored thereon, the instructions, when executed by the processing unit, causing the device to perform the following actions, the actions comprising: Determining a target window size based on a size of a target tensor, the target window size being used to perform a target operation associated with the target tensor on a processing device; Based on the target window size and the target tensor, determining at least one data block, wherein a set of elements included in the at least one data block at least covers the target tensor; and For each data block of the at least one data block, based on the position of each element in the data block in the target tensor, at least one predetermined task of a plurality of predetermined tasks corresponding to the target operation and the target window size is performed, and the plurality of predetermined tasks are pre-compiled for the processing device to perform the target operation on the tensor having the target window size.

11. The apparatus according to claim 10, wherein performing at least one of the plurality of predetermined tasks include: determining whether the data block includes an element that crosses a predetermined limit, the predetermined limit being associated with the target tensor; as well as In response to determining that the data block does not include an element crossing the predetermined boundary, a vectorized first predetermined task of the plurality of predetermined tasks is executed for the data block.

12. The apparatus according to claim 11, wherein said action further include: In response to determining that the data block includes an element that crosses the predetermined boundary, dividing the data block into a first sub-block that does not cross the predetermined boundary and a second sub-block that crosses the predetermined boundary; executing a vectorized second predetermined task among the plurality of predetermined tasks for the first sub-block, wherein the vectorization degree of the second predetermined task is less than the vectorization degree of the first predetermined task; as well as A non-vectorized third predetermined task among the plurality of predetermined tasks is executed for the second sub-block.

13. The apparatus according to claim 11, wherein the predetermined boundary divides the target tensor into different parts, and the action further include: The first predetermined task corresponding to the portion where the data block is located is selected from the plurality of predetermined tasks.

14. The apparatus according to claim 11, wherein the predetermined limit comprises at least one of: The bounds of the target tensor, bounds associated with the distribution of values ​​of elements in the target tensor, or Bounds that distinguish between predetermined operations that the target operation applies to the elements in the target tensor.

15. The apparatus according to claim 10, wherein said action further include: dividing the at least one data block into at least a first group of data blocks and a second group of data blocks; Initiate execution of at least one of the plurality of predetermined tasks for the first group of data blocks respectively; as well as Initiate asynchronous transmission of corresponding execution results of the first group of data blocks to another processing device; Simultaneously with the asynchronous sending, starting to execute at least one of the plurality of predetermined tasks respectively for the second group of data blocks; as well as Initiate asynchronously sending corresponding execution results of the second group of data blocks to the another processing device.

16. The device according to claim 10, wherein determining the target window size include: Based on the size of the target tensor, selecting a sampling tensor size from a plurality of sampling tensor sizes, wherein optimization window sizes are predetermined for each of the plurality of sampling tensor sizes; as well as An optimized window size predetermined for the selected sampling tensor size is determined as the target window size.

17. The apparatus of claim 16, wherein selecting the sampling tensor size from the plurality of sampling tensor sizes is further based on at least one of the following parameters associated with the processing device and the target operation: a first parameter indicating a tensor size of the plurality of sampling tensor sizes that changes corresponding to a sampling interval, or A second parameter indicating a maximum tensor size among the plurality of sampling tensor sizes.

18. The apparatus of claim 16, wherein the optimized window size for each of the plurality of sampling tensor sizes is determined as follows: For each candidate window size of a plurality of candidate window sizes, determining a throughput by performing the target operation on a tensor having the sampled tensor size with the candidate window size; and Based on the throughputs respectively determined for the multiple candidate window sizes, an optimized window size for the sampling tensor size is selected from the multiple candidate window sizes.

19. A computer program product, the computer program product being tangibly stored in a computer storage medium and comprising computer executable instructions which, when executed by a device, cause the device to perform the following actions: include: Determining a target window size based on a size of a target tensor, the target window size being used to perform a target operation associated with the target tensor on a processing device; Based on the target window size and the target tensor, determine at least one data block, where a set of elements included in the at least one data block at least covers the target tensor; as well as For each data block of the at least one data block, based on the position of each element in the data block in the target tensor, at least one predetermined task of a plurality of predetermined tasks corresponding to the target operation and the target window size is performed, and the plurality of predetermined tasks are pre-compiled for the processing device to perform the target operation on the tensor having the target window size.

20. The method according to claim 19, wherein performing at least one of the plurality of predetermined tasks include: determining whether the data block includes an element that crosses a predetermined limit, the predetermined limit being associated with the target tensor; as well as In response to determining that the data block does not include an element crossing the predetermined boundary, a vectorized first predetermined task of the plurality of predetermined tasks is executed for the data block.

Images