Tensor processing methods and apparatus, electronic devices and storage media

Abstract

This disclosure provides a tensor processing method, apparatus, electronic device, and storage medium. The tensor processing method is used in an artificial intelligence processor and includes: acquiring a predefined first logical layout for tensor processing in the artificial intelligence processor; receiving multiple tensors and physical layout information of the multiple tensors, wherein the physical layout information includes the physical layout of storage units in the artificial intelligence processor used to store the tensors; generating a mapping descriptor corresponding to each of the multiple tensors based on the first logical layout and the physical layout information of the multiple tensors, wherein the mapping descriptor is used for mapping between the physical address of the tensor and the logical coordinates of the first logical layout; and performing data transfer and processing logic for the multiple tensors based on the mapping descriptor. This method can realize the mapping from the physical layout of multiple tensors to a unified logical layout.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence, and embodiments of this disclosure relate to tensor processing methods and apparatus, electronic devices, and storage media. Background Technology

[0002] Tensors are widely used as a fundamental data structure for carrying multidimensional data in artificial intelligence, scientific computing, and high-performance computing. Tensors can include the specific values ​​(shapes) of each dimension of the data used in artificial intelligence, scientific computing, and high-performance computing, as well as the arrangement order of the elements of each dimension in storage units (such as memory or registers) (e.g., physical layout).

[0003] For example, in mainstream Large Language Models (LLMs), the attention mechanism is one of the core computational modules, and its performance directly affects the efficiency of model training and inference. Attention operators typically involve three input tensors (query, key, and value) and an output tensor. These tensors may have different tensor layouts in storage units depending on the model architecture, hardware platform, or optimization strategy, such as BHSD, BSHD, and SBHD, where B represents the number of tensors, S represents the sequence length, H represents the number of attention heads, and D represents the dimension of each head.

[0004] To adapt to different tensor layouts, developers need to write and optimize corresponding operator implementations separately for each tensor layout. This leads to a surge in development workload, significantly increases code maintenance costs, and reduces development efficiency. Summary of the Invention

[0005] At least one embodiment of this disclosure provides a tensor processing method and apparatus, electronic device, and storage medium to address the above-mentioned problems.

[0006] According to at least one embodiment of this disclosure, a tensor processing method is provided for an artificial intelligence processor. The tensor processing method includes: obtaining a predefined first logical layout for tensor processing in the artificial intelligence processor; receiving a plurality of tensors and physical layout information of the tensors, wherein the physical layout information includes the physical layout of storage units in the artificial intelligence processor for storing tensors; generating a mapping descriptor corresponding to each of the plurality of tensors based on the first logical layout and the physical layout information of the plurality of tensors, wherein the mapping descriptor is used for mapping between the physical address of the tensor and the logical coordinates of the first logical layout; and performing data transfer and processing logic for the plurality of tensors based on the mapping descriptor.

[0007] According to at least one embodiment of this disclosure, multiple tensors have different physical layout information.

[0008] According to at least one embodiment of the present disclosure, the physical layout of the storage cells includes at least one of a memory matrix physical layout and a register matrix physical layout.

[0009] According to at least one embodiment of this disclosure, the physical layout of the register matrix includes a register index layout and a thread layout.

[0010] According to at least one embodiment of this disclosure, the first logical layout, the memory matrix physical layout, the register index layout, and the thread layout each include a dimension vector and a step vector of the layout.

[0011] According to at least one embodiment of the present disclosure, the dimension vector represents the dimensional information of the layout, and the step vector represents the step size of each dimension.

[0012] According to at least one embodiment of the present disclosure, generating a mapping descriptor corresponding to each of the plurality of tensors based on a first logical layout and physical layout information of the plurality of tensors includes: generating a mapping descriptor corresponding to each of the plurality of tensors based on the first logical layout, the shape and size of the tensor, the physical layout of the tensor, the starting address pointer of the tensor, and the data type of the tensor.

[0013] According to at least one embodiment of the present disclosure, generating a mapping descriptor corresponding to each of the plurality of tensors based on a first logical layout and physical layout information of the plurality of tensors includes: using a driver interface to generate a mapping descriptor corresponding to each of the plurality of tensors based on a first logical layout and physical layout information of the plurality of tensors.

[0014] According to at least one embodiment of the present disclosure, the mapping descriptor describes the mapping relationship between the physical address of a tensor and the logical coordinates of a first logical layout.

[0015] According to at least one embodiment of this disclosure, the method further includes: passing a mapping descriptor as a parameter to an artificial intelligence processor; and having the artificial intelligence processor perform a mapping between the physical address of a tensor and the logical coordinates of a first logical layout based on the mapping descriptor.

[0016] According to at least one embodiment of the present disclosure, the mapping between the physical address of a tensor and the logical coordinates of a first logical layout is performed by an artificial intelligence processor based on a mapping descriptor, including: during data loading, the mapping from the physical address of an input tensor to the logical coordinates of the first logical layout is performed by the artificial intelligence processor based on a mapping descriptor; and during data output, the mapping from the logical coordinates of the first logical layout to the physical address of an output tensor is performed by the artificial intelligence processor based on a mapping descriptor.

[0017] According to at least one embodiment of the present disclosure, the mapping between the physical address of a tensor and the logical coordinates of a first logical layout is performed by an artificial intelligence processor based on a mapping descriptor, which includes: performing the mapping between the physical address of a tensor and the logical coordinates of a first logical layout by executing instructions corresponding to the mapping descriptor by the artificial intelligence processor.

[0018] According to at least one embodiment of the present disclosure, a first logical layout is used by an attention operator to perform attention computation logic on a plurality of tensors.

[0019] According to at least one embodiment of this disclosure, a tensor processing apparatus is provided for an artificial intelligence processor, wherein the tensor processing apparatus includes: an acquisition module configured to acquire a predefined first logical layout for tensor processing in the artificial intelligence processor; a receiving module configured to receive a plurality of tensors and physical layout information of the plurality of tensors, wherein the physical layout information includes a physical layout of storage units in the artificial intelligence processor for storing tensors; a generation module configured to generate a mapping descriptor corresponding to each of the plurality of tensors based on the first logical layout and the physical layout information of the plurality of tensors, wherein the mapping descriptor is used for mapping between the physical address of the tensor and the logical coordinates of the first logical layout; and an execution module configured to execute data transfer and processing logic for the plurality of tensors based on the mapping descriptor.

[0020] According to at least one embodiment of the present disclosure, a tensor processing electronic device is provided, wherein the electronic device includes: one or more processors; and a memory storing one or more computer programs; wherein, when executed by the one or more processors, the one or more computer programs cause the one or more processors to implement the method according to any one of the at least one embodiment of the present disclosure.

[0021] According to at least one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein computer-executable instructions are stored thereon, which, when executed by a processor, cause at least one processor to implement the method according to any one of the at least one embodiment of the present disclosure.

[0022] The tensor processing method, apparatus, electronic device, and storage medium according to at least one embodiment of the present disclosure can realize the mapping of the physical layout of multiple tensors to a unified logical layout. Attached Figure Description

[0023] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0024] Figure 1 A schematic structural diagram of a general-purpose graphics processor (GPGPU) according to at least one embodiment of the present disclosure is shown;

[0025] Figure 2 A tensor processing method according to at least one embodiment of the present disclosure is illustrated;

[0026] Figure 3 A tensor processing procedure according to at least one embodiment of the present disclosure is illustrated;

[0027] Figure 4 An example of a tensor processing procedure used by an attention operator for a first logical layout according to at least one embodiment of the present disclosure is shown;

[0028] Figure 5 A tensor processing apparatus according to at least one embodiment of the present disclosure is shown;

[0029] Figure 6 An electronic device according to at least one embodiment of the present disclosure is shown;

[0030] Figure 7 A non-transitory computer-readable storage medium according to at least one embodiment of the present disclosure is shown. Detailed Implementation

[0031] The following description, with reference to the accompanying drawings, is provided to aid in a thorough understanding of the various embodiments of this disclosure as defined by the claims and their equivalents. This description includes various specific details to aid understanding but should be considered exemplary only. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the various embodiments described herein without departing from the scope and spirit of this disclosure. Furthermore, for clarity and brevity, descriptions of well-known functions and structures may be omitted.

[0032] The terms and wording used in the following description and claims are not limited to their dictionary meanings, but are merely intended by the discloser to enable a clear and consistent understanding of this disclosure. Therefore, it will be apparent to those skilled in the art that the following description of various embodiments of this disclosure is for illustrative purposes only and not for limiting the purpose of this disclosure as defined in the appended claims and their equivalents.

[0033] It should be understood that the singular forms of “one,” “an,” and “the” include plural references unless the context clearly indicates otherwise. Thus, for example, the reference to “component surface” includes one or more such surfaces.

[0034] The terms “comprising” or “may include” refer to the presence of a corresponding disclosed function, operation, or component that may be used in the various embodiments of this disclosure, rather than limiting the presence of one or more additional functions, operations, or features. Furthermore, the terms “comprising” or “having” may be interpreted as indicating certain characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof, but should not be construed as excluding the possibility of the presence of one or more other characteristics, numbers, steps, operations, constituent elements, components, or combinations thereof.

[0035] The term "or" as used in the various embodiments of this disclosure includes any of the listed terms and all combinations thereof. For example, "A or B" may include A, may include B, or may include both A and B.

[0036] Unless otherwise defined, all terms used in this disclosure (including technical or scientific terms) have the same meaning as understood by one of those skilled in the art as described herein. Common terms as defined in dictionaries are to be interpreted as having a meaning consistent with the context in the relevant technical field and should not be interpreted ideally or overly formally unless expressly defined in this disclosure.

[0037] Throughout the embodiments described herein, the same reference numerals consistently denote the same elements. In this respect, the embodiments may take different forms and should not be construed as limited to the description set forth herein. Therefore, the embodiments are described below only with reference to the accompanying drawings to explain various aspects. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. Expressions such as "at least one of..." modify the entire list of elements when preceding it, but do not modify individual elements in the list. In the drawings, the same reference numerals denote the same elements, and for clarity and ease of explanation, the dimensions of components in the drawings may be exaggerated.

[0038] While terms such as "first" and "second" can be used to describe various components, these terms are only used to distinguish one component from another. These terms are not intended to specify that the components are made of different materials or have different structures.

[0039] A singular expression may encompass a plural expression unless it has a distinctly different meaning in the context. When a part "includes" a component, it may further include another component, rather than excluding the other component, unless otherwise stated.

[0040] Furthermore, the terms "part," "module," etc., refer to a unit that processes at least one function or operation, and can be implemented by hardware, software, or a combination thereof.

[0041] Tensors are widely used as a fundamental data structure for carrying multidimensional data in artificial intelligence, scientific computing, and high-performance computing. Tensors can include the specific values ​​(shapes) of each dimension of the data used in artificial intelligence, scientific computing, and high-performance computing, as well as the arrangement order of the elements of each dimension in storage units (such as memory or registers) (e.g., physical layout).

[0042] In practical software stacks and hardware platforms, the diversity of tensor physical layouts comes from a wide range of sources. Different deep learning frameworks or their internal optimizers may use or generate different layouts by default; different hardware accelerators (such as general-purpose graphics processing units (GPGPUs), neural network processing units (NPUs), tensor processing units (TPUs), etc.) may have better access efficiency for specific layouts due to their memory unit architecture and computation unit design; in addition, the physical layout of tensors may be dynamically changed in optimization processes such as model compilation, operator fusion, and cross-platform deployment in pursuit of ultimate performance.

[0043] As mentioned above, in mainstream Large Language Models (LLMs), the attention mechanism is one of the core computational modules, and its performance directly affects the efficiency of model training and inference. Attention operators typically involve three input tensors (Query, Key, and Value) and an output tensor. These tensors may have different tensor layouts in storage units due to differences in model architecture, hardware platform, or optimization strategies, such as BHSD, BSHD, and SBHD. Here, B represents the number of tensors, S represents the sequence length, H represents the number of attention heads, and D represents the dimension of each head.

[0044] To adapt to different tensor layouts, developers need to write and optimize the corresponding operator implementations separately for each tensor layout, and perform task splitting and optimization separately. This results in extremely high development costs: for each new layout, the operator logic and optimization methods need to be completely rewritten, and the manpower input increases linearly or even exponentially; maintenance is difficult: when the algorithm logic is updated, all operator versions need to be modified simultaneously, which can easily introduce inconsistencies or omissions; code redundancy is serious: more than 90% of the calculation logic is essentially the same, and only needs to be implemented repeatedly due to different indexing methods.

[0045] Although some deep learning frameworks provide automatic transpose or layout transformation interfaces, such transformations rely on explicit transpose at the deep learning framework layer. Before calling the general operator, the input tensor is first transposed to the target layout. Transpose operations typically require allocating a temporary buffer and performing a full data copy. In large batch or long sequence scenarios, storage consumption increases significantly, and data handling becomes a bottleneck, leading to overall performance degradation.

[0046] The tensor processing method, apparatus, electronic device, and storage medium provided according to at least one embodiment of this disclosure can realize the mapping of the physical layout of multiple tensors to a unified logical layout. The tensor processing method can realize the mapping of the physical layout of multiple tensors to a unified logical layout, so that the upper-level operator logic only needs to be developed for a unified logical layout, without the need to redevelop new operators, thereby achieving transparent support for multiple physical layouts and achieving relatively superior performance.

[0047] The tensor processing method, apparatus, electronic device, and storage medium according to at least one embodiment of the present disclosure are described in detail below with reference to the accompanying drawings.

[0048] Figure 1 A schematic structural diagram of a general-purpose graphics processor (GPGPU) according to at least one embodiment of the present disclosure is shown.

[0049] like Figure 1 As shown, a general-purpose graphics processor is actually an array of programmable multiprocessors. For example, a programmable multiprocessor can be a streaming processor cluster (SPC), such as including... Figure 1 The diagram shows streaming processor clusters 1, ..., M, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster handles one computational task, or multiple streaming processor clusters handle one computational task. Multiple streaming processor clusters share data through a global cache or global memory.

[0050] like Figure 1 As shown, taking streaming processor cluster 1 as an example, one streaming processor cluster includes multiple computing units, such as... Figure 1 The system is structured as Computation Unit 1, Computation Unit 2, ..., Computation Unit N, where N is a positive integer. Each Computation Unit (CU) performs arithmetic and logical operations, such as accumulation, reduction, and standard addition, subtraction, multiplication, and division. A Computation Unit includes multiple cores (also called computational kernels), each of which includes an Arithmetic Logic Unit (ALU), a floating-point unit, etc. These cores are used to execute specific computational tasks. Furthermore, the Computation Unit also includes registers (e.g., ...). Figure 1 The register file and shared memory in a computing unit are used to store source and destination data related to computing tasks in a hierarchical manner. The shared memory in a computing unit is used to share data between the cores of that computing unit.

[0051] like Figure 1 As shown, each computing unit also provides a tensor core for performing tensor-related computations, such as tensor shrinking operations. Tensor cores can accelerate tensor operations such as matrix multiplication. Tensor cores in multiple computing units can be scheduled and controlled uniformly.

[0052] like Figure 1 As shown, each streaming processor cluster also provides a buffer for caching data across the N computing units within the streaming processor cluster.

[0053] In parallel computing, computational tasks are typically executed by multiple threads. These threads are divided into multiple thread blocks before execution in a general-purpose graphics processor (or parallel computing processor), and then dispatched via a thread block distribution module. Figure 1 (Not shown in the image) Multiple thread blocks are distributed to various computation units. All threads in a thread block must be assigned to the same computation unit for execution. Simultaneously, thread blocks are broken down into minimum execution thread bundles (or simply warps), each containing a fixed number (or less than this fixed number) of threads, for example, 32 threads. Multiple thread blocks can execute in the same computation unit or in different computation units.

[0054] In each computing unit, the thread beam scheduling / distribution module ( Figure 1 (Not shown in the diagram) Thread bundles are scheduled and allocated so that multiple computing cores within the computing unit can run thread bundles. Depending on the number of computing cores in the computing unit, multiple thread bundles within a thread block can execute concurrently or in a time-sharing manner. Multiple threads within each thread bundle execute the same instructions. Memory-executed instructions are issued to shared memory within the computing unit or further issued to intermediate-level caches, global caches, or global memory (e.g., [example cache]). Figure 1High Bandwidth Memory (HBM) is used for read and write operations.

[0055] Figure 2 A tensor processing method 200 according to at least one embodiment of the present disclosure is illustrated. For example... Figure 2 As shown, tensor processing method 200 includes steps 202-208.

[0056] refer to Figure 2 According to at least one embodiment of the present disclosure, in step 202, the tensor processing method 200 may include obtaining a predefined first logical layout for tensor processing in an artificial intelligence processor.

[0057] According to at least one embodiment of this disclosure, the predefined first logical layout can be a user-predefined first logical layout for processing multiple tensors. For example, the predefined first logical layout for tensor processing can be obtained based on user input.

[0058] According to at least one embodiment of this disclosure, a first logical layout may correspond to a unified logical layout for tensor processing, which may be independent of the physical layout of multiple tensors. For example, multiple tensors may be mapped to a unified logical layout, and transport and processing logic for the multiple tensors may be performed based on the unified logical layout. For example, according to at least one embodiment of this disclosure, various operators for multiple tensors may be performed based on the unified logical layout.

[0059] According to at least one embodiment of this disclosure, the tensor processing method can be executed by an artificial intelligence processor. For example, the artificial intelligence processor may include a general-purpose graphics processing unit (GPGPU), a neural network processing unit (NPU), a tensor processing unit (TPU), etc., but these are merely embodiments. The artificial intelligence processor according to at least one embodiment of this disclosure may include various types of processing units for artificial intelligence and high-performance computing.

[0060] In step 204, the tensor processing method 200 may include receiving multiple tensors and physical layout information of the multiple tensors.

[0061] According to at least one embodiment of this disclosure, a plurality of tensors and physical layout information of the tensors specified by the user can be received. For example, the plurality of tensors may include one or more input tensors and one or more output tensors, and the one or more input tensors and one or more output tensors can be used for various processing logics. For example, according to at least one embodiment of this disclosure, the one or more input tensors and one or more output tensors can be used for various operators. For example, according to at least one embodiment of this disclosure, one or more input tensors can be loaded from a storage unit by executing instructions, corresponding computational logic can be performed on the loaded one or more input tensors to generate one or more output tensors, and the one or more output tensors can be written into the storage unit.

[0062] According to at least one embodiment of this disclosure, physical layout information may include the physical layout of storage units in an artificial intelligence processor for storing tensors. For example, according to at least one embodiment of this disclosure, the storage units may include at least one of a memory matrix and a register matrix. According to at least one embodiment of this disclosure, one or more input tensors can be loaded from a memory matrix to a register matrix by executing instructions, and one or more output tensors can be written from a register matrix to a memory matrix by executing instructions. According to at least one embodiment of this disclosure, one or more computational logics can be performed on the loaded tensors; for example, attention computation logic can be performed on the loaded tensors, but embodiments of this disclosure are not limited thereto. According to at least one embodiment of this disclosure, the artificial intelligence processor may include various types of storage units, not limited to memory matrices and register matrices.

[0063] According to at least one embodiment of the present disclosure, the physical layout of the storage cells can represent the arrangement of storage cells used to store tensors. According to at least one embodiment of the present disclosure, the physical layout of the storage cells can include at least one of a memory matrix physical layout and a register matrix physical layout. For example, according to at least one embodiment of the present disclosure, the memory matrix physical layout can represent the arrangement of a memory matrix used to store tensors. For example, according to at least one embodiment of the present disclosure, the register matrix physical layout can represent the arrangement of a register matrix used to store tensors.

[0064] According to at least one embodiment of this disclosure, the physical layout of the register matrix may include a register index layout and a thread layout. For example, according to at least one embodiment of this disclosure, the register index layout may represent the arrangement of one or more registers included in the register matrix, and the thread layout may represent the arrangement of threads in each register.

[0065] According to at least one embodiment of this disclosure, the first logical layout, the memory matrix physical layout, the register index layout, and the thread layout may each include a dimension vector and a step vector. Furthermore, the dimension vector may represent the dimensional information of the layout, and the step vector may represent the step size for each dimension. For example, according to at least one embodiment of this disclosure, the logical layout and the physical layout may be provided in the form of (shape, stride) or shape : stride. Here, shape may refer to the dimension vector of the layout, which represents the dimensional information of the layout, while stride may refer to the step vector of the layout, which represents the step information of the layout.

[0066] According to at least one embodiment of this disclosure, for example, when the layout (e.g., logical layout or physical layout) is a three-dimensional layout, the layout can be represented as ((shape1, shape2, shape3), (stride1, stride2, stride3)) or (shape1, shape2, shape3):(stride1, stride2, stride3). In this example, the vector (shape1, shape2, shape3) is the dimension vector shape of the layout, and (stride1, stride2, stride3) is the stride vector stride of the layout. Here, shape1, shape2, and shape3 represent the size of each dimension in the layout, and stride1, stride2, and stride3 represent the stride size of each dimension in the layout. For example, the layout can be represented as ((2, 2, 4), (1, 2, 4)), where the shape of the dimension vector of the layout is (2, 2, 4), which means that the size of each dimension in the three-dimensional layout is 2, 2, 4 respectively; the stride of the layout is (1, 2, 4), which means that the stride size of each dimension in the three-dimensional layout is 1, 2, 4.

[0067] like Figure 2 As shown, in step 206, the tensor processing method 200 may include generating a mapping descriptor corresponding to each of the multiple tensors based on the first logical layout and the physical layout information of the multiple tensors.

[0068] According to at least one embodiment of this disclosure, a mapping descriptor can be used to map the physical address of a tensor to the logical coordinates of a first logical layout. For example, the mapping descriptor can describe the mapping relationship between the physical address of a tensor and the logical coordinates of a first logical layout.

[0069] For example, according to at least one embodiment of this disclosure, a tensor memory access accelerator can be provided. The tensor memory access accelerator is a hardware-level memory access mechanism provided by an artificial intelligence processor. Its core principle is to automatically map the logical coordinates of the logical layout of a multidimensional tensor to the physical address of the storage unit through a mapping descriptor, and to achieve efficient, batch, and aligned data loading or storage by utilizing hardware optimization.

[0070] According to at least one embodiment of this disclosure, generating a mapping descriptor corresponding to each of the multiple tensors based on a first logical layout and the physical layout information of the multiple tensors may include generating a mapping descriptor corresponding to each of the multiple tensors based on the first logical layout, the shape and size of the tensor, the physical layout of the tensor, the starting address pointer of the tensor, and the data type of the tensor.

[0071] For example, according to at least one embodiment of this disclosure, the shape and size of the tensor, the starting address pointer of the tensor, and the data type of the tensor can be specified by the user, or the shape and size of the tensor, the starting address pointer of the tensor, and the data type of the tensor can be automatically inferred by an artificial intelligence processor based on the tensor input by the user.

[0072] According to at least one embodiment of this disclosure, a driver interface can be used to generate a mapping descriptor corresponding to each of the plurality of tensors based on a first logical layout and physical layout information of the tensors. For example, the driver interface can be used to generate a mapping descriptor corresponding to each of the plurality of tensors based on the first logical layout, the shape and size of the tensors, the physical layout of the tensors, the starting address pointer of the tensors, and the data type of the tensors.

[0073] Continue to refer to Figure 2 In step 208, the tensor processing method 200 may include performing data transfer and processing logic for multiple tensors based on a mapping descriptor.

[0074] According to at least one embodiment of this disclosure, data transfer of multiple tensors may include loading one or more tensors and outputting one or more tensors. For example, according to at least one embodiment of this disclosure, one or more input tensors may be loaded from a memory matrix to a register matrix based on a mapping descriptor for performing computational logic, and after the computational logic is completed, one or more output tensors may be written from the register matrix to the memory matrix based on the mapping descriptor.

[0075] For example, according to at least one embodiment of this disclosure, the mapping of the physical layout of one or more input tensors to a first logical layout (e.g., a unified logical layout) can be automatically performed based on a mapping descriptor. For example, the mapping of the physical addresses of one or more input tensors to the logical coordinates of the first logical layout can be automatically performed based on a mapping descriptor to perform the loading of one or more input tensors (e.g., loading one or more input tensors from a memory matrix to a register matrix). After the tensor loading is complete, one or more computational logics can be performed on the loaded tensors; for example, attention operator logic can be performed on the loaded tensors, but embodiments of this disclosure are not limited thereto. After the tensor computational logic is completed, the mapping of the logical coordinates of the first logical layout to the physical addresses of one or more output tensors can be automatically performed based on a mapping descriptor to perform the write-back of the results of one or more output tensors (e.g., writing one or more output tensors from a register matrix to a memory matrix).

[0076] According to at least one embodiment of this disclosure, a mapping descriptor can be passed as a parameter to an artificial intelligence processor, and the artificial intelligence processor performs the mapping between the physical address of a tensor and the logical coordinates of a first logical layout based on the mapping descriptor. For example, according to at least one embodiment of this disclosure, attention calculation logic can be performed on multiple tensors. When the attention operator is started, the mapping descriptor can be passed as an operator parameter to the artificial intelligence processor (e.g., a graphics processing unit (GPU)). The artificial intelligence processor can perform the mapping between the physical address of multiple tensors and the logical coordinates of a first logical layout based on the mapping descriptor when executing the attention calculation logic.

[0077] According to at least one embodiment of this disclosure, the mapping between the physical address of a tensor and the logical coordinates of a first logical layout performed by an AI processor based on a mapping descriptor may include, during data loading, the AI ​​processor performing a mapping from the physical address of an input tensor to the logical coordinates of the first logical layout based on a mapping descriptor; and during data output, the AI ​​processor performing a mapping from the logical coordinates of the first logical layout to the physical address of an output tensor based on a mapping descriptor. For example, according to at least one embodiment of this disclosure, during data loading, the mapping from the physical address of one or more input tensors to the logical coordinates of the first logical layout may be automatically performed based on a mapping descriptor to perform the loading of one or more input tensors (e.g., loading one or more input tensors from a memory matrix to a register matrix); during data output, the mapping from the logical coordinates of the first logical layout to the physical address of one or more output tensors may be automatically performed based on a mapping descriptor to perform the write-back of the results of one or more output tensors (e.g., writing one or more output tensors from a register matrix to a memory matrix).

[0078] According to at least one embodiment of this disclosure, the mapping between the physical address of a tensor and the logical coordinates of a first logical layout, performed by an artificial intelligence processor based on a mapping descriptor, may include executing the mapping between the physical address of the tensor and the logical coordinates of the first logical layout by the artificial intelligence processor executing instructions corresponding to the mapping descriptor. For example, according to at least one embodiment of this disclosure, a tensor can be loaded from a memory matrix to a register matrix using specific tensor memory accelerator instructions. The tensor memory accelerator can automatically complete the conversion from the physical address of the tensor to the logical coordinates of the first logical layout and efficiently move tensor data based on the mapping descriptor. After the data loading is completed and the corresponding computational logic is executed, the output result can be written back. For example, the output result can also be automatically written into the memory matrix based on specific tensor memory accelerator instructions.

[0079] According to at least one embodiment of this disclosure, multiple tensors may have different physical layout information. For example, according to at least one embodiment of this disclosure, multiple tensors may have different physical layouts. For example, according to at least one embodiment of this disclosure, a first logical layout (e.g., a unified logical layout) may be used by an attention operator to perform attention computation logic on multiple tensors. For example, an attention operator typically involves three input tensors (query, key, and value) and an output tensor, which may have different physical layouts in storage units due to differences in model architecture, hardware platform, or optimization strategy, such as BHSD, BSHD, SBHD, etc., where B represents the number of tensors, S represents the sequence length, H represents the number of attention heads, and D represents the dimension of each head.

[0080] The tensor processing method according to at least one embodiment of the present disclosure can realize the mapping of physical layouts of multiple tensors to a unified logical layout, so that the upper-level operator logic is developed only for a unified logical layout without the need to redevelop new operators, thereby achieving transparent support for multiple physical layouts and achieving relatively superior performance.

[0081] Figure 3 A tensor processing procedure 300 according to at least one embodiment of the present disclosure is shown.

[0082] According to at least one embodiment of this disclosure, Figure 3 The tensor processing procedure 300 shown can be executed by an artificial intelligence processor. For example, an artificial intelligence processor may include a general-purpose graphics processing unit (GPGPU), a neural network processing unit (NPU), a tensor processing unit (TPU), etc., but this is merely an embodiment. An artificial intelligence processor according to at least one embodiment of this disclosure may include various types of processing units for artificial intelligence and high-performance computing.

[0083] According to at least one embodiment of this disclosure, in the tensor processing procedure 300, a predefined first logical layout (e.g., a unified logical layout) for tensor processing in an artificial intelligence processor can be obtained. For example, the predefined first logical layout for tensor processing can be obtained based on user input. Figure 3 As shown, the first logical layout can be defined as (64, 32): (1, 64), that is, the dimension vector of the first logical layout is (64, 32), and the step vector is (1, 64). Figure 3 In the example shown, the first logical layout is defined as a two-dimensional layout, but the embodiments of this disclosure are not limited thereto. The first logical layout according to at least one embodiment of this disclosure may include a layout of any dimension.

[0084] refer to Figure 3 It can receive a first input tensor and a second input tensor, and can perform operator computation logic on the first input tensor and the second input tensor to generate a first output tensor and a second output tensor.

[0085] like Figure 3 As shown, the first input tensor and the second input tensor can have different physical layouts, and the first output tensor and the second output tensor can also have different physical layouts. For example, the physical layout of the first input tensor and the first output tensor can be (4,16,32):(1,4,64), while the physical layout of the second input tensor and the second output tensor can be (4,32,16):(1,4,128). According to at least one embodiment of this disclosure, the physical layout of the tensors can be included in physical layout information, for example, physical layout information of tensors specified by the user can be received from the user.

[0086] exist Figure 3 In the example shown, the first input tensor and the first output tensor are shown to have the same physical layout, and the second input tensor and the second output tensor are shown to have the same physical layout, but this is merely an example. According to at least one embodiment of this disclosure, the first input tensor and the second input tensor may have different physical layouts from the first output tensor and the second output tensor, respectively.

[0087] According to at least one embodiment of this disclosure, physical layout information may include the physical layout of storage cells in an artificial intelligence processor for storing tensors. For example, according to at least one embodiment of this disclosure, the storage cells may include at least one of a memory matrix and a register matrix, and the physical layout of the storage cells may include at least one of a memory matrix physical layout and a register matrix physical layout. According to at least one embodiment of this disclosure, tensors can be stored by executing instructions. Figure 3 The first and second input tensors are loaded from the memory matrix into the register matrix, and can be converted by executing instructions. Figure 3 The first and second output tensors are written from the register matrix to the memory matrix. According to at least one embodiment of this disclosure, one or more computational logics can be performed on the loaded first and second input tensors; for example, attention computation logic can be performed on the loaded tensors, but embodiments of this disclosure are not limited thereto. According to at least one embodiment of this disclosure, the artificial intelligence processor may include various types of storage units, not limited to memory matrices and register matrices.

[0088] According to at least one embodiment of this disclosure, Figure 3 The first logical layout shown can be independent of the physical layouts of the first and second input tensors, and various operators, such as attention operators, can be executed on the first and second input tensors based on a unified first logical layout. By executing various operators on tensors with different physical layouts based on a unified first logical layout, upper-level operator logic can be developed only for a unified logical layout, without having to redevelop new operators when processing tensors with different physical layouts, thus achieving transparent support for multiple physical layouts.

[0089] According to at least one embodiment of this disclosure, it is possible to base on Figure 3 The first logical layout shown, along with the physical layout information of the first input tensor, second input tensor, first output tensor, and second output tensor, generates a mapping descriptor corresponding to each of the first input tensor, second input tensor, first output tensor, and second output tensor. According to at least one embodiment of this disclosure, the mapping descriptor can be used to map the physical addresses of the first input tensor, second input tensor, first output tensor, and second output tensor to the logical coordinates of the first logical layout. For example, the mapping descriptor can describe the mapping relationship between the physical addresses of the first input tensor, second input tensor, first output tensor, and second output tensor and the logical coordinates of the first logical layout.

[0090] According to at least one embodiment of this disclosure, a mapping descriptor corresponding to each of the first input tensor, second input tensor, first output tensor, and second output tensor can be generated based on a first logical layout, the shape and size of the tensor, the physical layout of the tensor, the starting address pointer of the tensor, and the data type of the tensor. According to at least one embodiment of this disclosure, the shape and size of the tensor, the starting address pointer of the tensor, and the data type of the tensor can be specified by a user, or the shape and size of the tensor, the starting address pointer of the tensor, and the data type of the tensor can be automatically inferred by an artificial intelligence processor based on the tensor input by the user.

[0091] According to at least one embodiment of this disclosure, a driver interface can be used to generate a mapping descriptor corresponding to each of the first input tensor, second input tensor, first output tensor, and second output tensor based on a first logical layout and physical layout information of the first input tensor, second input tensor, first output tensor, and second output tensor. For example, the driver interface can be used to generate a mapping descriptor corresponding to each of the first input tensor, second input tensor, first output tensor, and second output tensor based on the first logical layout, the shape and size of the tensor, the physical layout of the tensor, the starting address pointer of the tensor, and the data type of the tensor.

[0092] refer to Figure 3 The mapping of the physical addresses of the first and second input tensors to the first logical layout can be performed based on the mapping descriptor. For example, the mapping of the physical addresses of the first and second input tensors to the logical coordinates of the first logical layout can be automatically performed based on the mapping descriptor to load the first and second input tensors (e.g., loading the first and second input tensors from the memory matrix to the register matrix). After the first and second input tensors are loaded, one or more computational logics can be performed on the loaded first and second input tensors via operators. For example, attention computation logic can be performed on the loaded first and second input tensors. After the computational logic on the first and second input tensors is performed via operators, the mapping of the logical coordinates of the first logical layout to the physical addresses of the first and second output tensors can be automatically performed based on the mapping descriptor to write back the results of the first and second output tensors (e.g., writing the first and second output tensors from the register matrix to the memory matrix).

[0093] Figure 4 An example of a tensor processing procedure used by an attention operator for a first logical layout according to at least one embodiment of the present disclosure is shown. References Figure 4 The diagram illustrates tensor processing procedures 410 and 420 for tensors with different physical layouts.

[0094] Attention operators can be used to implement the core AI model function of attention mechanisms. Attention operators dynamically assign different weights to each position by calculating the correlation between different positions in the input sequence, thereby capturing long-distance dependencies. For example, taking three tensors—query, key, and value—as input, attention weights are obtained by calculating the similarity between the query and the key, and then the values ​​are weighted and summed to output a context-aware representation tensor. Typically, the tensors involved in attention operators have different physical layouts in storage units depending on the model architecture, hardware platform, or optimization strategy, such as BHSD, BSHD, and SBHD, where B represents the number of tensors, S represents the sequence length, H represents the number of attention heads, and D represents the dimension of each head.

[0095] The method according to at least one embodiment of this disclosure can execute attention calculation logic for tensors with different physical layouts based on a unified first logical layout, achieving transparent support for different physical layouts without having to redevelop or modify attention operators for different physical layouts, thereby improving the efficiency and performance of code development.

[0096] like Figure 4 As shown, in the tensor processing procedure 410, a query input tensor, a key input tensor, and a value input tensor can be received, and the attention operator's computational logic can be executed on the query input tensor, the key input tensor, and the value input tensor to generate an output tensor. For example... Figure 4 As shown, the query input tensor, key input tensor, value input tensor, and output tensor can have a BSHD physical layout.

[0097] According to at least one embodiment of this disclosure, in the tensor processing procedure 410, a predefined first logical layout (e.g., a unified logical layout) for tensor processing in an artificial intelligence processor can be obtained. For example, the predefined first logical layout for tensor processing can be obtained based on user input.

[0098] like Figure 4 As shown, in tensor processing 410, the first logical layout can have a BHSD layout. The first logical layout can be independent of the physical layout of the query input tensor, key input tensor, and value input tensor, and attention operators for the query input tensor, key input tensor, and value input tensor can be executed based on a unified first logical layout.

[0099] According to at least one embodiment of this disclosure, in the tensor processing procedure 410, a mapping descriptor corresponding to each of the query input tensor, key input tensor, value input tensor, and output tensor can be generated based on the first logical layout and the physical layout information of the query input tensor, key input tensor, value input tensor, and output tensor. According to at least one embodiment of this disclosure, the mapping descriptor can be used to map the physical addresses of the query input tensor, key input tensor, value input tensor, and output tensor to the logical coordinates of the first logical layout. For example, the mapping descriptor can describe the mapping relationship between the physical addresses of the query input tensor, key input tensor, value input tensor, and output tensor and the logical coordinates of the first logical layout.

[0100] According to at least one embodiment of this disclosure, in tensor processing 410, the mapping of the physical addresses of the query input tensor, key input tensor, and value input tensor to the first logical layout can be performed based on a mapping descriptor. For example, the mapping of the physical addresses of the query input tensor, key input tensor, and value input tensor to the logical coordinates of the first logical layout can be automatically performed based on the mapping descriptor to load the query input tensor, key input tensor, and value input tensor (e.g., loading the query input tensor, key input tensor, and value input tensor from the memory matrix to the register matrix). After the query input tensor, key input tensor, and value input tensor are loaded, attention calculation logic can be performed on the loaded query input tensor, key input tensor, and value input tensor via an attention operator. After the calculation logic on the query input tensor, key input tensor, and value input tensor is performed via the attention operator and an output tensor is generated, the mapping of the logical coordinates of the first logical layout to the physical address of the output tensor can be automatically performed based on the mapping descriptor to write back the result of the output tensor (e.g., writing the output tensor from the register matrix to the memory matrix).

[0101] For example, in tensor processing 410, the query input tensor, key input tensor, value input tensor, and output tensor are shown to have the same physical layout (BSHD), but embodiments of this disclosure are not limited thereto, and the query input tensor, key input tensor, value input tensor, and output tensor may each have different physical layouts.

[0102] Continue to refer to Figure 4 , Figure 4The tensor processing procedure 420 shown is similar to the tensor processing procedure 410 described above. The difference between tensor processing procedure 420 and tensor processing procedure 410 is that in tensor processing procedure 410, the physical layout of the query input tensor, key input tensor, value input tensor, and output tensor is BSHD, while in tensor processing procedure 420, the physical layout of the query input tensor, key input tensor, value input tensor, and output tensor is SBHD. According to at least one embodiment of this disclosure, since the computational logic of the attention operator can be performed based on a unified first logical layout independent of the physical layout of the query input tensor, key input tensor, value input tensor, and output tensor, therefore... Figure 4 The attention operator shown can provide transparent support for different physical layouts. Even if the physical layouts of the query input tensor, key input tensor, value input tensor, and output tensor are different in tensor processing procedures 410 and 420, there is no need to redevelop the attention operator.

[0103] Figure 5 A tensor processing apparatus 500 according to at least one embodiment of the present disclosure is shown, the tensor processing apparatus 500 being used for an artificial intelligence processor. For example... Figure 5 As shown, the tensor processing device 500 may include an acquisition module 502, a receiving module 504, a generation module 506, and an execution module 508.

[0104] According to at least one embodiment of the present disclosure, the acquisition module 502 may be configured to acquire a predefined first logical layout for tensor processing in an artificial intelligence processor.

[0105] According to at least one embodiment of this disclosure, the predefined first logical layout can be a user-predefined first logical layout for processing multiple tensors. For example, the acquisition module 502 can be configured to acquire the predefined first logical layout for tensor processing based on user input. According to at least one embodiment of this disclosure, the first logical layout can correspond to a unified logical layout for tensor processing, which can be independent of the physical layout of the multiple tensors.

[0106] According to at least one embodiment of this disclosure, the receiving module 504 can be configured to receive a plurality of tensors and physical layout information of the tensors. According to at least one embodiment of this disclosure, the physical layout information includes the physical layout of storage cells in an artificial intelligence processor for storing tensors.

[0107] According to at least one embodiment of the present disclosure, the generation module 506 can be configured to generate a mapping descriptor corresponding to each of the plurality of tensors based on a first logical layout and physical layout information of a plurality of tensors, wherein the mapping descriptor is used for mapping between the physical address of the tensor and the logical coordinates of the first logical layout.

[0108] According to at least one embodiment of the present disclosure, execution module 508 can be configured to perform data transfer and processing logic for multiple tensors based on a mapping descriptor.

[0109] According to at least one embodiment of this disclosure, multiple tensors may have different physical layout information.

[0110] According to at least one embodiment of this disclosure, the physical layout of the storage cells may include at least one of a memory matrix physical layout and a register matrix physical layout.

[0111] According to at least one embodiment of this disclosure, the physical layout of the register matrix may include a register index layout and a thread layout.

[0112] According to at least one embodiment of this disclosure, the first logical layout, the memory matrix physical layout, the register index layout, and the thread layout may each include a dimension vector and a step vector of the layout.

[0113] According to at least one embodiment of this disclosure, a dimension vector can represent the dimensional information of a layout, and a step vector can represent the step size of each dimension.

[0114] According to at least one embodiment of the present disclosure, the generation module 506 can be configured to generate a mapping descriptor corresponding to each of a plurality of tensors based on a first logical layout, the shape and size of the tensor, the physical layout of the tensor, the starting address pointer of the tensor, and the data type of the tensor.

[0115] According to at least one embodiment of the present disclosure, the generation module 506 can be configured to use a driver interface to generate a mapping descriptor corresponding to each of the plurality of tensors based on a first logical layout and physical layout information of the plurality of tensors.

[0116] According to at least one embodiment of the present disclosure, a mapping descriptor can describe the mapping relationship between the physical address of a tensor and the logical coordinates of a first logical layout.

[0117] According to at least one embodiment of the present disclosure, the execution module 508 may be configured to pass a mapping descriptor as a parameter to the artificial intelligence processor and control the artificial intelligence processor to perform mapping between the physical address of a tensor and the logical coordinates of a first logical layout based on the mapping descriptor.

[0118] According to at least one embodiment of the present disclosure, the execution module 508 may be configured to, during data loading, control the artificial intelligence processor to perform mapping from the physical address of the input tensor to the logical coordinates of the first logical layout based on a mapping descriptor; and during data output, control the artificial intelligence processor to perform mapping from the logical coordinates of the first logical layout to the physical address of the output tensor based on a mapping descriptor.

[0119] According to at least one embodiment of the present disclosure, the execution module 508 may also be configured to perform the mapping between the physical address of a tensor and the logical coordinates of a first logical layout by controlling the artificial intelligence processor to execute instructions corresponding to the mapping descriptor.

[0120] According to at least one embodiment of the present disclosure, a first logical layout can be used by an attention operator to perform attention computation logic on a plurality of tensors.

[0121] The tensor processing method and apparatus provided according to at least one embodiment of this disclosure can map the physical layout of multiple tensors to a unified logical layout. By transferring the differences in the physical layout of tensors in memory cells to the memory cell access configuration of the tensor memory access accelerator, the upper-layer operator logic can be developed only for a unified logical layout without the need to redevelop new operators, thereby achieving transparent support for multiple physical layouts and relatively superior performance. The tensor processing method and apparatus provided according to at least one embodiment of this disclosure can significantly reduce development costs and improve maintenance efficiency. Only one set of general operator code needs to be written and maintained, without the need for separate implementation for each tensor layout; high-performance optimizations such as task partitioning / memory scheduling / parallel strategies can be universal across different tensor layouts without redesign, ensuring high-performance processing. With the help of the hardware mechanism of the tensor memory access accelerator, memory access efficiency and computational performance close to that of the native tensor layout can still be obtained while shielding the differences in tensor layouts.

[0122] Figure 6 An electronic device according to at least one embodiment of the present disclosure is shown.

[0123] The tensor processing method and apparatus provided in at least one embodiment of this disclosure can be applied to different systems or devices, such as those used in... Figure 6 The electronic device shown is 600.

[0124] The electronic device 600 can be a terminal, such as a mobile phone, tablet computer, laptop computer, AR device, VR device, vehicle terminal, etc., or it can be a server. The tensor processing method provided in at least one embodiment of this disclosure can be applied to scenarios involving tensor processing in the electronic device 600, such as high-performance computing (HPC) and artificial intelligence (AI), for example, a tensor computing unit. Of course, the embodiments of this disclosure are not limited to this; any scenario, device, or apparatus involving tensor processing can employ the tensor processing method or apparatus provided in at least one embodiment of this disclosure.

[0125] In some embodiments, the tensor processing apparatus provided in at least one embodiment of this disclosure can be a chip, such as a system-on-a-chip (SoC). A SoC includes a processor, which can be a single-core or multi-core processor, memory, and I / O interfaces, etc. The processor can load data and applications from memory and then process the data, for example, for tensor processing.

[0126] It should be noted that the data involved in the tensor processing method and apparatus provided in at least one embodiment of this disclosure may have different physical meanings depending on the application scenario. For example, the tensor processing method provided in at least one embodiment of this disclosure can be applied in fields such as speech processing, graphics processing, text processing, and video processing.

[0127] For example, in the field of speech processing, data can be input and output parameters in tasks such as feature extraction, speech enhancement, and speech recognition.

[0128] For example, in the field of graphics processing, data can be the input and output parameters for tasks such as image recognition, feature extraction, image segmentation, object detection, image classification, and scene reconstruction.

[0129] For example, in the field of text processing, data can be the input and output parameters for tasks such as text classification, sentiment analysis, and text generation.

[0130] For example, in the field of video processing, the data can be relevant parameters from the field of graphics processing, or input and output parameters specific to the field of video processing, such as optical flow operators (used to estimate motion between video frames) and target tracking operators (used to track specific targets in video).

[0131] Of course, the embodiments disclosed herein are not limited to these. For other application scenarios or fields, as long as tensor processing is required, the tensor processing method described in at least one embodiment of this disclosure can be applied, and will not be elaborated here.

[0132] like Figure 6 As shown, the electronic device 600 includes a processing unit 601, which can perform various appropriate actions and processes according to non-transitory computer-readable instructions stored in a memory to achieve various functions.

[0133] For example, when the computer-readable instruction processing device 601 executes, it can perform one or more steps of the tensor processing method according to any of the above embodiments. It should be noted that a detailed description of the tensor processing method process can be found in the relevant descriptions in the embodiments of the tensor processing method described above.

[0134] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 603 and / or cache memory, etc., for example, computer-readable instructions may be loaded from storage device 608 into RAM 603 to execute computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 602, hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0135] For example, the processing device 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0136] Typically, the following devices can be connected to I / O interface 605: input devices 606 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 607 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 608 including, for example, magnetic tapes, hard disks, flash memory, etc.; and communication devices 609. Communication device 609 allows electronic device 600 to communicate wirelessly or wiredly with other electronic devices to exchange data. Although Figure 6 An electronic device 600 with various devices is shown, but it should be understood that it is not required to implement or have all of the devices shown, and the electronic device 600 may alternatively implement or have more or fewer devices. For example, the processing device 601 may control other components in the electronic device 600 to perform desired functions.

[0137] For example, the processing device 601 can be a central processing unit (CPU), a tensor processor (TPU), or a graphics processing unit (GPU) with data processing and / or program execution capabilities. The CPU can be based on x86, ARM, RISC-V, or other architectures. The GPU can be directly integrated into the SoC, directly integrated onto the motherboard, or built into the motherboard's northbridge chip.

[0138] Figure 7 A non-transitory computer-readable storage medium 700 according to at least one embodiment of the present disclosure is shown.

[0139] For example, such as Figure 7 As shown, the storage medium 700 can be a non-transitory computer-readable storage medium on which one or more computer-readable instructions 701 can be stored non-transitory. For example, when the computer-readable instructions 701 are executed by a processor, one or more steps in the tensor processing method described above can be performed.

[0140] For example, the storage medium 700 can be used in an electronic device 600, such as the storage medium 700 including the storage device 608 in the electronic device 600.

[0141] For example, a storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB storage, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and a processor may execute these instructions to perform various functions of the processor. Various application programs and various data may also be stored in the storage medium.

[0142] For example, the storage medium may include a memory card for a smartphone, a cache component for a tablet computer, a hard disk for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media.

[0143] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0144] The units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units are not, in some cases, intended to limit the specific unit.

[0145] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0146] Those skilled in the art will understand that the illustrative embodiments described above are not intended to be limiting. It should be understood that any two or more of the embodiments disclosed herein can be combined in any combination. Furthermore, other embodiments may be utilized and other changes may be made without departing from the spirit and scope of the subject matter presented herein. It will be readily understood that aspects of this disclosure, as generally described herein and illustrated in the accompanying drawings, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are contemplated herein.

[0147] Various embodiments of this disclosure can be implemented as computer-readable code embodied on a computer-readable recording medium from a particular perspective. A computer-readable recording medium is any data storage device capable of storing data readable by a computer system. Examples of computer-readable recording media may include read-only memory (ROM), random access memory (RAM), optical disc read-only memory (CD-ROM), magnetic tape, floppy disk, optical data storage device, carrier wave (e.g., data transmission via the Internet), and the like. Computer-readable recording media can be distributed via computer systems connected via a network, and thus computer-readable code can be stored and executed in a distributed manner. Furthermore, the functional programs, code, and code segments used to implement the various embodiments of this disclosure can be readily interpreted by those skilled in the art applying the embodiments of this disclosure.

[0148] The medium can be read by a computer, stored in a memory, and executed by a processor. Various embodiments can be implemented by a computer or a portable terminal including a controller and a memory, and the memory can be an example of a non-transitory computer-readable recording medium suitable for storing programs(s) having instructions for implementing embodiments of this disclosure. This disclosure can be implemented by a program having code for specifically implementing the apparatus and methods described in the claims, said program being stored in a machine (or computer) readable storage medium. The program can be carried electronically on any medium, such as communication signals transmitted via wired or wireless connections, and this disclosure suitably includes its equivalents.

[0149] The embodiments described herein are not intended to be limiting. The aspects of this disclosure, as generally described herein and shown in the accompanying drawings, can be arranged, substituted, combined, separated, and designed in a variety of different configurations, all of which are conceivable herein. Furthermore, unless the context otherwise requires, the features shown in each drawing can be used in combination with each other. Therefore, the accompanying drawings should be considered... Figure 1 These features are generally considered to be part of one or more overall embodiments, but it should be understood that not all of the illustrated features are necessary for each embodiment.

[0150] The embodiments set forth in the foregoing description do not represent all embodiments consistent with the subject matter described herein. Rather, they are merely some examples consistent with aspects related to the described subject matter. Although some variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or changes may be provided in addition to those set forth herein. For example, the above embodiments may be applicable to various combinations and sub-combinations of the disclosed features and / or combinations and sub-combinations of several other features disclosed above. Furthermore, the logical flows depicted in the drawings and / or described herein do not necessarily require the specific order or sequence shown to achieve the desired results. Other implementations are within the scope of the appended claims.

[0151] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. Any person skilled in the art can make various changes or substitutions within the technical scope disclosed in this disclosure, and such changes or substitutions should all be covered within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the scope of the claims.

Claims

1. A tensor processing method for an artificial intelligence processor, characterized in that, The tensor processing method includes: Obtain a predefined first logical layout for tensor processing in the AI ​​processor; Receive multiple tensors and physical layout information of the multiple tensors, wherein the physical layout information includes the physical layout of the storage units in the artificial intelligence processor used to store the tensors; Based on the first logical layout and the physical layout information of the plurality of tensors, a mapping descriptor corresponding to each of the plurality of tensors is generated, wherein the mapping descriptor is used for mapping between the physical address of each tensor and the logical coordinates of the first logical layout. Execute data transfer and processing logic for the multiple tensors based on the mapping descriptor; The first logical layout is independent of the physical layout of the plurality of tensors, and the plurality of tensors have different physical layout information.

2. The tensor processing method according to claim 1, characterized in that, The physical layout of the storage unit includes at least one of a memory matrix physical layout and a register matrix physical layout.

3. The tensor processing method according to claim 2, characterized in that, The physical layout of the register matrix includes the register index layout and the thread layout.

4. The tensor processing method according to claim 3, characterized in that, The first logical layout, the memory matrix physical layout, the register index layout, and the thread layout each include a dimension vector and a step vector.

5. The tensor processing method according to claim 4, characterized in that, The dimension vector represents the dimensional information of the layout, and the step vector represents the step size of each dimension.

6. The tensor processing method according to any one of claims 1-5, characterized in that, Based on the first logical layout and the physical layout information of the plurality of tensors, a mapping descriptor corresponding to each of the plurality of tensors is generated, including: Based on the first logical layout, the shape and size of the tensor, the physical layout of the tensor, the starting address pointer of the tensor, and the data type of the tensor, a mapping descriptor corresponding to each of the plurality of tensors is generated.

7. The tensor processing method according to any one of claims 1-5, characterized in that, Based on the first logical layout and the physical layout information of the plurality of tensors, a mapping descriptor corresponding to each of the plurality of tensors is generated, including: Using the driver interface, a mapping descriptor corresponding to each of the multiple tensors is generated based on the first logical layout and the physical layout information of the multiple tensors.

8. The tensor processing method according to any one of claims 1-5, characterized in that, The mapping descriptor describes the mapping relationship between the physical address of each tensor and the logical coordinates of the first logical layout.

9. The tensor processing method according to any one of claims 1-5, characterized in that, The method further includes: The mapping descriptor is passed as a parameter to the artificial intelligence processor; and The artificial intelligence processor performs the mapping between the physical address of each tensor and the logical coordinates of the first logical layout based on the mapping descriptor.

10. The tensor processing method according to claim 9, characterized in that, The artificial intelligence processor performs the mapping between the physical address of each tensor and the logical coordinates of the first logical layout based on the mapping descriptor, including: During data loading, the AI ​​processor performs a mapping from the physical address of the input tensor to the logical coordinates of the first logical layout based on the mapping descriptor; and During data output, the AI ​​processor performs a mapping from the logical coordinates of the first logical layout to the physical address of the output tensor based on the mapping descriptor.

11. The tensor processing method according to claim 9, characterized in that, The artificial intelligence processor performs the mapping between the physical address of each tensor and the logical coordinates of the first logical layout based on the mapping descriptor, including: The mapping between the physical address of each tensor and the logical coordinates of the first logical layout is performed by the AI ​​processor executing instructions corresponding to the mapping descriptor.

12. The tensor processing method according to any one of claims 1-5, characterized in that, The first logical layout is used by the attention operator to perform attention computation logic on the plurality of tensors.

13. A tensor processing device for an artificial intelligence processor, characterized in that, The device includes: The acquisition module is configured to acquire a predefined first logical layout for tensor processing in the artificial intelligence processor; The receiving module is configured to receive multiple tensors and physical layout information of the multiple tensors, wherein the physical layout information includes the physical layout of the storage units in the artificial intelligence processor used to store the tensors; The generation module is configured to generate a mapping descriptor corresponding to each of the plurality of tensors based on the first logical layout and the physical layout information of the plurality of tensors, wherein the mapping descriptor is used for mapping between the physical address of each tensor and the logical coordinates of the first logical layout. The execution module is configured to perform data transfer and processing logic for the plurality of tensors based on the mapping descriptor; The first logical layout is independent of the physical layout of the plurality of tensors, and the plurality of tensors have different physical layout information.

14. A tensor processing electronic device, characterized in that, The electronic device includes: One or more processors; Memory, which stores one or more computer programs; Wherein, when executed by the one or more processors, the one or more computer programs cause the one or more processors to implement the method as described in any one of claims 1-12.

15. A non-transitory computer-readable storage medium, characterized in that, It stores computer-executable instructions thereon, which, when executed by a processor, cause at least one processor to implement the method as described in any one of claims 1-12.

Images