Tensor data access method, device, medium and program based on tpu processor

Abstract

Embodiments of the present application disclose a tensor data access method and device based on a TPU processor, a medium and a program, wherein the method comprises: determining tensor data correlation attribute information of tensor data to be accessed in a TPU processor; obtaining hardware structure characteristics of the TPU processor; determining a tensor data access scenario according to the tensor data correlation attribute information, and accessing the tensor data to be accessed according to the tensor data access scenario, tensor metadata in the tensor data correlation attribute information and the hardware structure characteristics of the TPU processor. The technical solution of the embodiments of the present application can realize tensor data access based on a TPU processor, and improve the access efficiency and accuracy of tensor data.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of data processing technology, and in particular to a tensor data access method, device, medium and program based on a TPU processor. Background Technology

[0002] In fields such as deep learning and scientific computing, tensor data access is a high-frequency, core requirement. Current implementations for this requirement suffer from significant performance bottlenecks or architectural compatibility issues.

[0003] When accessing tensor data on the CPU (Central Processing Unit), a common approach is to iterate through the tensor index using scalar loops, calculating the step offset element by element and copying the data. While this approach is logically intuitive and easy to implement, it relies entirely on a serial processing mode. When dealing with large-scale tensor data, its computational and memory access efficiency is extremely low, making it difficult to meet high-throughput business requirements.

[0004] The tensor data access solution on the GPU (Graphics Processing Unit) side leverages CUDA (Compute Unified Device Architecture) to break down computational tasks into thread blocks. Each thread independently performs single-element index calculations, and shared memory is used to cache local data, reducing the number of global memory accesses and significantly improving parallel processing efficiency. However, it's important to note that the GPU's unique "global memory + shared memory" two-level memory architecture differs fundamentally from the three-level memory architecture used by the TPU (Tensor Processing Unit). This makes the solution unsuitable for direct migration and adaptation, limiting its cross-architecture reusability. Summary of the Invention

[0005] This invention provides a method, device, medium, and program for accessing tensor data based on a TPU processor, which can realize tensor data access based on a TPU processor, while improving the access efficiency and accuracy of tensor data.

[0006] According to one aspect of the present invention, a tensor data access method based on a TPU processor is provided, applied to a TPU processor, comprising:

[0007] Determine the associated attribute information of the tensor data to be accessed in the TPU processor;

[0008] Obtain the hardware structure features of the TPU processor;

[0009] The tensor data access scenario is determined based on the tensor data association attribute information, and the tensor data to be accessed is accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor.

[0010] According to another aspect of the present invention, a tensor data access device based on a TPU processor is provided, configured in a TPU processor, comprising:

[0011] The tensor data association attribute information determination module is used to determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor;

[0012] A hardware structure feature acquisition module is used to acquire the hardware structure features of the TPU processor.

[0013] The tensor data access module is used to determine the tensor data access scenario based on the tensor data association attribute information, and to access the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one TPU processor; and

[0016] A memory communicatively connected to the at least one TPU processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one TPU processor, the computer program being executed by the at least one TPU processor to enable the at least one TPU processor to perform the TPU processor-based tensor data access method according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a TPU processor to execute and implement the TPU processor-based tensor data access method according to any embodiment of the present invention.

[0019] According to another aspect of the present invention, a computer program product is also provided, comprising a computer program that, when executed by a TPU processor, implements the TPU processor-based tensor data access method described in any embodiment of the present invention.

[0020] This invention, through embodiments thereof, determines the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtains the hardware structure characteristics of the TPU processor. Further, it determines the tensor data access scenario based on the tensor data association attribute information, and accesses the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. This solution addresses the shortcomings of low tensor data access efficiency and insufficient cross-architecture reusability in existing technologies, enabling tensor data access based on the TPU processor while improving the efficiency and accuracy of tensor data access.

[0021] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 1 of the present invention;

[0024] Figure 2 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 2 of the present invention;

[0025] Figure 3 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 3 of the present invention;

[0026] Figure 4 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 4 of the present invention;

[0027] Figure 5 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 5 of the present invention;

[0028] Figure 6 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment Six of the present invention;

[0029] Figure 7 This is a schematic diagram of a tensor data access device based on a TPU processor provided in Embodiment 7 of the present invention;

[0030] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 8 of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," "target," and "updated," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] Example 1

[0034] Figure 1 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 1 of the present invention. This embodiment is applicable to scenarios where tensor data access is implemented based on a TPU processor. The method can be executed by a tensor data access device based on a TPU processor. This device can be implemented in software and / or hardware, and is generally integrated into an electronic device, which can be a TPU processor. Correspondingly, as... Figure 1 As shown, the method includes the following operations:

[0035] S110. Determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor.

[0036] The TPU processor can be a type of application-specific integrated circuit (ASIC). The tensor data to be accessed can be the tensor data that the TPU processor needs to access during the execution of tensor data access tasks. The tensor data associated attribute information can be a set of data bound to the tensor data to be accessed, used to describe tensor characteristics, storage rules, operational constraints, and adaptation information.

[0037] In this embodiment of the invention, tensor data access can be implemented based on a TPU processor. During the process of accessing tensor data based on a TPU processor, the tensor data to be accessed in the TPU processor and its associated attribute information can first be determined.

[0038] S120. Obtain the hardware structure characteristics of the TPU processor.

[0039] Among them, hardware structural features can include the physical components that make up the TPU processor, the internal architecture layout, and the core technology design elements.

[0040] Correspondingly, in order to achieve efficient access to tensor data in the TPU processor environment, the hardware structure characteristics of the TPU processor can also be obtained, which can be used as a reference for the design and optimization of tensor data access schemes.

[0041] S130. Determine the tensor data access scenario based on the tensor data association attribute information, and access the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor.

[0042] In this context, a tensor data access scenario can be a data access scenario determined based on the continuity of the tensor data memory layout and the access purpose within a tensor data access task. For example, a tensor data access scenario can include continuous transformation access scenarios and non-continuous transformation access scenarios, etc. This embodiment of the invention does not limit the specific types included in the tensor data access scenario. A continuous transformation access scenario can be a scenario where tensor data with a non-continuous memory layout is converted to a continuous memory layout before data access. A non-continuous transformation access scenario can be a scenario where tensor data with a continuous memory layout is converted to a non-continuous memory layout before data access. Tensor metadata can be a collection of auxiliary information describing the core attributes of tensor data but excluding the original numerical data of the tensor data. For example, tensor metadata can include, but is not limited to, shape, step size, storage offset, and data type, etc. This embodiment of the invention does not limit the specific content included in the tensor metadata.

[0043] Accordingly, after obtaining the tensor data association attribute information and the hardware structure characteristics of the TPU processor, the tensor data access scenario can be determined based on the tensor data association attribute information. After determining the tensor data access scenario, the data access strategy for the tensor data to be accessed can be determined based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor, so as to access the tensor data to be accessed according to the data access strategy.

[0044] Therefore, the tensor data access method based on a TPU processor provided in this embodiment of the invention determines the data access strategy for tensor data based on the hardware structure characteristics of the TPU processor. This enables precise adaptation between the data access strategy and the hardware characteristics of the TPU processor, thereby shortening the processing time of tensor data and improving the access efficiency. Furthermore, by determining the data access strategy based on the tensor data access scenario and tensor metadata, the method achieves deep matching between the access strategy and tensor attributes and application scenarios. This avoids resource waste and efficiency loss caused by using a general access mode, and flexibly adapts to tensor data access needs of different dimensions, memory layouts, and data types.

[0045] This invention, through embodiments thereof, determines the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtains the hardware structure characteristics of the TPU processor. Further, it determines the tensor data access scenario based on the tensor data association attribute information, and accesses the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. This solution addresses the shortcomings of low tensor data access efficiency and insufficient cross-architecture reusability in existing technologies, enabling tensor data access based on the TPU processor while improving the efficiency and accuracy of tensor data access.

[0046] Example 2

[0047] Figure 2 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 2 of the present invention. This embodiment is a specific embodiment based on the above embodiment. In this embodiment, if the tensor data access scenario includes a general continuous transformation access scenario, then specific optional implementation methods are given for accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. Correspondingly, as Figure 2 As shown, the method in this embodiment may include:

[0048] S210. Determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor.

[0049] S220. Obtain the hardware structure characteristics of the TPU processor.

[0050] S230. In the case where the tensor data access scenario includes a general continuous transformation access scenario, the target memory offset of the tensor data to be accessed is determined based on the tensor metadata and the hardware structure characteristics of the TPU processor.

[0051] In this context, the general contiguous transformation access scenario can be a scenario where tensor data with a non-contiguous memory layout is converted into a contiguous memory layout before data access. The target memory offset can be the element index offset of the first valid element of the tensor data to be accessed, relative to the starting position of its underlying shared storage, after alignment processing according to the byte alignment requirements in the hardware structure characteristics.

[0052] In this embodiment of the invention, if the memory layout of the tensor data to be accessed is non-contiguous, the tensor data access scenario can be a general contiguous transformation access scenario. When the tensor data access scenario is a general contiguous transformation access scenario, the target memory offset of the tensor data to be accessed can be determined based on the tensor metadata and the hardware structure characteristics of the TPU processor, so that the tensor data to be accessed can be loaded from the target bandwidth memory region to the first target cache region of the TPU processor according to the target memory offset.

[0053] In an optional embodiment of the present invention, determining the target memory offset of the tensor data to be accessed based on the tensor metadata and the hardware structure features of the TPU processor may include: generating a linear index of the target output tensor data based on the TPU vector register memory; determining the original memory offset of the tensor data to be accessed based on the tensor metadata and the linear index of the target output tensor data; aligning the memory offset of the tensor data to be accessed according to the byte alignment requirements in the hardware structure features of the TPU processor to obtain the target memory offset of the tensor data to be accessed.

[0054] Here, TPU vector register memory can be the memory capacity occupied by vector registers in the TPU processor. Target output tensor data can be the tensor data obtained after transforming the tensor data to be accessed. Linear index can be an index in a contiguous address space. Raw memory offset can be the element index offset of the first valid element of the tensor data to be accessed relative to the starting location of its underlying shared memory.

[0055] In this embodiment of the invention, when determining the target memory offset of the tensor data to be accessed based on the tensor metadata and the hardware structure characteristics of the TPU processor, a linear index of the target output tensor data can first be generated based on the TPU vector register memory. For example, linear indices of the target output tensor data can be generated in batches based on the TPU 1024-bit vector register, and indices pointing to invalid padding regions can be filtered to ensure that only valid data is processed. Further, the linear index can be converted into multidimensional coordinates. After obtaining the multidimensional coordinates, the original memory offset of the tensor data to be accessed can be determined based on the tensor metadata and the linear index of the target output tensor data using the following formula:

[0056] ;

[0057] in, This is the original memory offset. Let n be the storage offset of the tensor data to be accessed, and n be the total number of dimensions of the tensor data to be accessed. This is the logical index of the element in the k-th dimension. The step size of the k-th dimension.

[0058] Furthermore, after determining the original memory offset of the tensor data to be accessed, it can be aligned according to the byte alignment requirements in the TPU processor's hardware architecture to obtain the target memory offset for accessing the tensor data. For example, the original memory offset can be adjusted downwards according to the TPU processor's 512-byte alignment requirement to obtain the target memory offset, thereby avoiding unaligned access.

[0059] ;

[0060] in, The remainder when the original memory offset is divided by the alignment strength. For alignment granularity. In a specific example, if It is 512 bytes. =96 bytes, then =96, target memory offset .

[0061] S240. Load the tensor data to be accessed from the target bandwidth memory region to the first target cache region of the TPU processor according to the target memory offset.

[0062] The target bandwidth memory region can be a memory region in the TPU processor that stores the tensor data to be accessed. For example, the target bandwidth memory region may include, but is not limited to, HBM (High Bandwidth Memory). The first target cache region can be a data cache region in the TPU processor. For example, the first target cache region may include, but is not limited to, VMEM (Vector Memory).

[0063] Accordingly, after determining the target memory offset, the TPU processor can load the tensor data to be accessed from the target bandwidth memory region to the first target cache region of the TPU processor according to the target memory offset.

[0064] Continuing with the example above, after determining that the target memory offset is 0, the TPU processor can start from 0 bytes and read 512 bytes of data to be accessed into VMEM.

[0065] S250. Aggregate the tensor data to be accessed according to the offset index of the tensor data to be accessed in the first target cache area, and determine the continuous transformation result of the tensor data to be accessed in the first target cache area.

[0066] The offset index can be the byte offset of each tensor data element relative to the block base address of its respective first target cache area after the tensor data to be accessed is loaded into the first target cache area.

[0067] Specifically, after loading the tensor data to be accessed into the first target cache region of the TPU processor, the offset index of the tensor data to be accessed in the first target cache region can be calculated.

[0068] In a specific example, the intra-block relative offset within the first target cache region can be calculated based on the following formula:

[0069] ;

[0070] Furthermore, the corrected intra-block offset within the first target cache region can be calculated based on the following formula:

[0071] ;

[0072] in, This is the corrected intra-block offset. This is a relative offset within the block. For the block index, The size of a single block of tensor data to be accessed.

[0073] Furthermore, the offset index of the tensor data to be accessed in the first target cache region can be determined based on the following formula:

[0074] ;

[0075] in, The offset index of the tensor data to be accessed in the first target cache region. This is the base address for the VMEM block.

[0076] Furthermore, the Permute instruction of the NWS unit can be invoked to aggregate the tensor data to be accessed based on its offset index in the first target buffer region, thereby determining the continuous transformation result of the tensor data to be accessed. This process can achieve non-continuous aggregation of multiple elements within a single cycle.

[0077] It should be noted that for bfloat16 (16-bit floating-point) data, it needs to be expanded to a 32-bit data format first to adapt to the Permute instruction input; for int32 (32-bit integer) data, the corresponding data type's Permute instruction can be called directly. This ensures compatibility and adaptation across multiple data types, thereby ensuring a balance between data access precision and hardware efficiency. If the next batch of data exceeds the addressing range of the current first target cache area, tensor data can be reloaded into the first target cache area, and the above non-continuous aggregation process can be repeated to ensure the continuity and integrity of data processing.

[0078] S260. The continuous transformation result of the tensor data to be accessed in the first target cache area is taken as the target output tensor data, and the target output tensor data is written back to the target continuous memory area of ​​the target bandwidth memory area.

[0079] The target contiguous memory region can be a region within the target bandwidth memory region that stores contiguous data.

[0080] Accordingly, after determining the continuous transformation result of the tensor data to be accessed in the first target cache region, the continuous transformation result can be used as the target output tensor data, and the target output tensor can be written back to the target continuous memory region of the target bandwidth memory region.

[0081] This invention, through embodiments thereof, determines the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtains the hardware structure characteristics of the TPU processor. Further, in cases where the tensor data access scenario includes a general continuous transformation access scenario, the target memory offset of the tensor data to be accessed is determined based on the tensor metadata and the hardware structure characteristics of the TPU processor. After determining the target memory offset, the tensor data to be accessed is loaded from the target bandwidth memory region to the first target cache region of the TPU processor according to the target memory offset. Further, the tensor data to be accessed is aggregated according to the offset index of the tensor data to be accessed in the first target cache region to determine the continuous transformation result of the tensor data to be accessed in the first target cache region. After obtaining the continuous transformation result, the continuous transformation result of the tensor data to be accessed in the first target cache region is used as the target output tensor data, and the target output tensor data is written back to the target continuous memory region of the target bandwidth memory region. The above solution solves the defects of low tensor data access efficiency and insufficient cross-architecture reusability in the prior art, enabling tensor data access based on the TPU processor while improving the access efficiency and accuracy of tensor data.

[0082] Example 3

[0083] Figure 3This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 3 of the present invention. This embodiment is a specific embodiment based on the above embodiment. In this embodiment, if the tensor data access scenario includes a first branch continuous transformation access scenario, then a specific optional implementation method is given for accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. Correspondingly, as Figure 3 As shown, the method in this embodiment may include:

[0084] S310. Determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor.

[0085] S320. Obtain the hardware structure characteristics of the TPU processor.

[0086] S330. In the case of tensor data access scenarios including the first branch continuous transformation access scenario, the first transmission parameters of the tensor data to be accessed are determined according to the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure features of the TPU processor.

[0087] The first branch continuous transformation access scenario can be a continuous transformation access scenario in which the first target dimension data in the tensor data to be accessed is continuously stored in memory. The first target dimension data can be other dimension data in the tensor data to be accessed besides the first dimension data. The second target dimension data can be the lower two dimensions of the tensor data to be accessed. The byte alignment requirement can be that the starting physical address of all multi-byte data to be accessed must be aligned to an address boundary that is an integer multiple of the width of the data itself. The first transmission parameter can be the parameter referenced when loading the second target dimension data in the tensor data to be accessed from the target bandwidth memory region to the second target cache region of the TPU processor. For example, the first transmission parameter may include, but is not limited to, the starting address and transmission length, etc., and the specific types included in the first transmission parameter are not limited in this embodiment of the invention.

[0088] In this embodiment of the invention, if the lower two dimensions of the tensor data to be accessed are stored contiguously in memory, the lower two dimensions can be treated as a complete data block for operation, without needing to break it down into byte-level granularity for transmission and computation. In this case, the secondary target cache region shared by the dual cores of the TPU processor, such as CMEM (Constant Memory), can be used as an intermediate relay to load the tensor data to be accessed from the target bandwidth memory region to the secondary target cache region of the TPU processor. Before performing the data loading operation, the first transmission parameters of the tensor data to be accessed can be determined based on the tensor metadata of the first target dimension data and the byte alignment requirements in the hardware structure characteristics of the TPU processor. In a specific example, the starting address of each batch of tensor data to be accessed can be calculated based on the following formula:

[0089] ;

[0090] The transmission length can be calculated based on the following formula:

[0091] ;

[0092] in, This is the starting address for each batch of tensor data to be accessed. Let N be the dimension size of the tensor data to be accessed. Let N be the storage offset of the Nth dimension of the tensor data to be accessed. For transmission length, The number of bytes representing the data type of the tensor data to be accessed.

[0093] S340. Load the second target dimension data in the tensor data to be accessed from the target bandwidth memory region to the second target cache region of the TPU processor according to the first transmission parameters.

[0094] Accordingly, after determining the first transmission parameters of the tensor data to be accessed, the second target dimension data, i.e., the lower two-dimensional continuous data, in the tensor data to be accessed can be loaded from the target bandwidth memory area to the second target cache area of ​​the TPU processor according to the first transmission parameters.

[0095] S350, take the second target dimension data in the second target cache region as the target output tensor, and write the target output tensor back to the target contiguous memory region of the target bandwidth memory region.

[0096] Accordingly, after loading the second target dimension data from the target bandwidth memory region into the second target cache region of the TPU processor, the second target dimension data in the second target cache region can be used as the target output tensor, and the target output tensor can be written back to the target contiguous memory region of the target bandwidth memory region.

[0097] Optionally, when accessing high-dimensional loops of stored data, only the CMEM read / write window (offset address) needs to be switched, without having to repeatedly allocate CMEM cache, thus saving hardware resources.

[0098] This invention, through embodiments thereof, determines the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtains the hardware structure characteristics of the TPU processor. In tensor data access scenarios including a first branch continuation transformation access scenario, a first transmission parameter for the tensor data to be accessed is determined based on the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure characteristics of the TPU processor. Further, based on the first transmission parameter, the second target dimension data in the tensor data to be accessed is loaded from the target bandwidth memory region to the second target cache region of the TPU processor. Further, the second target dimension data in the second target cache region is used as the target output tensor, and the target output tensor is written back to the target contiguous memory region of the target bandwidth memory region. The above solution solves the defects of low tensor data access efficiency and insufficient cross-architecture reusability in the prior art, enabling tensor data access based on the TPU processor while improving the access efficiency and accuracy of tensor data.

[0099] Example 4

[0100] Figure 4 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 4 of the present invention. This embodiment is a specific embodiment based on the above embodiment. In this embodiment, if the tensor data access scenario includes a second branch continuous transformation access scenario, then a specific optional implementation method is given for accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. Correspondingly, as Figure 4 As shown, the method in this embodiment may include:

[0101] S410. Determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor.

[0102] S420: Obtain the hardware structure characteristics of the TPU processor.

[0103] S430. In the case of tensor data access scenarios including second branch continuous transformation access scenarios, the second transmission parameters of the tensor data to be accessed are determined according to the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure features of the TPU processor.

[0104] The second branch continuous transformation access scenario can be a continuous transformation access scenario where the second target dimension data requires transposition. The second transmission parameter can be the parameter referenced when asynchronously loading the tensor data to be accessed from the target bandwidth memory region to the input area of ​​the first target cache region of the TPU processor. For example, the second transmission parameter may include, but is not limited to, source address and transmission length, etc., and the embodiments of the present invention do not limit the specific types included in the second transmission parameter.

[0105] In this embodiment of the invention, in a continuous transformation access scenario where the second target dimension data has a transposition requirement, the source address and transmission length of the tensor data to be accessed can be determined based on the tensor metadata of the first target dimension data and the byte alignment in the hardware structure features of the TPU processor.

[0106] In a specific example, independent input and output regions can be divided in the VMEM of the TPU processor. The capacity of both regions can be calculated and determined according to the formula "number of rows of transposed sub-blocks × number of columns × number of bytes of data type". At the same time, the physical address can be isolated and allocated through the vmem_alloc function to ensure that the operation of loading data into the input region and storing data into the output region does not interfere with each other.

[0107] When loading non-contiguous tensor data to be accessed from the target bandwidth memory region, the HBM2VMEM function can be called to calculate the source address based on the dimension offset and storage offset of the first target dimension data using the following formula:

[0108] ;

[0109] ;

[0110] in, The high-dimensional offset of the tensor data to be accessed. For the index of the Nth dimension, The storage offset for the Nth dimension. For the source address, The base address of the tensor data to be accessed is the HBM base address.

[0111] Optionally, the calculated source address can be aligned according to the byte alignment requirements of the TPU processor.

[0112] Furthermore, the product of the number of sub-block rows, the number of aligned columns, and the number of data type bytes can be calculated, and the calculated product can be used as the transmission length.

[0113] S440. According to the second transmission parameters of the tensor data to be accessed, the tensor data to be accessed is asynchronously loaded from the target bandwidth memory region to the input area of ​​the first target cache region of the TPU processor.

[0114] Accordingly, after determining the second transfer parameters of the tensor data to be accessed, the tensor data can be asynchronously loaded into the input area of ​​the first target cache region of the TPU processor according to the second transfer parameters, so as to maintain efficient handling of non-contiguous data. It is understood that the loading process can execute other initialization operations in parallel.

[0115] S450. Based on the attribute information of the transposed cache channel in the hardware structure features of the TPU processor, the tensor data to be accessed is written into the transposed cache channel in blocks and loops. The tensor data to be accessed in the transposed cache channel is transposed and rearranged according to the data mapping rules to obtain transposed continuous tensor data. The transposed continuous tensor data is then written to the output area of ​​the first target cache region.

[0116] The transpose cache channel can be a hardware-level cache channel in the TPU processor designed for operations such as tensor transpose and dimension permutation. For example, the transpose cache channel can be a TRF (Transpose Register File). The data mapping rules can be the rules followed when transposing the second target dimension data of the tensor data to be accessed. Transposed continuous tensor data can be the transposed continuous data obtained after transpose and rearrangement processing.

[0117] Correspondingly, after asynchronously loading the tensor data to be accessed into the input area of ​​the first target cache region of the TPU processor, the dedicated function `tile_transfer` for tensor transposition can be executed by the NWS Permute (dimension permutation) hardware unit in the TPU processor. Specifically, the transposition parameters can be initialized using the `m_transpose_start` instruction, locking the transposition width to 128, and writing the first block of 8 rows × 128 columns of data from the input area of ​​the first target cache region into the corresponding channel of the TRF according to the parallel path of XYS subcores 0-7. Based on this, the remaining 14 blocks of data (8 rows each) can be processed cyclically using the `m_transpose_mid` instruction to maintain continuous filling of the TRF. During this process, 8 × 128 = 1024 elements can be transferred in parallel per cycle. Finally, the `m_transpose_end` instruction can be called to trigger hardware-level row and column swapping after the last block of data is passed in. According to the data mapping rule of "original address [i][j] → new address [j][i]", the data stored discretely by row in the TRF is rearranged into a continuous format by column to obtain transposed continuous tensor data. Furthermore, the obtained transposed continuous tensor data can be written to the output area of ​​the first target buffer region.

[0118] S460. The transposed continuous tensor data is used as the target output tensor, and the target output tensor is written back to the target continuous memory region of the target bandwidth memory region.

[0119] Accordingly, after writing the transposed continuous tensor data to the output area of ​​the first target cache region, a dual-core task splitting strategy can be adopted. For example, XYS0 processes the first half of the data, and XYS1 processes the second half. The transposed continuous tensor data in the output area of ​​the first target cache region is then transferred in batches to the target continuous memory region of the target bandwidth memory region. Optionally, for bfloat16 type data, 32-bit to 16-bit format compression can be performed using the f32ToBf16_256 function.

[0120] The entire process described above uses VMEM as an intermediary to transform the O(N²) random accesses of HBM into two batch DMA transfers. Combined with the single-cycle 1024-element transpose capability of NWS hardware, it significantly reduces memory latency and computational overhead.

[0121] This invention, in its embodiments, determines the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtains the hardware structure characteristics of the TPU processor. In cases where the tensor data access scenario includes a second branch continuous transformation access scenario, the second transmission parameters of the tensor data to be accessed are determined based on the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure characteristics of the TPU processor. Based on these second transmission parameters, the tensor data to be accessed is asynchronously loaded from the target bandwidth memory region to the input area of ​​the first target cache region of the TPU processor. Further, based on the attribute information of the transposed cache channel in the hardware structure characteristics of the TPU processor, the tensor data to be accessed is written to the transposed cache channel in blocks in a cyclic manner. According to the data mapping rules, the tensor data to be accessed in the transposed cache channel is transposed and rearranged to obtain transposed continuous tensor data, which is then written to the output area of ​​the first target cache region. Further, the transposed continuous tensor data is used as the target output tensor, and the target output tensor is written back to the target continuous memory region of the target bandwidth memory region. The above solution addresses the shortcomings of low tensor data access efficiency and insufficient cross-architecture reusability in existing technologies, enabling tensor data access based on TPU processors while improving tensor data access efficiency and accuracy.

[0122] Example 5

[0123] Figure 5 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment 5 of the present invention. This embodiment is a specific embodiment based on the above embodiment. In this embodiment, if the tensor data access scenario includes an ordered discontinuous transformation access scenario, then specific optional implementation methods are given for accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. Correspondingly, as Figure 5 As shown, the method in this embodiment may include:

[0124] S510. Determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor.

[0125] S520: Obtain the hardware structure characteristics of the TPU processor.

[0126] S530. In the case of tensor data access scenarios including ordered discontinuous transformation access scenarios, determine the first nonlinear index of the target output tensor data based on the tensor metadata of the tensor data to be accessed.

[0127] The ordered discontinuous transformation access scenario can be a scenario where tensor data to be accessed, which has a continuous memory layout and an increasing step size, is transformed into tensor data access in a discontinuous memory storage format. The first non-linear index can be the index of the target output tensor data in the third target cache region and the target discontinuous memory region.

[0128] In this embodiment of the invention, if the tensor data access scenario is an ordered discontinuous transformation access scenario, the first nonlinear index of the target output tensor data can be determined based on the tensor metadata of the tensor data to be accessed. In a specific example, in a TPU processor, 1024 linear indices can first be generated. Furthermore, the linear indices can be added to the storage offset recorded in the tensor metadata of the tensor data to be accessed, and the result can be used as the first nonlinear index.

[0129] S540. The tensor data to be accessed is split based on the computing core, and the split tensor data to be accessed is loaded into the source data current area corresponding to each computing core in the third target cache area of ​​the TPU processor.

[0130] The computing core can be the XYS core in a TPU processor. The third target cache region can be the data cache region in a TPU processor. For example, the third target cache region can be, but is not limited to, SMEM (Scalar Memory). This embodiment of the invention does not limit the specific type of the third target cache region.

[0131] In this embodiment of the invention, the third target cache area of ​​the current computing core can be divided into six functionally independent partitions: source offset storage area, target offset storage area, source data current area, source data preloading area, target data current area, and target data pre-storage area. This scheme avoids read / write conflicts in the third target cache area through partition isolation, ensuring the correctness of indexing operations and data reassembly. Specifically, the source offset storage area stores linear indexes of continuous data for locating the HBM data source location; the target offset storage area stores non-contiguous address indexes calculated with a step size for locating the HBM target location; the source data current area stores the currently processed continuous data blocks; the source data preloading area asynchronously loads the next batch of continuous data blocks for subsequent processing; the target data current area stores the current data blocks reassembled using non-contiguous addresses; and the target data pre-storage area temporarily stores the next batch of reassembled non-contiguous data blocks to ensure processing continuity. It is understood that the first non-linear index can be stored in the target offset storage area.

[0132] In this embodiment of the invention, the tensor data to be accessed can be split according to the identifier of each computing core, with XYS0 processing the first half of the data and XYS1 processing the second half, and there is no data interaction between the cores. Furthermore, the split tensor data to be accessed can be loaded into the source data current area corresponding to each computing core in the third target cache region of the TPU processor.

[0133] S550. Reorganize the tensor data to be accessed in each of the source data current areas according to the first nonlinear index to obtain the first non-continuous tensor data, and load the first non-continuous tensor data into each of the target data current areas of the third target cache area.

[0134] The first non-continuous tensor data can be obtained by mapping the tensor data to be accessed according to the first non-linear index.

[0135] Correspondingly, after loading the split tensor data to be accessed into the source data current area corresponding to each computing core in the third target cache area, the continuous tensor data to be accessed can be mapped to the target data current area corresponding to each computing core in the third target cache area according to the first nonlinear index, so as to convert the continuously stored tensor data to be accessed into the first non-contiguous tensor data.

[0136] S560. The first non-contiguous tensor data is used as the target output tensor data, and the target output tensor data is written back to the target non-contiguous memory region of the target bandwidth memory region.

[0137] Accordingly, after loading the first non-contiguous tensor data into the target data current area of ​​the third target cache area, the first non-contiguous tensor data can be used as the target output tensor data, and the target output tensor data can be written back to the target non-contiguous memory area of ​​the target bandwidth memory area.

[0138] This invention, through determining the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtaining the hardware structure characteristics of the TPU processor, addresses the tensor data access scenario, including ordered discontinuous transformation access scenarios. It determines the first nonlinear index of the target output tensor data based on the tensor metadata of the tensor data to be accessed. Further, the tensor data to be accessed is split based on the computing core, and the split tensor data is loaded into the source data current area corresponding to each computing core in the third target cache region of the TPU processor. Further, the tensor data to be accessed in each source data current area is reorganized according to the first nonlinear index to obtain the first discontinuous tensor data. This first discontinuous tensor data is loaded into the target data current area of ​​the third target cache region, and then used as the target output tensor data. The target output tensor data is written back to the target discontinuous memory area of ​​the target bandwidth memory region. This solution solves the defects of low tensor data access efficiency and insufficient cross-architecture reusability in the prior art, enabling tensor data access based on the TPU processor while improving the access efficiency and accuracy of tensor data.

[0139] Example 6

[0140] Figure 6 This is a flowchart of a tensor data access method based on a TPU processor provided in Embodiment Six of the present invention. This embodiment is a specific embodiment based on the above embodiment. In this embodiment, if the tensor data access scenario includes an unordered discontinuous transformation access scenario, then specific optional implementation methods are given for accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. Correspondingly, as Figure 6 As shown, the method in this embodiment may include:

[0141] S610. Determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor.

[0142] S620: Obtain the hardware structure characteristics of the TPU processor.

[0143] S630. In the case of tensor data access scenarios including unordered discontinuous transformation access scenarios, obtain the second nonlinear index of the target output tensor data.

[0144] The unordered, non-contiguous transformation access scenario can be a scenario where tensor data to be accessed, which has a contiguous memory layout but unordered step size, is transformed into tensor data access in a non-contiguous memory storage format. The second non-linear index can be the index of the target output tensor data in the third target cache region and the target non-contiguous memory region.

[0145] In this embodiment of the invention, if the tensor data access scenario is an unordered and discontinuous transformation access scenario, then the second nonlinear index of the target output tensor data can be obtained first.

[0146] S640. Load the first batch of tensor data to be accessed into the source data current area of ​​the third target cache region in the TPU processor, and load the second batch of tensor data into the source data preload area of ​​the third target cache region.

[0147] The first batch of tensor data can be the first batch of data obtained after dividing the tensor data to be accessed into batches. The second batch of tensor data can be the second batch of data obtained after dividing the tensor data to be accessed into batches.

[0148] Accordingly, after obtaining the second nonlinear index of the target output tensor data, the first batch of tensor data to be accessed can be loaded into the source data current area of ​​the third target cache region in the TPU processor, and the second batch of tensor data can be loaded into the source data preload area of ​​the third target cache region.

[0149] S650. Reorganize the first batch of tensor data in the current area of ​​the source data according to the second nonlinear index to obtain the first batch of non-continuous tensor data, and load the first batch of non-continuous tensor data into the current area of ​​the target data in the third target cache area.

[0150] Specifically, the first batch of tensor data in the current area of ​​the source data can be reorganized according to the second non-linear index to obtain the first batch of non-contiguous tensor data, and then loaded into the current area of ​​the target data in the third target cache area. It can be understood that the address of the first batch of non-contiguous tensor data in the current area of ​​the target data can be: the base address of the current area of ​​the target data + the second non-linear index × the number of data bytes.

[0151] S660. Transfer the first batch of non-contiguous tensor data in the current region of the target data to the target non-contiguous memory region.

[0152] Accordingly, after loading the first batch of non-contiguous tensor data into the target data current area of ​​the third target cache region, the first batch of non-contiguous tensor data in the target data current area can be further transferred to the target non-contiguous memory area to realize tensor data access.

[0153] S670. The source data preloading area storing the second batch of tensor data is used as the updated source data current area, and the source data current area is used as the updated source data preloading area. According to the byte alignment requirements in the hardware structure features of the TPU processor, the tensor data to be accessed is loaded in batches into the updated source data current area and the updated source data preloading area until all the tensor data to be accessed is reassembled and written back to the target non-contiguous memory area of ​​the target bandwidth memory region.

[0154] Accordingly, after processing the first batch of tensor data, the pointer to the third target cache area can be directly switched, using the source data preload area as the updated source data current area and the original source data current area as the updated source data preload area. This pointer switching replaces data movement, achieving seamless batch processing and reducing process latency. Furthermore, based on the byte alignment requirements of the TPU processor's hardware architecture, the tensor data to be accessed can be loaded in batches into the updated source data current area and the updated source data preload area until all the tensor data to be accessed is reassembled and written back to the target non-contiguous memory area of ​​the target bandwidth memory region.

[0155] This invention, through determining the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtaining the hardware structure characteristics of the TPU processor, addresses the following scenarios: In cases where tensor data access includes unordered, discontinuous transformation access, a second nonlinear index of the target output tensor data is obtained. A first batch of tensor data to be accessed is loaded into the source data current area of ​​the third target cache region in the TPU processor, and a second batch of tensor data is loaded into the source data preloading area of ​​the third target cache region. Further, the first batch of tensor data in the source data current area is reorganized according to the second nonlinear index to obtain the first batch of discontinuous tensor data, which is then loaded into the target data current area of ​​the third target cache region. Finally, the first batch of discontinuous tensor data in the target data current area is transferred to the target discontinuous memory region. Furthermore, the source data preloading area storing the second batch of tensor data is used as the updated source data current area, and the source data current area is used as the updated source data preloading area. Based on the byte alignment requirements in the TPU processor's hardware architecture, the tensor data to be accessed is loaded in batches into the updated source data current area and the updated source data preloading area until all the tensor data to be accessed is reassembled and written back to the target non-contiguous memory area of ​​the target bandwidth memory region. This solution addresses the shortcomings of low tensor data access efficiency and insufficient cross-architecture reusability in existing technologies, enabling tensor data access based on the TPU processor while improving both access efficiency and accuracy.

[0156] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0157] It should be noted that all information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data comply with the relevant laws, regulations and standards of the relevant regions.

[0158] It should be noted that any arrangement or combination of the technical features in the above embodiments also falls within the protection scope of this invention.

[0159] Example 7

[0160] Figure 7 This is a schematic diagram of a tensor data access device based on a TPU processor provided in Embodiment 7 of the present invention, as shown below. Figure 7 As shown, the device is configured on a TPU processor and includes: a tensor data association attribute information determination module 710, a hardware structure feature acquisition module 720, and a tensor data access module 730, wherein:

[0161] Tensor data association attribute information determination module 710 is used to determine the tensor data association attribute information of the tensor data to be accessed in the TPU processor.

[0162] The hardware structure feature acquisition module 720 is used to acquire the hardware structure features of the TPU processor.

[0163] Tensor data access module 730 is used to determine the tensor data access scenario based on the tensor data association attribute information, and to access the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor.

[0164] This invention, through embodiments thereof, determines the tensor data association attribute information of the tensor data to be accessed in the TPU processor and obtains the hardware structure characteristics of the TPU processor. Further, it determines the tensor data access scenario based on the tensor data association attribute information, and accesses the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. This solution addresses the shortcomings of low tensor data access efficiency and insufficient cross-architecture reusability in existing technologies, enabling tensor data access based on the TPU processor while improving the efficiency and accuracy of tensor data access.

[0165] Optionally, if the tensor data access scenario includes a general continuous transformation access scenario, the tensor data access module 730 is specifically configured to: determine the target memory offset of the tensor data to be accessed based on the tensor metadata and the hardware structure characteristics of the TPU processor; load the tensor data to be accessed from the target bandwidth memory region to the first target cache region of the TPU processor based on the target memory offset; aggregate the tensor data to be accessed based on the offset index of the tensor data to be accessed in the first target cache region to determine the continuous transformation result of the tensor data to be accessed in the first target cache region; use the continuous transformation result of the tensor data to be accessed in the first target cache region as the target output tensor data, and write the target output tensor data back to the target continuous memory region of the target bandwidth memory region.

[0166] Optionally, the tensor data access module 730 is further configured to: generate a linear index of the target output tensor data based on the TPU vector register memory; determine the original memory offset of the tensor data to be accessed according to the tensor data and the linear index of the target output tensor data; align the memory offset of the tensor data to be accessed according to the byte alignment requirements in the hardware structure features of the TPU processor to obtain the target memory offset of the tensor data to be accessed.

[0167] Optionally, if the tensor data access scenario includes a first branch continuation transformation access scenario, the tensor data access module 730 is further configured to: determine the first transmission parameters of the tensor data to be accessed based on the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure features of the TPU processor; load the second target dimension data in the tensor data to be accessed from the target bandwidth memory region to the second target cache region of the TPU processor based on the first transmission parameters; use the second target dimension data in the second target cache region as the target output tensor, and write the target output tensor back to the target continuation memory region of the target bandwidth memory region.

[0168] Optionally, if the tensor data access scenario includes a second branch continuous transformation access scenario, the tensor data access module 730 is further configured to: determine the second transmission parameters of the tensor data to be accessed based on the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure features of the TPU processor; asynchronously load the tensor data to be accessed from the target bandwidth memory region to the input area of ​​the first target cache region of the TPU processor based on the second transmission parameters of the tensor data to be accessed; write the tensor data to be accessed in blocks and loops into the transposed cache channel based on the attribute information of the transposed cache channel in the hardware structure features of the TPU processor; transpose and rearrange the tensor data to be accessed in the transposed cache channel according to the data mapping rules to obtain transposed continuous tensor data; and write the transposed continuous tensor data to the output area of ​​the first target cache region; use the transposed continuous tensor data as the target output tensor; and write the target output tensor back to the target continuous memory region of the target bandwidth memory region.

[0169] Optionally, if the tensor data access scenario includes an ordered discontinuous transformation access scenario, the tensor data access module 730 is further configured to: determine a first nonlinear index of the target output tensor data based on the tensor metadata of the tensor data to be accessed; split the tensor data to be accessed based on the computing core, and load the split tensor data to be accessed into the source data current area corresponding to each computing core in the third target cache region of the TPU processor; reorganize the tensor data to be accessed in each of the source data current areas according to the first nonlinear index to obtain first discontinuous tensor data, and load the first discontinuous tensor data into each target data current area of ​​the third target cache region; use the first discontinuous tensor data as the target output tensor data, and write the target output tensor data back to the target discontinuous memory area of ​​the target bandwidth memory region.

[0170] Optionally, if the tensor data access scenario includes an unordered, discontinuous transformation access scenario, the tensor data access module 730 is further configured to: obtain a second nonlinear index of the target output tensor data; load a first batch of tensor data to be accessed into the source data current area of ​​the third target cache region in the TPU processor, and load a second batch of tensor data into the source data preloading area of ​​the third target cache region; reorganize the first batch of tensor data in the source data current area according to the second nonlinear index to obtain the first batch of discontinuous tensor data, and load the first batch of discontinuous tensor data into the first... The target data current area of ​​the three target cache regions; the first batch of non-contiguous tensor data in the target data current area is transferred to the target non-contiguous memory area; the source data preload area storing the second batch of tensor data is used as the updated source data current area, and the source data current area is used as the updated source data preload area. According to the byte alignment requirements in the hardware structure features of the TPU processor, the tensor data to be accessed is loaded in batches into the updated source data current area and the updated source data preload area until all the tensor data to be accessed is reassembled and written back to the target non-contiguous memory area of ​​the target bandwidth memory area.

[0171] The aforementioned TPU processor-based tensor data access device can execute the TPU processor-based tensor data access method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment can be found in the TPU processor-based tensor data access method provided in any embodiment of the present invention.

[0172] Since the TPU processor-based tensor data access device described above is capable of executing the TPU processor-based tensor data access method in the embodiments of the present invention, those skilled in the art can understand the specific implementation and various variations of the TPU processor-based tensor data access device in this embodiment based on the TPU processor-based tensor data access method described in the embodiments of the present invention. Therefore, how the TPU processor-based tensor data access device implements the TPU processor-based tensor data access method in the embodiments of the present invention will not be described in detail here. Any device used by those skilled in the art to implement the TPU processor-based tensor data access method in the embodiments of the present invention falls within the scope of protection of this application.

[0173] Example 8

[0174] Figure 8A schematic diagram of an electronic device 10, which can be used to implement embodiments of the present invention, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0175] like Figure 8 As shown, the electronic device 10 includes at least one TPU processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one TPU processor 11. The memory stores computer programs executable by the at least one TPU processor. The TPU processor 11 can perform various appropriate actions and processes based on the computer programs stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The TPU processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0176] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0177] The TPU processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the TPU processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The TPU processor 11 performs the various methods and processes described above, such as the tensor data access method based on the TPU processor.

[0178] In some embodiments, the TPU processor-based tensor data access method can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by TPU processor 11, one or more steps of the TPU processor-based tensor data access method described above can be performed. Alternatively, in other embodiments, TPU processor 11 can be configured to perform the TPU processor-based tensor data access method by any other suitable means (e.g., by means of firmware).

[0179] Optionally, a tensor data access method based on a TPU processor, applied to a TPU processor, may include: determining tensor data association attribute information of tensor data to be accessed in the TPU processor; obtaining the hardware structure characteristics of the TPU processor; determining a tensor data access scenario based on the tensor data association attribute information; and accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor.

[0180] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable TPU processor, which may be a dedicated or general-purpose programmable TPU processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0181] Computer programs used to implement the methods of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a TPU processor of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the TPU processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0182] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, inspection, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0183] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0184] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0185] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0186] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this invention disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this invention can be achieved, and this is not limited herein.

[0187] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for accessing tensor data based on a TPU processor, characterized in that, Applied to TPU processors, including: Determine the tensor data association attribute information of the tensor data to be accessed in the Tensor Processing Unit (TPU); wherein, the tensor data association attribute information includes the tensor characteristics, storage rules, operation constraints, and adaptation information of the tensor data to be accessed; Obtain the hardware structure features of the TPU processor; The tensor data access scenario is determined based on the tensor data association attribute information, and the tensor data to be accessed is accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor. The tensor data access scenario is determined based on the continuity of the memory layout of the tensor data to be accessed and the access purpose. The tensor data access scenario includes a continuous conversion access scenario and a non-contiguous conversion access scenario. The continuous conversion access scenario is a scenario where the tensor data to be accessed with a non-contiguous memory layout is converted to a contiguous memory layout before data access. The non-contiguous conversion access scenario is a scenario where the tensor data to be accessed with a contiguous memory layout is converted to a non-contiguous memory layout before data access. The tensor metadata includes a set of auxiliary information describing the core attributes of the tensor data to be accessed but excluding the original numerical data of the tensor data to be accessed.

2. The method according to claim 1, characterized in that, If the tensor data access scenario includes a general continuous transformation access scenario, then accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor includes: The target memory offset of the tensor data to be accessed is determined based on the tensor data and the hardware structure characteristics of the TPU processor. The tensor data to be accessed is loaded from the target bandwidth memory region into the first target cache region of the TPU processor according to the target memory offset; The tensor data to be accessed is aggregated based on the offset index of the tensor data to be accessed in the first target cache region to determine the continuous transformation result of the tensor data to be accessed in the first target cache region; The continuous transformation result of the tensor data to be accessed in the first target cache region is used as the target output tensor data, and the target output tensor data is written back to the target continuous memory region of the target bandwidth memory region.

3. The method according to claim 2, characterized in that, Determining the target memory offset of the tensor data to be accessed based on the tensor metadata and the hardware architecture characteristics of the TPU processor includes: A linear index for generating the target output tensor data based on the TPU vector register memory; The original memory offset of the tensor data to be accessed is determined based on the linear index of the tensor metadata and the target output tensor data. The memory offset of the tensor data to be accessed is aligned according to the byte alignment requirements in the hardware structure features of the TPU processor to obtain the target memory offset of the tensor data to be accessed.

4. The method according to claim 1, characterized in that, If the tensor data access scenario includes a first branch continuous transformation access scenario, then accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor includes: The first transmission parameters of the tensor data to be accessed are determined based on the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure features of the TPU processor. According to the first transmission parameters, the second target dimension data in the tensor data to be accessed is loaded from the target bandwidth memory region to the second target cache region of the TPU processor; The second target dimension data in the second target cache region is used as the target output tensor, and the target output tensor is written back to the target contiguous memory region of the target bandwidth memory region.

5. The method according to claim 1, characterized in that, If the tensor data access scenario includes a second branch continuous transformation access scenario, then accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor includes: The second transmission parameters of the tensor data to be accessed are determined based on the tensor metadata of the first target dimension data in the tensor data to be accessed and the byte alignment requirements in the hardware structure features of the TPU processor. According to the second transmission parameters of the tensor data to be accessed, the tensor data to be accessed is asynchronously loaded from the target bandwidth memory region to the input area of ​​the first target cache region of the TPU processor; According to the attribute information of the transposed cache channel in the hardware structure features of the TPU processor, the tensor data to be accessed is written into the transposed cache channel in blocks and loops. According to the data mapping rules, the tensor data to be accessed in the transposed cache channel is transposed and rearranged to obtain transposed continuous tensor data. The transposed continuous tensor data is then written to the output area of ​​the first target cache area. The transposed continuous tensor data is used as the target output tensor, and the target output tensor is written back to the target continuous memory region of the target bandwidth memory region.

6. The method according to claim 1, characterized in that, If the tensor data access scenario includes an ordered discontinuous transformation access scenario, then accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor includes: Determine the first nonlinear index of the target output tensor data based on the tensor metadata of the tensor data to be accessed; The tensor data to be accessed is split based on the computing core, and the split tensor data to be accessed is loaded into the source data current area corresponding to each computing core in the third target cache area of ​​the TPU processor. The tensor data to be accessed in each of the current regions of the source data are reorganized according to the first nonlinear index to obtain the first non-continuous tensor data, and the first non-continuous tensor data is loaded into each of the current regions of the target data in the third target cache region. The first non-contiguous tensor data is used as the target output tensor data, and the target output tensor data is written back to the target non-contiguous memory region of the target bandwidth memory region.

7. The method according to claim 1, characterized in that, If the tensor data access scenario includes an unordered, discontinuous transformation access scenario, then accessing the tensor data to be accessed based on the tensor data access scenario, the tensor metadata in the tensor data association attribute information, and the hardware structure characteristics of the TPU processor includes: Obtain the second nonlinear index of the target output tensor data; The first batch of tensor data to be accessed is loaded into the source data current area of ​​the third target cache region in the TPU processor, and the second batch of tensor data is loaded into the source data preload area of ​​the third target cache region. The first batch of tensor data in the current area of ​​the source data is reorganized according to the second nonlinear index to obtain the first batch of non-continuous tensor data, and the first batch of non-continuous tensor data is loaded into the current area of ​​the target data in the third target cache area. The first batch of non-contiguous tensor data in the current region of the target data is transferred to the target non-contiguous memory region. The source data preload area storing the second batch of tensor data is used as the updated source data current area, and the source data current area is used as the updated source data preload area. According to the byte alignment requirements in the hardware structure features of the TPU processor, the tensor data to be accessed is loaded in batches into the updated source data current area and the updated source data preload area until all the tensor data to be accessed is reassembled and written back to the target non-contiguous memory area of ​​the target bandwidth memory region.

8. An electronic device, characterized in that, The electronic device includes: At least one TPU processor; and A memory communicatively connected to the at least one TPU processor; wherein, The memory stores a computer program executed by the at least one TPU processor, which enables the at least one TPU processor to perform the TPU processor-based tensor data access method according to any one of claims 1-7.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause the TPU processor to execute and implement the tensor data access method based on any one of claims 1-7.

10. A computer program product, characterized in that, It includes a computer program / instruction, wherein the computer program / instruction, when executed by a TPU processor, implements the TPU processor-based tensor data access method according to any one of claims 1-7.

Images