Tensor operation device, method, storage medium, and program product

By employing an architecture with multiple tensor cores, independent memory access control modules, and a shared transmission path in GPGPU, the problem of high memory access latency of a single large-scale tensor core is solved, achieving efficient tensor computation task decomposition and parallel processing, thereby improving computational efficiency and resource utilization.

CN123115579APending Publication Date: 2026-11-17ZHIHAOTONG (TIANJIN) INFORMATION TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611308446.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-08-26
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

The single large-scale tensor core architecture in existing GPGPUs is insufficient in masking memory access latency during large-scale tensor computations, resulting in high memory access latency and reduced core computing power utilization.

Method used

The architecture employs multiple tensor cores with independent memory access control modules and memory access queues. By splitting tensor computation tasks into multiple sub-computation tasks along a preset dimension, these tasks are processed in parallel by multiple tensor cores. Data transmission is optimized through a shared transmission path, thereby reducing memory access latency.

Benefits of technology

It significantly improves the ability to mask memory access latency, enhances the computing power utilization and overall computing efficiency of the tensor core, and reduces hardware resources and power consumption.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123115579A_ABST
    Figure CN123115579A_ABST
Patent Text Reader

Abstract

This application relates to the field of artificial intelligence accelerator chip technology, and in particular to a tensor computation device, method, storage medium, and program product. It aims to address the problem of high memory access latency in existing general-purpose graphics processors using a single large-scale tensor core. This application proposes a tensor computation scheme that splits the same tensor computation task into multiple subtasks along a preset dimension, which are processed separately by multiple tensor cores. Tensor data is retrieved in parallel from a remote storage medium through an independent memory access control module and memory access queue corresponding to each tensor core. After each tensor core completes its computation, the output is written to an accumulation storage unit for accumulation, yielding the final computation result of the tensor computation task. This split architecture, employing multiple tensor cores with independent memory access control modules, only slightly increases the total hardware area, yet enables parallel initiation of multiple memory access requests, effectively improving memory access latency masking capabilities and the throughput and computational efficiency of tensor parallel computation.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application relates to the field of artificial intelligence accelerator chip technology, and in particular to a tensor computing device, method, storage medium and program product. Background Technology

[0002] In large language model (LLM) training and inference scenarios, existing GPGPUs (General-Purpose computing on Graphics Processing Units) typically rely on a single tensor core to perform matrix operations and mitigate remote memory access latency by expanding the ability to request concurrent memory accesses.

[0003] However, for a single large-scale tensor core architecture, only expanding the queue concurrency limit can alleviate remote storage access latency, resulting in a low optimization ceiling and still relatively high memory access latency. Longer memory access latency causes tensor cores to frequently be in a data waiting state, significantly reducing core computing power utilization.

[0004] Therefore, improving the memory access latency masking capability of tensor cores has become an urgent technical problem to be solved. Summary of the Invention

[0005] This application provides a tensor computation device to solve the aforementioned technical problems. This tensor computation device is designed for tensor computation scenarios in general-purpose graphics processors. By adaptively splitting tensor computation tasks and collaboratively scheduling multiple tensor cores for parallel processing, it significantly reduces memory access latency and improves tensor core utilization and tensor computation efficiency.

[0006] In a first aspect, embodiments of this application provide a tensor computation device deployed on a general-purpose graphics processor. The tensor computation device includes: an accumulation storage unit, multiple tensor cores, a remote storage medium, and memory access control modules corresponding to each tensor core connected to the remote storage medium. Each tensor core is configured to receive sub-computation tasks, generate memory access requests for the sub-computation tasks, and send them to the corresponding memory access control module. The sub-computation tasks received by the multiple tensor cores are obtained by splitting the same tensor computation task along a preset dimension. Each memory access control module corresponds to an independent memory access queue and is configured to queue the memory access requests received from the corresponding tensor core, read tensor data from the remote storage medium according to the memory access queue, and return to the corresponding tensor core. Each tensor core is configured to perform operations on the read tensor data to obtain an output vector and write the output vector into the accumulation storage unit. The accumulation storage unit is configured to accumulate the newly written output vector to obtain the computation result of the tensor computation task.

[0007] In one possible embodiment, the memory access control module includes an incomplete memory access transaction circuit, which is equipped with a memory access queue for caching memory access requests for tensor data that have been initiated but not yet returned.

[0008] In one possible embodiment, a shared transmission path extending along a preset direction is also included; the shared transmission path is configured to transmit the common tensor data of the upstream tensor core to the downstream tensor core, so that the downstream tensor core performs operations on the common tensor data and the read tensor data to obtain the operation output vector.

[0009] In one possible embodiment, the number of tensor cores is a power of 2, where N is an integer greater than 1.

[0010] In a second aspect, embodiments of this application provide a tensor operation method, applied to the tensor operation apparatus provided in the first aspect of this application, the method comprising:

[0011] The tensor computation task is divided into multiple sub-computation tasks along a preset dimension. Each sub-computation task is then distributed to a tensor core, which generates memory access requests for each sub-computation task. The received memory access requests are queued to obtain a memory access queue. Tensor data is read from the remote storage medium according to the memory access queue, and the read tensor data is returned to the corresponding tensor core. Each tensor core then performs operations on the read tensor data to obtain an output vector. The output vector is written to an accumulation storage unit. The newly written output vector is accumulated in the accumulation storage unit to obtain the computation result of the tensor computation task.

[0012] In one possible embodiment, the method further includes:

[0013] The common tensor data is transmitted to the downstream tensor core through a shared transmission path, so that the downstream tensor core can perform operations on the tensor data read from the remote storage medium and the common tensor data to obtain the operation output vector.

[0014] In one possible embodiment, the tensor computation task is divided into multiple sub-computation tasks along a preset dimension, including:

[0015] The tensor computation task is split into a preset number of sub-computation tasks along a preset dimension; the preset number is the number of tensor cores; or, the number of splits is determined based on the data size of the tensor computation task; the number of splits is less than or equal to the preset number; the tensor computation task is split into the preset number of sub-computation tasks along the preset dimension.

[0016] In one possible embodiment, after splitting the tensor computation task into a predetermined number of sub-computation tasks along a preset dimension and distributing each sub-computation task to each tensor core, the method further includes:

[0017] If the number of splits is less than the preset number, the hardware resources of the memory access control module corresponding to the tensor core of the unallocated sub-computation task will be released.

[0018] Thirdly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods provided above.

[0019] Fourthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the methods provided above.

[0020] The tensor computation apparatus, method, storage medium, and program product provided in this application split the same tensor computation task into multiple sub-computation tasks along a preset dimension, and have multiple tensor cores process the corresponding sub-computation tasks respectively. Each tensor core is configured with an independent memory access control module and an independent memory access queue connected to a remote storage medium, thereby achieving memory access pressure distribution. Compared with the architecture of a single large-scale tensor core and a large-capacity concurrent memory access queue, the parallelism of memory access operations and computation processes is improved, thereby improving the remote storage memory access latency masking effect, improving the computing power utilization of tensor cores, and reducing the overhead of control logic and cache resources. Attached Figure Description

[0021] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0022] Figure 1 A block diagram of the structure of a general-purpose graphics processor in existing technology;

[0023] Figure 2 A schematic diagram of a tensor computing device provided in this application;

[0024] Figure 3 A schematic diagram of another tensor computing device provided in this application;

[0025] Figure 4 This is a flowchart illustrating a tensor operation method provided in this application.

[0026] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0027] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application.

[0028] Tensor computing and graphics processor architecture optimization techniques are widely used in training and inference scenarios for large language models. In such scenarios, general-purpose graphics processors typically undertake large-scale matrix or tensor operations and rely on on-chip tensor cores and remote storage media to collaboratively complete data reading, computation, and result writing.

[0029] In distributed training or high-throughput inference systems, tensor computation tasks often require continuously retrieving weight and activation data from high-bandwidth memory or other remote storage media. In this case, the execution efficiency of the tensor core depends not only on the computing power itself but also on the memory access path.

[0030] Figure 1 A block diagram of the structure of a general-purpose graphics processor in existing technology, such as Figure 1 As shown, in existing solutions, general-purpose graphics processors typically employ a single large-scale tensor core to execute complete tensor computation tasks. They mask the access latency of remote storage media by expanding concurrent memory access request capabilities, such as a memory access control module with 1024 OTS (Outstanding Transaction Slots). The basic approach is for the tensor core to generate multiple memory access requests for the matrix operations to be processed. These requests are then queued by a unified memory access control logic, the required tensor data is read from the remote storage media, and the data is returned to the tensor core to complete the computation.

[0031] While such solutions can improve memory access concurrency to some extent, supporting a higher number of incomplete memory access requests often requires configuring large-capacity memory access queues, along with corresponding control logic and cache resources, significantly increasing hardware area and power consumption. At the same time, a single large-scale tensor kernel exhibits poor scheduling flexibility when facing computational tasks of varying sizes or granularities. If the latency of remote storage access fluctuates or increases, the computation process may still idle while waiting for data to return.

[0032] Furthermore, because memory access requests are concentrated in a limited number of control paths, latency masking capabilities are limited, making it difficult to achieve a stable match between computing resources and storage bandwidth. This not only reduces tensor kernel utilization but also restricts overall training throughput and inference efficiency in large language model scenarios, which is particularly prominent under conditions of limited hardware resources.

[0033] In view of this, how to improve the memory access latency masking capability of tensor computation has become an urgent technical problem to be solved. To solve the above problem, this application proposes a tensor computation device deployed on a general-purpose graphics processor. Through multiple tensor cores, a memory access control module corresponding to each tensor core and having an independent memory access queue, a remote storage medium, and an accumulation storage unit, multiple sub-computation tasks of the same tensor computation task, which are split along a preset dimension, can be initiated and executed by multiple tensor cores respectively. The output vector of the operation is then written to and accumulated in the accumulation storage unit, thereby obtaining the computation result of the tensor computation task, which significantly improves the latency masking capability.

[0034] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0035] Figure 2 This is a schematic diagram of a tensor computation device provided in this application. The tensor computation device is deployed on a general-purpose graphics processor, such as... Figure 2 As shown, the tensor operation device includes: an accumulation storage unit, multiple tensor cores, a remote storage medium, and a memory access control module corresponding to each tensor core connected to the remote storage medium.

[0036] The tensor core is configured to receive sub-computation tasks, generate memory access requests for these tasks, and send them to the corresponding memory access control module. Multiple tensor cores receive sub-computation tasks by splitting the same tensor computation task along a preset dimension. Each memory access control module corresponds to an independent memory access queue, configured to queue the memory access requests received from the corresponding tensor core, read tensor data from the remote storage medium according to the queue, and return the data to the corresponding tensor core. The tensor core is configured to perform operations on the read tensor data to obtain an output vector and write the output vector to an accumulation storage unit. The accumulation storage unit is configured to accumulate the newly written output vector to obtain the computation result of the tensor computation task.

[0037] In some embodiments, a plurality of tensor cores are arranged along a preset direction.

[0038] Tensor cores can be composed of digital multiply-accumulate arrays, which are hardware-fixed tensor operation circuits used to perform basic tensor operations in neural networks such as matrix multiplication and addition. Each tensor core is an independent hardware unit with a built-in data buffer register, which can temporarily store the tensor data read and intermediate vectors for operation.

[0039] The memory access control module is a data logic circuit that integrates a FIFO-style memory access queue to store unfinished memory access requests. It includes address decoding, bus interaction, and request sorting logic. The memory access request queue storage space of each memory access control module is not shared.

[0040] For example, the memory access control module can be implemented using a missing state holding register array, a multi-level FIFO scheduling circuit with matching flags, a sliced ​​transaction tracking register file, or a pipelined transaction state machine.

[0041] The accumulation storage unit can be an on-chip integrated high-speed register file or an on-chip SRAM storage array, equipped with a hardware addition tree circuit, which has real-time accumulation and writing function and can continuously perform superposition operations on multiple input vectors.

[0042] The remote storage medium is an off-chip high-speed storage hardware that is connected to each memory access control module via an on-chip high-speed transmission bus to uniformly store complete tensor data.

[0043] Each tensor core is connected to its own memory access control module via a dedicated on-chip control bus and data bus. The links between different tensor cores and their corresponding memory access control modules are physically isolated and do not interfere with each other.

[0044] Each memory access control module is connected in parallel to the same remote storage high-speed transmission bus, and can independently and simultaneously initiate data read operations to the remote storage medium, realizing the parallel transmission of multiple memory access requests.

[0045] The operation output ports of each tensor core are connected to the data write ports of the accumulator storage unit through a multi-channel convergence bus, and each tensor core can independently write operation output vectors to the accumulator storage unit.

[0046] Unlike traditional single large-scale tensor cores, this embodiment employs multiple small tensor cores to achieve parallel processing of multiple sub-computation tasks from a complete tensor computation task. During tensor computation task decomposition, the complete large-size tensor computation task can be evenly divided into multiple sub-computation tasks along a preset dimension, and each sub-computation task can be assigned to a tensor core.

[0047] In one possible implementation, the number of tensor cores is a power of 2, where N is an integer greater than 1.

[0048] Setting the number of tensor cores to a power of 2 allows multiple tensor cores to form a regular structure in array arrangement, task partitioning, and address mapping, thus facilitating balanced splitting and parallel scheduling along a preset direction.

[0049] In some embodiments, N is an integer greater than or equal to 4.

[0050] Because this numerical form is naturally compatible with binary encoding, it makes it easier to achieve symmetrical configuration of the memory access control modules corresponding to each tensor core, thereby reducing control complexity. At the same time, the number of powers of 2 is beneficial for the size planning and boundary alignment of shared transmission paths, accumulated storage units, and memory access queues. Therefore, it can reduce additional hardware overhead while ensuring scalability and improve throughput stability and resource utilization during multi-core collaborative computing.

[0051] For example, considering the matrix multiplication tensor in the reasoning process of a large language model, the tensor dimension is N×K, and N is selected as the preset dimension. Assuming there are a total of 16 tensor cores, they are equally divided along the N dimension to obtain 16 sub-computation tasks, each with a corresponding dimension of (N / 16)×K. Each sub-computation task is distributed to different tensor cores for independent execution. Each tensor core relies on a dedicated memory access control module to read the corresponding tensor fragment data in parallel. After the operation is completed, the output vector is uniformly sent to the accumulation storage unit to complete the result fusion.

[0052] In one possible implementation, the memory access control module includes an Outstanding Transaction Circuit, which has a memory access queue for caching memory access requests for tensor data that have been initiated but have not yet returned as read requests, i.e., incomplete memory access requests.

[0053] The incomplete memory access transaction circuit has a built-in hardware FIFO queue register array. The queue has independent storage space, and the queue depth can be set according to the hardware design. After the tensor core sends a memory access request to read tensor data to the remote storage medium, there is a certain delay in the return of the tensor data from the remote storage medium. Before the tensor data returns, its memory access request (which can be simply referred to as the memory access request) is stored in the memory access queue for continuous tracking. The incomplete memory access transaction circuit can send multiple new memory access requests simultaneously without waiting for the data of the previous request to return, thereby initiating multiple memory access operations in parallel.

[0054] The incomplete memory access transaction circuit can continuously track data read requests that have been initiated by the tensor core but have not yet returned, and save the initiation status, return order and correlation of each request with the help of the memory access queue, so that the memory access control module can maintain a stable number of requests in transit even when there is access latency on the remote storage medium.

[0055] This avoids frequent pauses in the tensor core due to a single read not returning, thus masking memory access latency more effectively and improving the continuity of tensor data supply. Therefore, in a system where multiple tensor cores are configured with independent memory access queues, concurrent memory access organization capabilities can be improved without excessively expanding the size of the unified queue, thereby balancing tensor computation throughput, hardware area, and power consumption control.

[0056] The tensor computation device provided in this embodiment splits the same tensor computation task into multiple sub-computation tasks along a preset dimension, which can be processed in parallel by multiple tensor cores, thus avoiding a single large-scale tensor core bearing all the computation. Each tensor core corresponds to an independent memory access control module and an independent memory access queue, enabling memory access requests to be queued and initiated in parallel, thereby improving the ability to mask the latency of remote storage media access, thus improving the remote storage latency masking effect and increasing the computing power utilization of the tensor core. After the data is returned, the tensor cores complete the computation and output vectors, and the accumulating storage unit continuously accumulates the newly written results, thereby achieving efficient aggregation of sub-task results. Therefore, under limited hardware resources, it can balance memory access concurrency, computational utilization, and implementation cost, improving the overall throughput efficiency of tensor computation tasks.

[0057] Figure 3 A schematic diagram of another tensor operation device provided in this application is shown below. Figure 3 As shown, based on the aforementioned embodiments, the tensor operation device further includes a shared transmission path extending along a preset direction; the shared transmission path is configured to transmit the common tensor data of the upstream tensor core to the downstream tensor core, so that the downstream tensor core performs operations on the common tensor data and the read tensor data to obtain the operation output vector.

[0058] The preset direction is the direction in which the tensor cores are arranged, which can be either horizontal or vertical.

[0059] The shared transmission path is a dedicated unidirectional hardware transmission bus on-chip, with integrated wiring along the preset direction of the tensor core. It is a fixed metal trace hardware integrated inside the tensor computing unit of the general-purpose graphics processor.

[0060] The shared transmission path has a built-in data latch register for temporarily caching the common tensor vector output from the upstream, ensuring stable data transmission downstream. The path is designed for unidirectional transmission, meaning data can only be transferred from the upstream tensor core at the front of the path to the downstream tensor core at the back. The shared transmission path is an independent hardware line and does not occupy the memory access bus between the memory access control module and the remote storage; data forwarding does not consume the bandwidth of the remote storage.

[0061] A shared transmission path is used to enable data reuse between adjacent tensor cores. Since a complete tensor computation task is broken down into multiple sub-computation tasks, and matrix multiplication naturally involves a uniform one-dimensional common vector, this common vector is essential data for all tensor cores to perform operations. If each tensor core independently reads this common vector from remote storage, it would generate a large number of duplicate memory access requests, consuming bandwidth and exacerbating memory access latency. Therefore, this embodiment introduces a shared transmission path to reuse this common vector, i.e., the common tensor data.

[0062] The tensor data requested by the first tensor core includes common tensor data. The tensor data requested by other tensor cores does not include this common tensor data; instead, they reuse the common tensor data read by the upstream tensor core through a shared transmission path. The upstream tensor core being read by a tensor core is located in front of it along a preset direction.

[0063] After the upstream tensor core reads the common tensor data, it transmits the common tensor data unidirectionally to the downstream tensor core through the shared transmission path. The downstream tensor core directly reuses the common tensor data to participate in the calculation, eliminating repeated remote read operations, reducing memory access pressure, and improving overall computing efficiency.

[0064] By establishing a shared transmission path extending along a preset direction, the common tensor data already acquired by the upstream tensor core can be directly reused by the downstream tensor core, thereby reducing the need for repeated memory accesses to similar common tensor data by the downstream tensor core. After receiving the shared common tensor data, the downstream tensor core only needs to combine it with its own read tensor data to complete the corresponding operation, enabling data reuse and pipelined collaboration among multiple tensor cores. This further reduces the pressure on remote storage media access, shortens data waiting time, and, in conjunction with the mechanism of independent memory access queues for each tensor core, improves the ability to mask memory access latency. Therefore, it can improve tensor core utilization and overall tensor computation throughput with relatively small hardware overhead, making it more suitable for high-frequency tensor operation scenarios in large language model training and inference.

[0065] Figure 4 This is a flowchart illustrating a tensor operation method provided in this application, applied to the tensor operation apparatus provided in the foregoing embodiments, such as... Figure 2 or Figure 3 The tensor operation device shown is as follows: Figure 4 As shown, the method includes:

[0066] Step S401: Divide the tensor computation task into multiple sub-computation tasks along a preset dimension.

[0067] The preset dimension can be the dimension of the rows or columns of the matrix corresponding to the tensor computation task.

[0068] For example, taking a tensor computation task with a computation size of 256×256×16 as an example, for instance, to calculate the product of a 256×16 dimension matrix A and a 16×256 dimension matrix B, the tensor computation task can be evenly divided along the dimension where "16" is located, resulting in 16 sub-computation tasks of size 256×256×1. Each sub-task performs the multiplication operation of a 256×1 piecewise matrix and a 1×256 piecewise matrix respectively.

[0069] By splitting tasks, computational tasks that were originally concentrated on a single large-scale tensor core are distributed across multiple tensor cores. Each core can then independently initiate memory access requests. As a result, there are fluctuations in the access latency of remote storage media. This causes some tensor cores to be in a data waiting state while others are still performing tensor operations synchronously. This allows the computation process and memory access waiting to overlap in the time dimension, effectively masking the memory access latency caused by remote storage and improving the overall computing power utilization of tensor cores.

[0070] Step S402: Distribute each sub-computation task to each tensor core so that each tensor core can generate memory access requests for each sub-computation task.

[0071] Each sub-computation task after being split is assigned to a tensor core, and different sub-computation tasks are assigned to different tensor cores.

[0072] The task scheduling module can execute the steps of splitting and distributing tensor computation tasks, namely, executing steps S401 and S402. The sub-computation tasks can be sent to the corresponding tensor cores through the on-chip control bus or dedicated distribution link.

[0073] Sub-computation tasks can be encapsulated as task descriptors and sent to the corresponding tensor cores. The task descriptor can include the sub-computation task number, input tensor address range, access stride, block size, output vector address offset, data format, operation type, and end flag. Upon receiving the task descriptor, the tensor core writes it to its local task register set or task cache and generates a memory access request based on the data block shape defined in the sub-computation task. The memory access request can include a read request identifier, target address, burst length, data block sequence number, and a corresponding return destination identifier, used to route the data back to the tensor core that initiated the request after the data is returned from the remote storage medium.

[0074] This step establishes a clear mapping between sub-computation tasks and tensor cores, and each tensor core independently generates requests, forming multiple parallel memory access initiation paths. Compared to a structure where a single control path uniformly handles all requests, this approach does not require building a very large centralized queue to maintain a large number of incomplete requests, thus maintaining a high memory access coverage depth with limited hardware resources.

[0075] Step S403: Queue the received memory access requests to obtain a memory access queue, read tensor data from the remote storage medium according to the memory access queue, and return the read tensor data to the corresponding tensor core, so that each tensor core can perform operations on the read tensor data to obtain the operation output vector.

[0076] Before queuing, memory access requests can be validated for validity, including address alignment, burst length exceeding the limit, request type being read access, and task identifier validity. Once validated, the memory access request is written to the memory access queue within the memory access control module. The memory access queue can use a first-in, first-out (FIFO) structure, or it can add a priority field to queue entries and dequeue according to priority. After a memory access request is dequeued, the memory access control module generates a standard read transaction for the remote storage medium based on the target address and burst length in the memory access request, and sends it to the remote storage medium via an off-chip controller or storage interconnect.

[0077] When tensor data is returned from the remote storage medium, the memory access control module writes the read tensor data into the local return buffer according to the tag information in the return transaction, and then sends the cached tensor data back to the corresponding tensor core according to the return destination identifier carried in the request. The returned tensor data can be an activation vector, a weight matrix block, or other data fragments involved in tensor operations.

[0078] If the returned data arrives in multiple frames, the memory access control module concatenates them in the buffer until a complete data block is formed before notifying the tensor core to retrieve the data. If the tensor core supports streaming reception, the memory access control module can also push data in frames simultaneously with the arrival of the returned data, allowing the computation pipeline to start ahead of time. Since each tensor core has an independent memory access control module and memory access queue, the queuing depth, return latency, or momentary congestion of a request for a particular core will not directly block the request initiation path of other cores.

[0079] Based on the above analysis, this step transforms memory access queuing from centralized to distributed processing. Multiple smaller, independent memory access queues serve their respective tensor cores. With a relatively uniform number of incomplete requests, the control logic is distributed across multiple modules, reducing the probability of single-path congestion. Furthermore, multiple memory access control modules initiate reads from the remote storage medium in parallel, ensuring that some cores continue to receive data even when remote access latency fluctuates, thus supporting subsequent parallel computing processes.

[0080] After receiving tensor data returned from the corresponding memory access control module, or after reading tensor data from the remote storage medium, the tensor core completes tensor computation according to the operation rules specified in the sub-computation task or its task descriptor.

[0081] The tensor core, as a computational unit, can include an input buffer, a multiplication array, an addition reduction tree, a result register, and a write-back interface.

[0082] During execution, the tensor core first loads the returned tensor data into the local input buffer according to the data type and matrix block layout. Then, it configures the computation pipeline based on the operation type specified in the task descriptor. If the operation is matrix multiplication and addition, one data block can be used as the left operand matrix fragment, and the other data block as the right operand matrix fragment. The product term is generated through the multiplication array, and then the addition reduction tree forms the operation output vector within the core. If the sub-computation task corresponds to segmented computation on the reduction dimension, the operation output vector output by the core only represents a portion of the accumulated amount of the target operation output vector. In this case, it still needs to be written to the accumulation storage unit and merged with the results generated by other cores or other rounds of the core. If the sub-computation task corresponds to an independent block on the output dimension, the operation output vector generated by the core can directly correspond to a unique output address range, and subsequent write-and-hold or zero-based accumulation is completed by the accumulation storage unit.

[0083] Step S404: Write the operation output vector into the accumulation storage unit.

[0084] The accumulation storage unit is used to receive the operation output vectors generated by each tensor core. The operation output vector can be understood as a partial result or intermediate result generated by the corresponding sub-computation task, and its address range is predetermined when the sub-computation task is issued. Since multiple tensor cores are arranged along a preset direction and each processes its corresponding sub-task, multiple operation output vectors can be successively sent to the accumulation storage unit within the same computation cycle.

[0085] During the write process, the tensor core encapsulates the operation output vector into a write-back transaction based on the address offset of the operation output vector in the task descriptor, and sends it to the accumulation storage unit via the on-chip write-back link. The write-back transaction includes at least the target address, vector length, data content, and valid identifier. When multiple tensor cores may write to the same logical output range in adjacent cycles, the accumulation storage unit receives the write according to the address granularity and prepares an accumulation entry port for the data item corresponding to the same address.

[0086] In one possible embodiment, the tensor core calculates and outputs an operation output vector for each batch of complete input data, and then continuously writes it to the accumulation storage unit in a streaming manner; in another possible embodiment, the tensor core can first accumulate multiple small blocks of results locally, and then synthesize them into a larger vector before writing them all at once.

[0087] This step centrally writes the execution results of each sub-computation task into the accumulation storage unit in the form of operation output vectors, so that the partial results scattered in different tensor cores are structurally converged into a unified result storage domain. In this way, subsequent result merging does not require writing back to the remote storage medium and then reading it again for merging, but instead directly enters the accumulation processing path on the chip, shortening the result aggregation link in terms of data flow.

[0088] Step S405: The newly written operation output vector is accumulated in the accumulation storage unit to obtain the operation result of the tensor computation task.

[0089] The accumulation storage unit is a result aggregation component in a tensor operation device. It receives operation output vectors from multiple tensor cores and performs element-wise accumulation on the newly written operation output vector and the existing data at the target address. The accumulation storage unit may include address decoding logic, read-modify-write paths, an accumulator array, and a result holding storage area.

[0090] In practice, after receiving the output vector written by the tensor core, the accumulation storage unit first locates the corresponding storage unit in the storage area based on the target address in the write-back transaction. If the data corresponding to the target address is the first write of the tensor computation task, the current output vector is written as the initial value; if the target address already has a previously written partial result, element-wise addition is performed between the previous result and the newly written vector, and the addition result is stored back to the same address range. For vector lengths greater than the processing width of a single accumulator, the accumulation storage unit performs multi-phase accumulation in a segmented manner, processing a fixed number of elements in each phase until the entire vector is completed. If multiple tensor cores write to the same target address range close in time, the accumulation storage unit ensures that the update order of the same address range is consistent through internal write arbitration and address conflict management mechanisms, thereby ensuring that the final result is consistent with the mathematical summation relationship of the sub-computation task.

[0091] Regarding task completion determination, based on the total number of sub-computation tasks generated in step S401 and the completion identifier of each sub-computation task, it can be determined whether all corresponding operation output vectors have been written and accumulated. When all sub-computation tasks are completed, the corresponding address range in the accumulation storage unit is marked as the operation result of the tensor computation task.

[0092] By setting up on-chip accumulation storage units to accumulate newly written operation output vectors instantly or in segments, partial results generated in parallel by multiple tensor cores can be directly aggregated to form the operation result of the tensor computation task. When this method is combined with the process of multiple tensor cores accessing memory and performing independent computations, memory access latency masking and result merging are completed simultaneously on-chip, thus completing the tensor computation task without relying on a large-scale centralized memory access queue.

[0093] The tensor computation method provided in this embodiment decomposes the same tensor computation task into multiple parallel sub-computation tasks along a preset dimension, and configures each tensor core with a corresponding memory access control module and an independent memory access queue. This allows multiple tensor cores to initiate reads, perform operations, and aggregate some results into an accumulation storage unit for accumulation. Memory access waits during tensor computation are distributed across multiple independent paths, and result merging is performed directly on-chip. Therefore, even under conditions of fluctuating access latency to remote storage media, some tensor cores can still continuously obtain data and advance computation without significantly expanding the global memory access queue to support the number of incomplete requests. From an overall implementation principle perspective, this method balances computational resource utilization, memory access latency masking capability, and hardware implementation cost.

[0094] Based on the foregoing embodiments, the method further includes:

[0095] The common tensor data is transmitted to the downstream tensor core through a shared transmission path, so that the downstream tensor core can perform operations on the tensor data read from the remote storage medium and the common tensor data to obtain the operation output vector.

[0096] In implementation, after completing the computation on the tensor data read locally, the upstream tensor core stores the common tensor data that still needs to be used by the downstream tensor core in an on-chip temporary register or local cache, and sends it to the adjacent downstream tensor core via a shared transmission path in a preset direction, such as to the right. Upon receiving this common tensor data, the downstream tensor core inputs it together with its own tensor data read from a remote storage medium into a multiply-accumulate unit or tensor operation array to calculate the corresponding output vector. This vector is then written to the accumulation storage unit for subsequent accumulation.

[0097] The shared transmission path can use a fixed-length and wide data cable bundle or a cascaded transmission structure with configurable bandwidth to adapt to tensor core arrays of different sizes.

[0098] Through the above structure, tensor cores can reuse common tensor data along a preset direction, reducing repeated reading and loading of the same data across multiple tensor cores. This allows adjacent tensor cores to continuously perform operations around the shared data and write their respective output vectors into the accumulation storage unit. Thus, common tensor data only needs to be forwarded once or a few times between tensor cores to support computation by multiple downstream cores, thereby reducing repeated memory access pressure and improving the data reuse efficiency and overall execution consistency of the tensor computing device in high-throughput computing scenarios.

[0099] In one possible implementation, the tensor computation task is split into multiple sub-computation tasks along a preset dimension, including: splitting the tensor computation task into a preset number of sub-computation tasks along a preset dimension; the preset number is the number of tensor cores; or, determining the number of splits based on the data scale of the tensor computation task; the number of splits is less than or equal to the preset number; and splitting the tensor computation task into the preset number of sub-computation tasks along the preset dimension.

[0100] When splitting a task, the number of sub-computation tasks can be defaulted to the number of tensor cores contained in a general-purpose graphics processor or tensor computing device, such as 16, 64, or other values. In some embodiments, the number of sub-computation tasks can also be determined based on the data size of the tensor computing task.

[0101] In the specific implementation, after receiving the tensor computation task to be processed, the task scheduling module first determines the splitting direction, i.e., the preset dimension, based on the size of the row or column dimensions of the matrix corresponding to the tensor computation task; then, it determines the number of splits based on the data scale of the tensor computation task; thus, the tensor computation task is split into the corresponding number of sub-computation tasks along the preset dimension.

[0102] When the task data size is large, the number of splits can be determined to be a preset number. When the task data size is small, a number of splits smaller than the preset number can be determined.

[0103] In one implementation, the split sub-computation tasks maintain the same operation type and data alignment rules, distinguishing each other only in terms of data index range, thus enabling each tensor core to independently generate memory access requests and complete local computations. If a preset number of sub-computation tasks is used, each sub-computation task corresponds to one tensor core; if a number of splits less than the preset number is used, tensor cores that are not assigned tasks remain idle or are transferred to other task scheduling.

[0104] By using the above method, the granularity of tensor computation tasks can be matched with the number of tensor cores and the scale of task data, resulting in a more balanced resource consumption during the tensor computation process.

[0105] In one possible implementation, after splitting the tensor computation task into a predetermined number of sub-computation tasks along a preset dimension and distributing each sub-computation task to each tensor core, the method further includes: if the number of splits is less than a preset number, releasing the hardware resources of the memory access control module corresponding to the tensor core that has not been allocated sub-computation tasks.

[0106] In the specific implementation, after determining the number of splits, the number of splits is compared with a preset number. When the number of splits is less than the preset number, it indicates that some tensor cores are not occupied by the current tensor computation task. The memory access control modules corresponding to these tensor cores that are not allocated to sub-computation tasks can be switched to the resource release state, and their internal queues, caches, and control states can be cleared or reclaimed, so that they no longer occupy memory access bandwidth, queue depth, or control context. The released hardware resources can be reinitialized and reassigned to the corresponding memory access control modules when a new tensor computation task is received subsequently.

[0107] First, based on the task splitting results, sub-computation tasks are distributed to the allocated tensor cores. Then, it is determined whether there are any idle tensor cores whose corresponding memory access control modules need to be released. When the number of splits is less than a preset number, the idle parts no longer retain their original memory access control state, but instead release their hardware resources in a timely manner, so that the current task only occupies the memory access capacity actually required for computation. This ensures that memory access control resources are matched with the task size and reduces the situation where modules not involved in computation occupy on-chip resources for a long time, thereby improving the resource utilization efficiency of the tensor computing device.

[0108] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0109] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0110] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0111] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0112] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0113] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0114] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0115] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0117] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope.

Claims

1. A tensor computation device, characterized in that, Deployed on a general-purpose graphics processor, the tensor operation device includes: an accumulation storage unit, multiple tensor cores, a remote storage medium, and a memory access control module corresponding to each of the tensor cores connected to the remote storage medium; The tensor core is configured to receive sub-computation tasks, generate memory access requests for the sub-computation tasks, and send them to the corresponding memory access control module; wherein, the sub-computation tasks received by the multiple tensor cores are obtained by splitting the same tensor computation task along a preset dimension; Each memory access control module corresponds to an independent memory access queue, which is configured to queue the memory access requests of the corresponding tensor cores received, and read tensor data from the remote storage medium according to the memory access queue before returning the corresponding tensor core. The tensor core is configured to perform operations on the read tensor data to obtain an operation output vector, and write the operation output vector into the accumulation storage unit; The accumulation storage unit is configured to accumulate newly written operation output vectors to obtain the operation result of the tensor computation task.

2. The apparatus according to claim 1, characterized in that, The memory access control module includes an incomplete memory access transaction circuit, which is equipped with a memory access queue for caching memory access requests for tensor data that have been initiated but have not yet returned a read response.

3. The apparatus according to claim 1, characterized in that, It also includes a shared transmission path extending along a preset direction; the preset direction is the direction in which the plurality of tensor cores are arranged. The shared transmission path is configured to transmit the common tensor data of the upstream tensor core to the downstream tensor core, so that the downstream tensor core can perform operations on the common tensor data and the read tensor data to obtain the operation output vector.

4. The apparatus according to any one of claims 1-3, characterized in that, The number of tensor cores is a power of 2, where N is an integer greater than 1.

5. A tensor operation method, characterized in that, The method is applied to the tensor operation apparatus provided in any one of claims 1-4; the method comprises: The tensor computation task is split into multiple sub-computation tasks along a preset dimension; Each of the sub-computation tasks is distributed to each of the tensor cores, so that each of the tensor cores generates a memory access request for each of the sub-computation tasks; The received memory access requests are queued to obtain a memory access queue. Tensor data is read from the remote storage medium according to the memory access queue, and the read tensor data is returned to the corresponding tensor core. Each tensor core performs operations on the read tensor data to obtain an operation output vector. Write the operation output vector into the accumulation storage unit; The result of the tensor computation task is obtained by accumulating the newly written operation output vector in the accumulation storage unit.

6. The method according to claim 5, characterized in that, The method further includes: The common tensor data is transmitted to the downstream tensor core through a shared transmission path, so that the downstream tensor core can perform operations on the tensor data read from the remote storage medium and the common tensor data to obtain the operation output vector.

7. The method according to claim 5 or 6, characterized in that, The process of splitting the tensor computation task into multiple sub-computation tasks along a preset dimension includes: The tensor computation task is divided into a predetermined number of sub-computation tasks along a predetermined dimension; the predetermined number is the number of tensor cores; or... Based on the data size of the tensor computation task, the number of splits is determined; the number of splits is less than or equal to the preset number. The tensor computation task is split into the number of sub-computation tasks along a preset dimension.

8. The method according to claim 7, characterized in that, After splitting the tensor computation task into the predetermined number of sub-computation tasks along a preset dimension, and distributing each of the sub-computation tasks to each of the tensor cores, the method further includes: If the number of splits is less than the preset number, then the hardware resources of the memory access control module corresponding to the tensor core of the unallocated sub-computation task are released.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 5-8.

10. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method described in any one of claims 5-8.