A heterogeneous computing direct memory access method and system based on tensor semantic perception
By constructing a sparse mask bitmap and tensor address generation unit within the DMA engine, the problem of non-continuous tensor data transmission is solved, achieving efficient hardware-level data transmission, reducing CPU load and bandwidth waste, and making it suitable for scenarios such as medical image processing, industrial robot control, and edge large model inference.
Patent Information
- Application Number
- CN202610062930.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-19
- Publication Date
- 2026-05-19
- Estimated Expiration
- 2046-01-19
AI Technical Summary
Existing direct memory access (DMA) mechanisms cannot effectively handle non-continuous multidimensional tensor data, causing the CPU to need to perform software-level data rearrangement and full sparse data transfer, increasing system latency and bandwidth waste.
The method adopts a tensor semantic awareness-based DMA approach. By constructing a sparse mask bitmap and tensor address generation unit within the DMA engine, hardware-level data filtering and packaging are achieved, enabling the direct transmission of non-continuous tensor data and reducing CPU computing power consumption and bandwidth waste.
It significantly reduces CPU data rearrangement overhead, improves PCIe bandwidth utilization, and reduces system power consumption, making it particularly suitable for performance-constrained device and edge applications.
Smart Images
Figure CN121524110B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer technology, and in particular to a method and system for direct memory access in heterogeneous computing based on tensor semantic awareness. Background Technology
[0002] In current artificial intelligence and high-performance computing scenarios, computational tasks are typically completed collaboratively by a central processing unit (CPU) and dedicated AI acceleration chips (such as GPUs, NPUs, or ASICs). This heterogeneous architecture relies on the PCIe bus for extensive data exchange. However, existing Direct Memory Access (DMA) data transfer mechanisms are designed based on traditional linear memory models, which presents a significant mismatch with the data characteristics of AI applications, leading to the following technical challenges.
[0003] First, AI model data typically exists in the form of multidimensional tensors, such as feature maps or weight matrices in convolutional neural networks. Their logical dimensions may include batch, channel, height, and width. In physical memory, this data is often stored non-contiguously due to alignment requirements, padding, or to accommodate the specific layout of the computational core. Traditional DMA controllers only support linear block transfer modes of "source address, destination address, contiguous length." To move non-contiguous tensor data to the AI chip, the CPU must first perform software-level data reordering or packing operations to organize it into contiguous memory blocks. This process consumes a significant amount of CPU computing power, significantly increasing system latency. Especially in scenarios where CPU performance is limited, such as edge devices or all-in-one machines, the CPU often becomes the bottleneck in data pipeline operation.
[0004] Secondly, with the development of large model technology, sparsity has become an important means of reducing computational load. However, existing DMA mechanisms cannot detect data sparsity and still treat matrices containing a large number of zero values as dense data for full transmission. This not only wastes valuable PCIe physical bandwidth but also increases the power consumption of ineffective writes to the receiving end memory.
[0005] The above background information is provided only to assist in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this application, nor does it necessarily provide technical teaching. In the absence of clear evidence that the above information was disclosed before the filing date of this application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention
[0006] The purpose of this invention is to provide a method and system for direct memory access in heterogeneous computing based on tensor semantic awareness.
[0007] To achieve the above objectives, the technical solution adopted in the first aspect of the present invention is as follows:
[0008] A direct memory access method for heterogeneous computing based on tensor semantic awareness includes the following steps:
[0009] A tensor DMA descriptor is constructed in the memory of the transmission end, wherein the tensor DMA descriptor includes a sparse mask bitmap;
[0010] The first DMA engine reads the sparse mask bitmap;
[0011] The first tensor address generation unit in the first DMA engine queries the mask bits in the sparse mask bitmap to generate and filter addresses. When the data corresponding to the mask bit is a non-zero value, the first DMA engine generates a read request. When the data corresponding to the mask bit is a zero value, the first DMA engine does not generate a read request.
[0012] The first DMA engine packages the non-zero value data it reads into a data packet.
[0013] In some embodiments, the first DMA engine is located at the transmitting end, and after the data packet is generated, the data packet is transmitted to the receiving end via the PCIe link.
[0014] In some embodiments, when the first DMA engine generates a data packet, it adds the sparse mask bitmap to the header of the data packet.
[0015] In some embodiments, the second tensor address generation unit in the second DMA engine of the receiving end queries the mask bits in the sparse mask bitmap in the data packet to write the non-zero value data in the data packet to the corresponding offset position in the memory of the receiving end.
[0016] In some embodiments, after the first DMA engine reads the sparse mask bitmap, it stores the sparse mask bitmap in a cache within the first DMA engine so that the first tensor address generation unit can query the mask bits in the cache.
[0017] In some embodiments, the tensor DMA descriptor further includes fields for describing tensor geometric features, and / or fields for describing data attributes, and / or fields for describing data addresses.
[0018] In some embodiments, the fields used to describe the geometric features of the tensor include a dimension field and a step size field.
[0019] In some embodiments, the field used to describe the data attributes includes a type field and the sparse mask bitmap.
[0020] In some embodiments, the fields used to describe the data address include a data source base address field and a destination base address field.
[0021] In some embodiments, after the data transfer task is initiated by the transfer driver, the tensor DMA descriptor is constructed in the memory of the transfer end.
[0022] In some embodiments, the tensor DMA descriptor further includes an identifier field. When the first DMA engine reads the identifier field to identify that the data to be transmitted is a sparse tensor, the first DMA engine reads the sparse mask bitmap.
[0023] In some embodiments, when the first DMA engine reads the identification field to identify that the data to be transmitted is a continuous tensor, the first DMA engine packages the data to be transmitted to generate a data packet.
[0024] In some embodiments, the transmitting end is a host end, and the receiving end is a device end or an AI chip end.
[0025] To achieve the above objectives, the technical solution adopted in the second aspect of the present invention is as follows:
[0026] A heterogeneous computing direct memory access system based on tensor semantic awareness is proposed, applying the tensor semantic awareness-based heterogeneous computing direct memory access method described above.
[0027] The beneficial effects of the technical solution provided by this invention are as follows:
[0028] a. The heterogeneous computing direct memory access method and system based on tensor semantic awareness of the present invention sinks the "tensor address calculation" and "data rearrangement" tasks originally undertaken by CPU software to the DMA hardware, which significantly reduces CPU computing power;
[0029] b. This invention defines a tensor DMA descriptor format, abandoning the single length field and instead introducing a sparse mask bitmap. A first tensor address generation unit (Tensor AGU) is integrated inside the first DMA engine. This unit can parse the above sparse mask bitmap and calculate the discrete address sequence of non-contiguous data in physical memory in real time, thereby generating a series of data packets. This process realizes hardware-level "scatter-gather" transmission, enabling non-contiguous tensor data to be transmitted as efficiently as continuous data, completely eliminating the CPU's data rearrangement overhead. Attached Figure Description
[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0031] Figure 1 A schematic diagram of the architecture of a heterogeneous computing direct memory access system based on tensor semantic awareness is provided as an exemplary embodiment of the present invention.
[0032] Figure 2 A flowchart of a direct memory access method for heterogeneous computing based on tensor semantic awareness, provided as an exemplary embodiment of the present invention. Detailed Implementation
[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0034] It should be noted that the terms "comprising" and "having" and any variations thereof in this invention are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.
[0035] This invention provides a method for direct memory access in heterogeneous computing based on tensor semantic awareness and a system for direct memory access in heterogeneous computing based on tensor semantic awareness using the same method. In a PCIe bus environment, this method can perceive the multi-dimensional tensor features of artificial intelligence data and supports sparse compressed transmission. Specifically, it includes the following steps:
[0036] The S1 transmission end driver initiates a data transmission task;
[0037] S2 constructs a tensor DMA descriptor in the memory of the transmission end, wherein the tensor DMA descriptor includes a sparse mask bitmap. In some embodiments, the tensor DMA descriptor also includes fields describing tensor geometric features, fields describing data attributes, and fields describing data addresses. The fields describing tensor geometric features include a dimension field and a stride field. The fields describing data attributes include a type field and a sparse mask bitmap. The fields describing data addresses include a data source base address field and a destination base address field.
[0038] S3 determines whether the data to be transmitted is a sparse tensor. When it is a continuous tensor, the first DMA engine packages the data to be transmitted to generate a data packet, without performing subsequent address generation and filtering based on the sparse mask bitmap.
[0039] S4 When the data to be transmitted is a sparse tensor, the first DMA engine reads the sparse mask bitmap. In some embodiments, after reading the sparse mask bitmap, the first DMA engine stores the sparse mask bitmap in the buffer area inside the first DMA engine so that the first tensor address generation unit can query the mask bit in the buffer area.
[0040] The first tensor address generation unit in the S5 first DMA engine queries the mask bits in the sparse mask bitmap to generate and filter addresses. When the data corresponding to the mask bit is non-zero, the first DMA engine generates a read request. When the data corresponding to the mask bit is zero, the first DMA engine does not generate a read request. After each mask bit is judged, the first DMA engine also needs to judge whether the current mask bit is less than the total mask bits in the sparse mask bitmap in order to complete the judgment of all mask bits. The first DMA engine is set at the transmission end.
[0041] The S6 first DMA engine packages the read non-zero value data into a data packet. Specifically, when the first DMA engine generates a data packet, it adds a sparse mask bitmap to the header of the data packet.
[0042] S7 data packets are transmitted to the receiving end via the PCIe link;
[0043] The second tensor address generation unit in the second DMA engine of the S8 receiver queries the mask bits in the sparse mask bitmap of the data packet to write the non-zero value data in the data packet to the corresponding offset position in the memory of the receiver.
[0044] In some embodiments, the transmitting end is specifically the host end, and the receiving end is the device end or the AI chip end.
[0045] The dimension field indicates the logical shape of the data, including batch, channel, height, and width. Specifically, N represents batch, C represents channel, H represents height, and W represents width. It records the number of elements in each logical dimension. It also includes a dimension field, which indicates whether the data is a 2D matrix or a 4D tensor.
[0046] The stride field is used to indicate the number of bytes that each dimension of the data jumps in physical memory, including height and width.
[0047] A sparse mask bitmap points to a distribution map that stores the data to be transmitted. It can be described using a binary stream or a hexadecimal stream.
[0048] The following description is based on specific embodiments:
[0049] In one embodiment, a 4x4 INT8 single-byte matrix needs to be transmitted, as follows:
[0050] Line 0: [ 10 ] [ 0 ] [ 0 ] [ 5 ];
[0051] Line 1: [ 0 ] [ 0 ] [ 0 ] [ 0 ];
[0052] Line 2: [ 0 ] [ 20 ] [ 0 ] [ 0 ];
[0053] Line 3: [ 0 ] [ 0 ] [ 0 ] [ 3 ].
[0054] Its logical shape is: N=1, C=1, H=4, W=4; its data content is mostly 0, with only 4 non-zero values, i.e., sparsity of 75%.
[0055] Source base address: 0x80001000 represents the data area to be transmitted, and 0x80002000 represents the address where the sparse mask micromap is stored.
[0056] Sparse mask bitmap: The 16 positions are described by a 16-bit bitmap: binary stream: 1001 0000 0100 0001, hexadecimal: 0x9041.
[0057] The tensor DMA descriptors specifically constructed in this embodiment are shown in Table 1.
[0058] Table 1
[0059]
[0060] The first DMA engine reads the flags field and identifies the data to be transferred as a sparse tensor. Based on the address indicated in the sparse mask pointer (mask_ptr) field, the first DMA engine reads the sparse mask bitmap 1001 0000 0100 0001 and stores the sparse mask bitmap in the buffer inside the first DMA engine so that the first tensor address generation unit can query the mask bits in the buffer.
[0061] The first tensor address generation unit (AGU) within the first DMA engine queries the mask bits in the sparse mask bitmap to generate and filter addresses. In this embodiment, it loops from Index0 to 15. The loop processes of Index0, Index1, and Index3 are listed below:
[0062] Index0: Corresponds to the first number in the sparse mask bitmap.
[0063] The AGU calculates the physical address as src_addr + 0.
[0064] The 0th bit in the sparse mask bitmap of AGU is 1.
[0065] Action: The first DMA engine generates a read request and moves 10 data points.
[0066] Index 1: The second number corresponding to the sparse mask bitmap.
[0067] The AGU calculates the physical address as src_addr + 1.
[0068] The first bit in the sparse mask bitmap of AGU is 0.
[0069] Action: Skip, the first DMA engine does not generate a read request.
[0070] Index 3: The fourth number corresponding to the sparse mask bitmap.
[0071] The AGU calculates the physical address as src_addr + 3.
[0072] The 3rd bit in the sparse mask bitmap of AGU is 1.
[0073] Action: The first DMA engine generates a read request and moves data 5.
[0074] The above address generation and filtering rely on the first tensor address generation unit (AGU), which is a pure hardware logic module that does not depend on the embedded CPU, does not consume CPU computing power, and only generates 4 read requests in this case, ultimately only needing to transmit 4 non-zero data values, which is significantly reduced compared to transmitting 16 data values.
[0075] The first DMA engine then packages the read non-zero value data into a data packet. When the first DMA engine generates the data packet, it adds a sparse mask bitmap to the header of the data packet.
[0076] Data packets are transmitted to the receiving end via the PCIe link, achieving line-speed bandwidth compression.
[0077] The second tensor address generation unit in the second DMA engine of the receiving end queries the mask bits in the sparse mask bitmap in the data packet to write the non-zero value data in the data packet into the corresponding offset position in the memory of the receiving end, that is, to restore the data position according to the sparse mask bitmap.
[0078] Through the above-described embodiments, the present invention can significantly reduce the CPU load in heterogeneous computing systems, improve PCIe bandwidth utilization, and ensure the real-time performance of key control signals. It is particularly suitable for application scenarios with stringent performance and stability requirements, such as medical image processing, industrial robot control, and edge large model inference.
[0079] The tensor semantic-aware heterogeneous computing direct memory access method of the present invention has the following significant beneficial effects and technical advantages:
[0080] Achieving "zero-copy" transmission and completely freeing up CPU computing power: In existing technologies, processing sparse tensors requires CPU intervention to perform "read-rearrange-packaging" software operations. This not only consumes valuable host computing power but also increases the read-write back-and-forth of the system bus. This invention integrates a pure hardware first tensor address generation unit (Tensor AGU) within the first DMA engine, which can directly understand the logical structure of high-dimensional tensors, such as step size and dimension. The first DMA engine can directly retrieve data from discrete addresses in physical memory and automatically complete the concatenation, achieving true "zero-copy" transmission. This eliminates the CPU's data handling load, allowing the CPU to focus on business logic processing, making it particularly suitable for scenarios with limited CPU resources, such as device-side or embedded AI all-in-one machines.
[0081] Breaking through the bottleneck of sparse transmission and significantly improving the effective bandwidth utilization: With the development of large model sparsity technology, the existing DMA mechanism cannot detect "zero values" and still performs full data transmission, resulting in PCIe bandwidth being occupied by invalid data and increasing the write power consumption of the receiver's memory. This invention introduces a sparse mask bitmap prefetching and hardware filtering mechanism. Before generating a bus request, the first DMA engine first queries the sparse mask bitmap, automatically removes invalid zero values, and generates data packets only for valid data. In typical sparsity scenarios, such as 70%-80%, it reduces invalid write operations at the receiver and lowers the overall dynamic power consumption of the system.
[0082] High flexibility and compatibility of hardware and software collaboration: This invention shields the driver from the details of the underlying physical address calculation and achieves efficient execution through hardware logic. This architecture has extremely high versatility, supporting not only conventional dense matrix transmission, but also various sparse formats such as Bitmap and N:M Sparsity. Moreover, it does not require modification of the existing PCIe physical link layer protocol and is easy to integrate into existing SoC systems.
[0083] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A direct memory access method for heterogeneous computing based on tensor semantic awareness, characterized in that, Includes the following steps: A tensor DMA descriptor is constructed in the memory of the transmission end, wherein the tensor DMA descriptor includes a sparse mask bitmap; The first DMA engine reads the sparse mask bitmap; The first tensor address generation unit in the first DMA engine queries the mask bits in the sparse mask bitmap to generate and filter addresses. When the data corresponding to the mask bit is a non-zero value, the first DMA engine generates a read request. When the data corresponding to the mask bit is a zero value, the first DMA engine does not generate a read request. The first DMA engine packages the non-zero value data it reads into a data packet.
2. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 1, characterized in that: The first DMA engine is set at the transmission end. After the data packet is generated, the data packet is transmitted to the receiving end through the PCIe link.
3. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 2, characterized in that: When the first DMA engine generates a data packet, it adds the sparse mask bitmap to the header of the data packet.
4. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 3, characterized in that: The second tensor address generation unit in the second DMA engine of the receiving end queries the mask bits in the sparse mask bitmap in the data packet to write the non-zero value data in the data packet into the corresponding offset position in the memory of the receiving end.
5. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 1, characterized in that: After reading the sparse mask bitmap, the first DMA engine stores the sparse mask bitmap in the cache area inside the first DMA engine so that the first tensor address generation unit can query the mask bit in the cache area.
6. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 1, characterized in that: The tensor DMA descriptor also includes fields for describing tensor geometric features, and / or fields for describing data attributes, and / or fields for describing data addresses.
7. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 6, characterized in that: The fields used to describe the geometric features of the tensor include a dimension field and a step size field.
8. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 6, characterized in that: The fields used to describe data attributes include a type field and the sparse mask bitmap.
9. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 6, characterized in that: The fields used to describe the data address include the data source base address field and the destination base address field.
10. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 1, characterized in that: After the data transfer task is initiated by the transfer driver, the tensor DMA descriptor is constructed in the memory of the transfer end.
11. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 10, characterized in that: The tensor DMA descriptor also includes an identifier field. When the first DMA engine reads the identifier field to identify that the data to be transmitted is a sparse tensor, the first DMA engine reads the sparse mask bitmap.
12. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 11, characterized in that: When the first DMA engine reads the identifier field to identify that the data to be transmitted is a continuous tensor, the first DMA engine packages the data to be transmitted to generate a data packet.
13. The heterogeneous computing direct memory access method based on tensor semantic awareness according to claim 2, characterized in that: The transmitting end is the host end, and the receiving end is the device end or the AI chip end.
14. A heterogeneous computing direct memory access system based on tensor semantic awareness, characterized in that, The method for direct memory access in heterogeneous computing based on tensor semantic awareness as described in any one of claims 1-13 is applied.